Post Project Review: Geraboam Anesthesia Clinical

Information System
Donovan F. McNabb
e-mail: choke@psu.edu
Abstract
A post project review is an activity where participants reflect upon a previous project for the
purpose of learning from past experiences. The goal is to improve future projects by applying
lessons learned from the past. This paper is a post project review, based on the work of Norman
Kerth, of the development of the Geraboam Anesthesia Clinical Information System, in which
this author was a participant.

Project Description

Hyslip Medical has been a leading provider of anesthesia equipment throughout the US for the
past 25 years. As a natural extension of this core business, Hyslip has also developed clinical
information systems for use in hospital operating rooms since the early 1990s. During the close
of the last decade and in conjunction with a major teaching hospital, Hyslip developed a perioperative recording application for creating an electronic anesthesia record. The application,
running on a personal computer in the operating room, physically interfaced with the anesthesia
machines, ventilators, and patient monitoring devices recording patient physiological and
respiratory data. A computer monitor with a touch screen interface provided the anesthesiologist
easy access for input of additional data such as operating room events, drugs administered,
patient assessments and outcomes. The system could follow a patient from surgery through
recovery. The results were a detailed, computer printed record for the patients chart instead of a
handwritten record, and a database of patient surgical data for clinical research purposes and
easy export to automated billing systems.
In 2001, Hyslip decided to create a completely new system based upon this existing application.
The new enterprise grade platform would employ the latest technology and design principles,
and provide a base upon which future applications could be built that would expand the
functionality of the system into other areas of the hospital. The project was called Geraboam
Anesthesia. The first phase of the project entailed an analysis of the current application to
determine requirements for the new system, and the design of a completely new multi-tiered
architecture following current best practices. The system was implemented in C# and the
Microsoft .NET Framework through four iterations, each lasting three to four months. The
project schedule showed that development officially began on January 16, 2002. The system
was deployed to its first beta test site on November 21, 2003, with the general production release
following in April of 2004.

In preparation for this project, the software development staff was expanded All staff were on
board and part of the project from day one, with the exception of the engineering test team,
which began with a single person. The author joined the test team in May, 2003, as one of the
last two members. A knowledgeable Hyslip marketing staff worked closely with the software
developers and test teams developing and verifying requirements, testing, and beta site feedback
analysis and implementation. The Quality Assurance team worked closely with the engineering
test team in verification and validation. The full time development team is represented in Figure
1.

Figure 1 Geraboam Anesthesia Project Staffing

Post Project Review: Planning

Of the many planning activities that would normally go into a post project review, two activities
stand out as strategic in nature, influencing both other planning activities and the conduct of the
post project review sessions. These activities are selecting goals for review, and evaluating the
health of the organization.
2.1

Goals of the Post Project Review

Selecting goals for the post project review process provides the strategic foundation for the
process, and Kerth (2006) suggests a number of possible goals for holding a post project review.
Some of these goals are related directly to the software development process, such as: 1)
capturing effort data; 2) improving upon the process, procedures, management and culture; and
3) capturing the collective wisdom of a team that will disperse at the conclusion of the project.
Other goals are related to the emotional needs of the team members, such as: 1) getting the story
out about little known events and issues, both positive and negative; 2) repairing damage to team
relationships; and 3) enjoying the accomplishment of producing a new product. Any number of
these goals may be selected for a single project review.

For this review of the Geraboam Anesthesia project, the author selected the goal of improving
the development and testing process, specifically to develop lists of things that worked well, and
things to do differently next time. A post project review conducted on an individual basis could
meet this goal. Without the team involvement in the review process, most of the goals listed
above are not attainable.
2.2

Health of the Organization

A primary consideration in planning and conducting a post project review is the health of the
organization. Observations of a groups interactions can give clear clues as to the level of
dysfunction present in the organizations culture.
The Geraboam Anesthesia project team was long lived group, meaning the team was together
before this project, and would remain together after the project. Most members were long time
employees of Hyslip (5 to 15 years). About a quarter of the team had joined only recently
(within 2 years). As mentioned earlier, the author was among the newest members. In
evaluating the health of the Geraboam Anesthesia development team, the author used Kerths
criteria for distinguishing functional and dysfunctional cultures. The results of this evaluation
are represented in table 1.
Table 1 Health of the Geraboam Organization (Kerth, 2006)
Dysfunctional Cultures

Functional Cultures

Guarded language and secrets

Distrust of other groups

Well defined boundaries between groups; lots

of discussion over whose responsibility it is;
management driven
Blame and lack of respect for other groups

Skepticism of someone else's new idea or

approach. Rewards for fighting someone else's
idea.
Pressure to produce
Behavior, process and activities highly
influenced by past
Strong pressure to conform to the standard

Internal competition and survival are key

issues; looking good is the way to progress

Meetings are confrontive

Engineers and managers
change the organization
Decision making involves
the goal is to win the
discussion centers around
accomplish something.

feel powerless to
lots of debate, and
debate. Often the
"the best way" to

Honest communication
Alliance and cooperation with other
groups
Boundaries are mutually discussed and
agreed upon between groups; participants
driven
Appreciates and uses the differences
between groups
Group refinement of other's idea. Careful
evaluation once experimentation with the
idea has been performed.
Encouragement to improve
Situations handled in the present with
creative new solutions
Flexibility available for situations that are
unique or new
High quality results is the key issue;
everyone looking good is the way to
progress
Meetings are constructive
Engineers and managers believe they can
change their organization
Decision making is consensus driven,
where "your idea is good enough" is
combined with the author's responsibility
to "make it work."

Dysfunctional Cultures
-

During decision making, proof of concept is

required from colleagues; distrust of ideas and
approaches is the foundation of the working
relationship.

Functional Cultures

During decision making, respect for and

trust of colleagues' skills is obvious.

Indeed, the author evaluated the Geraboam Anesthesia development team as an extremely
functional group, in many ways. Members of the team were respectful, considerate of others,
welcomed newcomers into the group, worked well together, and behaved as mature adults in
stressful situations. Management, likewise, showed support for the team to upper management,
made sensible and informed decisions, communicated in non-threatening ways, and created an
enjoyable atmosphere in which to work. This glowing depiction does not mean the team was
devoid of interesting interpersonal issues, but that the character of the working environment truly
made one feel good about going work everyday. After being with the company about two years,
the author was surprised to learn that the software engineering department was very much an
oasis in the desert. This supportive environment did not exist in other groups within the
company. Much credit must be given to the management team (program manager, project
manager, software manager) for their selection of team members and fostering a functional
culture.
Another interesting observation about the software development process at Hyslip concerns its
similarity to the Principles behind the Agile Manifesto. While the development process was not
agile, the following Principles behind the Agile Manifesto principles were evident at Hyslip:
Build projects around motivated individuals. Give them the environment and
support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a
development team is face-to-face conversation.
Agile processes promote sustainable development. The sponsors, developers, and
users should be able to maintain a constant pace indefinitely.
Management entrusted development tasks to individual team members, doing the best they could
to provide necessary resources, and did not hound them for status reports and schedule updates.
Small conferences took place in various cubicles on a regular basis with participants sitting on
desks and the floor, while email was left primarily for passing information. Critical times, such
as the end of iterations and the first beta test installation, required extra effort, but overall a
sustainable pace was encouraged to keep the team performing at its best.

Post Project Review: Activities

A series of activities designed to explore the significant events of a project and the participants
feelings about the project form the core of the review. Many activities suggested by Kerth
require team involvement to be meaningful. Specific activities may vary depending upon the

goals selected for the review, and the health of the organization that is revealed as the review
process progresses. The activities listed here as part of the Geraboam Anesthesia project
includes only those that could be completed individually.
3.1

Define Success

The Geraboam Anesthesia development project was considered successful by all involved. Even
upper management had significant praise once the project was completed. Prior to completion,
some displeasure was communicated regarding a schedule overrun (and associated cost overrun),
but this complaint seemed to evaporate once the product was officially released. The
development team took great pride in delivering a product of high quality with all the
functionality promised when the project began.
In contemplating the factors that made this a successful project, the author reviewed the patterns
of software systems failure and success from Jones (1996). Table 2 contains the authors
analysis of technological factors in Geraboam Anesthesia project.
Table 2 Patterns of Software Systems Failure and Success: Technology (Jones, 1996)
Technologies on unsuccessful projects

No historical software measurement data

Failure to use automated estimating tools
Failure to use automated planning tools
Failure to monitor progress or milestones
Failure to use effective architecture
Failure to use effective development methods
Failure to use design reviews
Failure to use code inspections
Failure to include formal risk management
Informal, inadequate testing
Manual design and specification
Failure to use formal configuration control
More than 30% creep in user requirements
Inappropriate use of 4GLs
Excessive and unmeasured complexity
Little or no reuse of certified materials
Failure to define database elements

Technologies on successful projects

Accurate software measurement

Early use of estimating tools
Continuous use of planning tools
Formal progress reporting
Formal architecture planning
Formal development methods
Formal design reviews
Formal code inspections
Formal risk management
Formal testing methods
Automated design and specifications
Automated configuration control
Less than 10% creep in user requirements
Use of suitable languages
Controlled and measured complexity
Significant reuse of certified materials
Formal database planning

Most of the technologies associated with successful projects were used on the Geraboam
Anesthesia project. One curiosity was the lack of historical software measurement data, even
though the team had previous software development experience. In fact, there was a general
unconcern with measurement of the software of any kind. When the author questioned the
number of source lines of code in the system, no one knew, nor cared to count them. The

attitude was that there were higher priority items to worry about than the number of lines of
code.
Table 3 contains the authors analysis of social factors in Geraboam Anesthesia project.
Table 3 Patterns of Software Systems Failure and Success: Social Factors (Jones, 1996)
Social factors observed on
projects

Excessive schedule pressure

Executive rejection of estimates
Severe friction with clients
Divisive corporate politics
Poor team communications
Nave senior management
Project management malpractice
Unqualified technical staff

Social factors observed on successful

projects

unsuccessful

Realistic schedule expectation

Executive understanding of estimates
Cooperation with clients
Congruent management goals
Excellent team communications
Experienced senior executives
Capable project management
Capable technical staff

Once again, the social factors observed on successful projects were observed in the Geraboam
Anesthesia project. The sole item in the first column was marked because the initial project time
estimates were rejected by upper management. This issue will be discussed further under What
Worked Well, and What to Do Differently Next Time.
3.2

Review of project artifacts

This activity involved collecting and reflecting on project and personal artifacts, items of
significance gathered during the project execution. The purpose is to think back across the life
of the project and remember what had been forgotten. The authors list of artifacts consisted
primarily of project documents, along with two personal items, and included:

Project schedules The project managers original project schedule, and a copy of the
updated schedule from the end of the project.

SCR (Software Change Request) Reports The author assumed the responsibility of
tracking bugs through each build of the software with statistics such as bugs fixed, new
bugs discovered, and classifications.

Performance Test Reports The author spent two months driving in to work at odd hours
to kick off 12 hour automated performance tests designed to determine if the application
would run satisfactorily on some customers current hardware.

Miniature Flag of St. Andrew and Scottish Quaich The author and a colleague visited
our development partner in Scotland to begin testing the upgrade process for European
customers. The flag stood atop a Christmas pudding, and the Quaich (a traditional
Scottish drinking vessel) was a gift from the partners management team.

An examination of these artifacts and reflection on their meaning had the desired effect,
rekindling many memories of the events of two years ago. Many of these memories are
referenced in the next activity of developing a timeline.
3.3

Develop a Timeline / Mining the Timeline

The objective of this activity is to develop a timeline of significant project events. In a group
setting, Kerth (2006) accomplishes this by having participants write events on cards and then
arranging them chronologically on the wall. In this case, the author wrote down on the updated
project schedule significant events from his freshly rekindled memories. Some events were
already on the schedule, such as the go live at the beta test site. Other events were added, such
as the performance testing and the trip to Scotland.
Once the timeline has been constructed, it is then analyzed or mined for information. The goal
is to scrutinize these significant events to generate entries in lists titled what worked well,
lessons learned, and what to do differently next time. These entries are the gold that is
being mined, the collective wisdom of the project team. These three lists are then prioritized.
The results of the authors are listed in the sections below.
3.4

Determine Costs

Determining costs can be an important part of the post project review. Project effort and
software size data can be collected with the project teams help. For the Geraboam Anesthesia
project, both effort data and size data were not available, nor was a method of extrapolating that
data. Kerth (2006) asserts that there is another kind of cost data that is valuable and should be
collected. That cost is the personal sacrifice made by the project team members in the course of
bringing the project to completion.
As the author considered this personal cost aspect for the Geraboam Anesthesia project, he
remembered the many summer weekends spent in the lab testing because the project was falling
behind schedule, the all night session during the data conversion at the beta test sight, missing a
daughters Christmas concert (first time ever) while away on business, and many days of being
one of the last to leave the building. Yet these memories all have another side. When bonuses
were handed out at the end of the project, extra was received because management took notice of
the weekend test effort. The all night session during data conversion was more of a pizza and
game party than a mission in drudgery. The missed concert was because of the company paid
vacation to Scotland (or so it seemed). While the author has heard stories and witnessed
incidents of personal sacrifice among those in the software development industry, in this
particular environment and on this project, the personal sacrifices were rewarded.

Post Project Review: Results

Listed in the sections below are the items identified during the mining the timeline activity.
The goal that was chosen at the beginning of this project review was to improve the development

and testing process. These are the observations and suggestions that, if applied, will fulfill this
goal.
4.1

What Worked Well

Functional corporate culture and qualified team The management of the Software engineering
group was successful in building a functional corporate culture in contrast to the corporate
culture at large. This proved a significant factor in the success of the project.
Iterative software development process This project was the first time an iterative approach to
software development was used by this team. It worked well.
Automated test tool An automated test tool was used for the first time. While it proved rather
difficult to use, it was invaluable during performance and system testing.
Estimation process Despite the fact that the project was completed four months late, the project
did finish within one month of the original engineering estimate. This estimate was cut by
marketing and management for political reasons (make the project more palatable to upper
management).
Process for managing partners The visit to our partner in Scotland at a critical juncture in the
development of the European upgrade process was crucial to building relationships and
establishing a framework for continued development. A team member was designated to
maintain daily contact and follow up on any issues immediately with the project manager.
4.2

Lessons Learned

Late consideration of performance issues The ability of customers to run the application on
existing hardware (promised by marketing) was not considered early enough in the testing cycle.
Feature implementation slipped during iterations As an iteration was completed, features
planned but not implemented were pushed off to the final iteration. This caused the final
iteration to have many more features implemented for the first time, resulting in many bugs
being discovered, requiring much more regression testing and many more test builds (schedule
optimistically called for four test builds based on earlier iterations; 10 were needed, plus three
beta test builds to handle the beta site validation and user issues).
Difficulty in developing an international product - Managing our German peers within the
company proved to be the most challenging people issue. Our German counterparts felt they had
little input into the project because many features of the anesthesia product currently in use in
Europe were not added to this new product. The European beta test was poorly handled by the
German sales and marketing staff, and was basically of little benefit (Almost two years later,
while the Germans have ignored the product, the tiny sales team in Belgium and the Netherlands
have a dozen new installations, because they didnt care that it didnt look like what they had
before).

4.3

What to Do Differently Next Time

More iterations in the development cycle Long iterations introduced too much new
functionality all at once. The test team would discover many bugs quickly, and then sit around
waiting for the next iteration to regress the fixes and test the next round of new features. Shorter
iterations may also help control the problem encountered where the final iteration introduced far
more new functionality than it should.
Develop an automated method for project tracking The process for tracking the software
change requests for each build of the software provided a good basis for predicting test effort.
This process required far too much manual effort, and an automated tool needs to be developed
to extract the information from the bug tracking database.
Conduct performance testing as soon as the product is ready As soon as the product reaches a
state where performance testing is possible, it should be conducted, regardless of the whether it
requested or considered important. Issues regarding hardware in the field will surely arise in the
future.
Application savvy personnel at the beta installation - While there were business and technical
people on site, there was no one familiar enough with the application to answer the questions of
the users as they transitioned from the old product to the new product. This can be extended to
include proper training for new users. Inadequate training accounts for an excessive number of
service calls.
Push back harder when pressured to cut schedules With the validation of the engineering
estimation process provided by this project, pushing back against upper management pressure to
unrealistically shorten schedules may be easier.

Conclusions and Recommendations

Based on the results obtained, the author would consider this post project review quite
successful. Some items appearing the what worked well, lessons learned, and what to do
differently next time lists were observations made during the project, but forgotten. Some items
were realized for the first time as a result of reflecting on the project artifacts and constructing
the project timeline. The value in this formal review process is that these insights have been
gathered, recorded, and are available for review in the preparation for future projects.

References

Jones, C., (1996), Patterns of Software Systems Failure and Success, Boston, MA: International Thomson Computer
Press.
Kerth, N. L., Elite Systems, (2006), An Approach to Postmorta, Postparta & Post Project Reviews. Retreived
February 5, 2006, from http://c2.com/doc/ppm.pdf.
The Agile Alliance, (2001), Principles behind the Agile Manifesto. Retreived February 21, 2006, from
http://agilemanifesto.org/principles.html.