Iterative Development - delivering projects

on time
The history
In 1970, Winston W.Royce wrote an article1 in which he described a sequential model for
developing software wherein development flows, waterfall-like, through phases of requirements,
analysis, design, implementation, testing, and maintenance. He then went on to explain why this
model couldn't work and described an iterative process as a much better alternative. Inexplicably,
not only did the software community latch on to the concept of the "Waterfall" approach, but
they even credited Royce with having proposed it. A common feature of waterfall projects is that
they tend to fail (18% cancelled, 53% late, over budget or descoped2).

This is how Royce illustrated the approach which subsequently became universally used and
known as the Waterfall method:

The approach is predicated on the principle of completing, in detail, each stage before moving on
to the next; the optimistic assumption is that the customer knows and can express the detailed
requirements at the outset, that the software requirements as specified can be implemented
predictably and that the requirements don't change during development.

Royce points out that the Testing stage, which occurs towards the end of the development cycle,
is the first time that the developed system is experienced, as distinct from analysed. In the event
that the delivered system is not what was expected or some requirements have been missed, then
a major redesign is necessary. At the very least, the system design must be revised in response to
weaknesses or errors exposed in testing, but if external criteria have not been satisfied then either
the requirements must be changed (descoped) or the whole system must be revisited. He
comments "In effect, the development process has returned to the origin and one can expect up to
a 100% overrun in schedule and/or costs".

Royce accepts that the sequence of events is logically sound but that the principle of finalising
each stage before the next commences is flawed. His outline alternative proposal comprises 5
steps:

1. Insert a preliminary Program Design phase between Software Requirements and

Analysis; whilst this must be based on incomplete information (as there is no analysis), it
will be able to deal with fundamental non-functional requirements such as performance
and volumes, thus setting some constraints for subsequent analysis.
2. Document the Design so that subsequent phases are working towards a common, well-
defined specification
3. Do it Twice. Construct a pilot version of the system, the contents of which will depend on
the extent and nature of critical problem areas. It is the second version which is delivered
to the customer.
4. Plan, Control and Monitor Testing
5. Involve the Customer so that he commits himself to the direction the development is
taking at the earliest stage.

In modern-day parlance, this proposed approach would be described as risk-driven and iterative.

The majority of software projects are today still based on the waterfall approach, perhaps with
some fine-tuning, and are often accompanied by intimidating methods and standards whose sheer
complexity implies sophistication - this can be misleading. Characteristics of such projects
typically include:

• Major problems typically occur only during the Integration stage, usually too late to be
accommodated within the project calendar. The effect tends to be either a descoping of
the project to deliver it on time, a ‘squeeze’ on the time allowed for testing to try to hit
the end date, an overrun or a combination of any or all of these.
• The tendency to build detailed project plans, right up to the end of the project, at as early
a stage as possible. These soon become obsolete and are typically subject to frequent and
time-consuming revision. This has a contingent effect in terms of disruption to the team’s
work schedules and often morale.
• Too much optimism in the earlier stages in terms of the amount of rework that will
ultimately be necessary and the lack of adequate planning for rework.
• Potential disconnect with the users between requirements gathering and delivery because
of the time lapse.

and last, but certainly not least -

• the insistence by management that an accurate estimate of cost, timescales and resource
requirements is submitted before the work even starts
The Iterative Approach
Royce set the scene for iterative development, although it must be recognised that it was not a
new concept: other scientific and engineering disciplines had been following an iterative
approach for generations, if often informally. The simple rationale is that if you're doing
something new, first explore the technical boundaries and constraints, by experimentation or
some other means, then, when you know what is technically viable, review the design with the
customer and confirm that both the technical and functional content are acceptable and
justifiable. There can be few among us who cannot recall a number of major public sector
projects which would have been abandoned at an early stage had the true ultimate costs and
timescales been known.

Of course, much has changed since Royce wrote his article in 1970. Software development
languages and tools, architectural concepts, reusable components, hardware cost and
performance, sophisticated modelling techniques - and the maturing of iterative development
techniques by a relatively small (and relatively successful) proportion of the development
community.

The iterative process adopts the logical sequence of events of the Waterfall model but it follows
the full lifecycle several times within a single project. Each iteration goes through the activities
of Requirements, Analysis, Design, Coding, Integration and Testing and produces production-
quality software at the end of the iteration. The content of each iteration is determined according
to risk - the new, the most challenging, the most critical elements are built first, usually by a core
team of the most skilled practitioners, so that the true nature of the project and its implications in
terms of cost and time are constantly under review. Thus, certainty increases as the project
proceeds - or, if the development is no longer viable, it can be halted at an early stage and at
minimum cost.

The actual number of iterations in a given project will depend on the nature of the project, in
particular the content that is considered risky, but it is likely to be between 3 and 10. An
important aspect of the approach is that each iteration is "time-boxed" - if it is planned to take 4
weeks, then it will cease at the end of that time. The subsequent iteration will incorporate any
unfinished work but with the benefit of knowledge gained during the execution of its
predecessor.

Summary and Benefits of Iterative Development

Iterative developments display these characteristics:

• Highest risk items are built first

• Change is expected and accommodated, but controlled
• Testing occurs early and often
• Iterations are time-boxed not function-boxed
• Deliver early, deliver often
• Deliver production quality software each time

Some benefits are:

• Early risk discovery and mitigation

• Accommodates change and provokes earlier identification of change
• Manageable complexity
• Confidence from early, repeated success
• Early partial product
• Better progress tracking and predictability
• Higher quality, fewer defects
• Software better matches user needs
• Early and regular process improvement
• Communication and engagement demanded
• Prototyping and feedback encouraged

A widely-used process based on the iterative approach is IBM's Rational Unified Process®,
commonly known as RUP