Whitepaper Combining Agile,

RRBT & TestFrame

About LogicaCMG in testing
LogicaCMG is a major international force in IT services and a thought leader in testing
methodologies. It employs around 7,000 testers out of total workforce of 40,000 across
the globe, has test factories in three different continents, and has developed industry-
leading testing methodologies and techniques from millions of hours testing over 30 years.
LogicaCMG's focus is on helping leading organisations worldwide to increase the
predictability of IT changes by a business outcome-led approach to testing while
improving quality, reducing time to market and lowering their IT costs. The company
provides managed testing and managed test environment services, business acceptance
management, quality assurance and niche testing services across diverse markets
including telecoms and media, financial services, energy and utilities, industry, distribution
and transport and the public sector. Headquartered in Europe, LogicaCMG is listed on
both the London Stock Exchange and Euronext (Amsterdam) (LSE:LOG; Euronext:LOG)
and traded on the Xternal List of the Nordic Exchange in Stockholm. More information is
available at www.logicacmg.com.
 Combining Agile, RRBT & TestFrame

Contents
1 WHAT IS AGILE? 0

2 AGILE, RRBT AND TESTFRAME 2

3 MAPPING AGILE AND RRBT 3

4 MAPPING AGILE AND TESTFRAME 5

5 AGILE TEST WARE 8

6 RRBT, TESTFRAME AND AGILE LIFECYCLES 10

7 CONCLUSION 14

A GLOSSARY OF TERMS 15

B RECOMMENDED READING 16

Author: Peter Kalmijn

00
1 What is Agile?
“Agile is an iterative and incremental (evolutionary) approach to software development
which is performed in a highly collaborative manner by self-organizing teams with "just
enough" ceremony that produces high quality software in a cost effective and timely
manner which meets the changing needs of its stakeholders.” definition by Scott W.
Ambler, one of the leading thinkers of the Agile movement.
This paper will demonstrate ways RRBT TestManagement and TestFrame can be used
inside Agile environments and contribute strongly to the success of Agile projects. Also it
will show how TestFrame can be applied in an Agile way itself.
Business sponsors, Project Managers and Process Improvement initiatives continue to search
for means to improve predictability of Software development projects many of which overrun
in terms of time to market and/or budget. In response to such overruns, quality or functionality
is usually sacrificed.
Various methodologies have evolved to solve this dilemma. Of these, Agile emerged in the
late 1990’s. The roots of Agile originated from the insight that a common contributor to many
project failures are poor people management and communication. Individuals who had
separately solved this issue in individual projects coalesced as the Agile Alliance and as result
in 2001 the Agile Manifesto was published:

) Agile Software Development Manifesto

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
www.agilemanifesto.org

Key features are:

• close collaboration between project managers, developers, testers and business subject
matter experts
• emphasis on face to face communication and agreement over written documentation
• frequent delivery of software with demonstrable new deployable business value
• tight, self-organizing teams rather than on rigid plans and mandated deliverables
• ways to manage the inevitable change to requirements without the need to to evoke a
crisis
Do Agile methods replace conservative or plan-oriented methodologies such as for example
Prince2? The answer is No; they can successfully co-exist. Wise project managers already
look to combine best practice from the available methodologies. Cit:1 ‘’.. the way PRINCE is
applied to each project will vary considerably, and tailoring the method to suit the
circumstances of a particular project is critical to its successful use”. Agile methods do
include practices that can be applied generally - regardless of methodology.

1
PRINCE2 par. 2.3: PRINCE in context

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
01
Appropriately applied, benefits of Agile include
• Faster software deliverables that are Business Value orientated, focussed on what is
needed by the business right now.
• Improved analysis and management of Risk (Project Risks and Product Risks)
• Frequent feedback to and re-direction from business facilitating better and faster decision
–making and so delivering more value
o Lower cost of change
o Early measurable business benefits
• Improving communications within and between project team & customer

Agile and Testing

Agile and Testing In the Agile environment, with its de-emphasis on process, there are many challenges for the
Tester. What is the role of the tester in Agile development projects? How does Testing fit
within Agile lifecycles?
In seeking to answer these questions, this paper will focus on:
1) Applying TestFrame in Agile ways, responding to change
2) Using RRBT and TestFrame inside Agile projects (SCRUM, XP)
For this we look into:
• the Role and Practice of Testing inside Agile projects
• The practical application of RRBT TestManagement and the TestFrame Framework
within Agile projects
• Agile Test Ware documentation practices
One of the controversial things about testing the agile way is that testing is far more explicitly
a service role rather than the more traditional independent risk and assurance role. The ability
to facilitate progress in the right direction is what counts. The tester serves the business expert
and the developers to this end.
The general purpose of testing remains valid: to provide timely information about the qualities
of the system being build. Testing is used particularly in agile projects to boost agility of both
the design and development process:
Testing is an Agility Testing is structurally interwoven with Requirements engineering, Software Development,
enabling factor and Analysis and Design.
Testing actively engages in defining requirements – Agile projects use testing to clarify
requirements and design.
Completion is only reached after passed tests – testing has become the real measure of
progress.
Continuous build and test – regression testing captures faults at the earliest stage possible,
while it is still quick and cheap to fix. This practice also supports safe refactoring.
Use of found flaws to tighten tests – future tests will capture this type of flaw.

Next we will investigate how RRBT and TestFrame can be applied in an agile way and how
they are blending well into agile development methods.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 02
2 Agile, RRBT and TestFrame
It is important to use a methodology that really fits the organisational situation. Both RRBT
and TestFrame are designed to enable this important fit to happen. Before continuing there is a
need to ensure a clear understanding of both RRBT and TestFrame.
What is RRBT Risk & Requirement Based Testing (RRBT) is a very successful TestManagement method that
has several strategic advantages:
• Informed decisions about management of risk; where to reduce risk by investing in testing
and were to accept risk and so save time and test budget
• Early detection of vague or missing requirements by mandating (peer) reviewing of
requirements for testability and linking risks with requirements
• Informed decisions about priority of testing; focusing test effort on areas of highest risk
first
• Communicating risk and associated decisions required in the language of the stakeholders
The core of the RRBT test strategy is risk management through appropriate application of test
effort, which is carried over to all RRBT Test Management activities and to TestFrame test
execution. This is described in more detail in “Successful Test Management: an integral
approach” [Pinkster 2004].
What is TestFrame TestFrame provides a testing framework within which RRBT test strategy, test set up and test
tools are tuned to complement one another, supporting the entire testing process, from
preparation to completion. TestFrame has been for years extensively by scores of
organizations and in many projects all over the globe.
The TestFrame vision is that all testing material should be simple to maintain and should be
reusable. The core components of this vision are: fitting, structuring and tooling.
Fitting - the testing process needs to fit the organization and its specific ways of systems
development.
Structuring - executed tests will only offer quality control if they are set up in a structured
manner and are easy to maintain.
Tooling - aid of automated tools is indispensable in a modern testing process.
TestFrame is an open standard, is supported by several tools which are provided in the
TestFrame support box and is described in “Integrated Test Design and Automation”
[Buwalda 2001].
Combining with Agile The agile testing movement has gained increasing popularity within development and test
departments because of its appeal of the perceived ease of setting up “lightweight” processes
vs. traditional “heavyweight” processes. Both operational culture and organization must be
considered too. In the implementation of Agile, as in any other process change, RRBT
TestManagement and TestFrame are readily adaptable to any organisation and process
frameworks. Therefore, a healthy and effective combination of RRBT TestManagement,
TestFrame framework and Agile test practices is an appealing option; with Agile test practices
oiling the process and RRBT and TestFrame contributing to support sustainable, repeatable
and improving agility over time.
Adopting ‘Agile testing’ inside more traditional method strongholds may prove tricky, but
Agile adoption here are some practical first steps:
Plan for training, select the right pilot project, and staff with the ‘A’ team.
One agile practice or technique at a time – people need time to get used to change.
Focus on selling the benefits and demonstrating the returns - While certain agile methods
may not fit in particular projects, some of their practices or techniques do provide valuable
benefits when applied in certain contexts. You will win people over by producing measurable
results.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
03
3 Mapping Agile and RRBT
In this section we look into the mapping of RRBT TestManagement and Agile. Contrary to
Agile, traditional development approaches try to ensure the projects success by rigidly
following strict processes and procedures. This may be good for repeatedly assembling the
same product, but rarely works for the innovative and fast creation process of software
systems delivering business value recognised by the users. Agile seeks to do this, and RRBT
TestManagement complements Agile by tracking the evolving business needs and safe
guarding their implementation into the system build.

Differences
Agile Conservative, plan-oriented
Outcome result guarantee process guarantee, repeatable
(Test) guiding vision, communicating Plan driven, taskmaster
Project leader
Plan Responding to change Following the Plan
Processes Simple rules & open information Defined work processes and controls
Teams Organic teams Fixed roles
Stakeholders Active Stakeholder participation upfront involvement
Design Evolving according to Stakeholder Big upfront design
Value
Delivery Short increments and frequent Delivery on the end of the project
delivery in UAT and BAT to real users

RRBT practices RRBT Risk Orientation – Complimentary to the Agile Project Management principle of
supporting Agile mitigating Product Risks (mostly architectural) at an early stage.
Product Risk Assessment & Classification – By classifying according to ISO9126 standard
RRBT provides a sound basis for an Agile Test Strategy.
Combining Product Risks and Requirements to determine Priorities – This fundamental
RRBT practice provides Agile Project Management with an excellent tool to manage risks and
to deliver the most valuable requirements first.
Risk Visibility – Since key Stakeholders are actively involved it is much easier to synthesize
both risks and requirements to reach a cohesive baseline. RRBT emphasizes the need for
quality, face-to-face interaction with Stakeholders to identify and prioritize risk. RRBT
supports continuous reporting to individual stakeholders, linking individual risk and the
associated mitigations to the stakeholders concerned.
Traceability – Both product Requirements and product Risks need to be traceable to the Test
Cases and Test Conditions that validate them. Getting this right is one of the greatest
challenges within an Agile project. RRBT supports this by supplying practical guidelines.
RRBT Cluster Cards – matches closely with SCRUM Feature Groups.
SCRUM Sprints (iterations) – are supported by RRBT/TestFrame Strategic Test Slicing
Method (STSM)
Active Stakeholder Participation – Since key Stakeholders are actively involved it is much
Agile practices easier to synthesize both risks and requirements to reach a cohesive vision as well as a
supporting RRBT cohesive set of requirements and risks to base testing on.
User story –creation and refinement is supported through workshops.
Acceptance tests – are supported by involving customer subject matter experts to create Test
Cases and their variants. Once automated the Test Cases are run frequently for regression
monitoring purposes.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 04
Refactoring and optimization - is supported by tactically using TestFrame Automation once
functionality has been accepted.
Continuous Testing – This finds defects in the earlier stages of software development,
minimizing their effect on the time schedule. TestFrame supports this practice by making Test
Automation easier to implement.
Feature Driven Development (FDD) – A group of features usually can be mapped directly to
a TestFrame test cluster. Combined with Feature-Based Design, Planning, Engineering,
Testing and Reporting; FDD enables the whole project to be geared around features and
associated Test Clusters. TestFrame has as a standard feature for Product Quality Monitoring
and Reporting on Test Clusters (i.e. Feature Groups).
Test Driven Development (TDD) – enables early involvement, so getting more grip on
Product Quality upfront.
Joint Test Ware Design (JTD) – Supports collaborative development
Concurrent Design, Engineering and Testing– This is especially useful in projects evolving
in response to external changes.
Agile Management Techniques – many of these are not new – many have been around for a
long time, and represent best practices of “traditional” project management. Using them within
the right context, they strengthen the overall approach.
Agile Meetings – Agile requires short and frequent team meetings and meetings with
stakeholders. These include Stand up Meetings, Scrum meetings, Whiteboard Sessions and the
like. This level of communication with business and development facilitates the
communication of risk that is a key requirement of RRBT.
In summary, the traditional test methodologies are sometimes considered bureaucratic,
prescriptive or "predictive" in nature. The emphasis is that the process is followed.
Lightweight and Agile methodologies on the other hand set the focus on the outcome and are
much more flexible about the process that leads to it.
Scale for Agility Implementing Agile practices requires both business insight and leadership skills. What
method to implement depends on the context of the project, which might include principles,
problem type, team dynamics and of course organizational culture and structure. The right
choice between using a light or heavy methodology will significantly influence the success of
the project. Keep in mind: Managing different stages and parts of the RRBT agile managed
project could well demand a different balance between a light scaled version of TestFrame and
a heavy scaled version of TestFrame. Both the balance and the flexible scalability determine
success. The right balance is what makes a project really agile.

Agility balance indicators

Aspect Rationale
Team size Greater number of staff needs more management overhead.
Project criticality Both urgency and critically of the project greatly influence the balance.
Technology used The usage of groupware or other aids to communicate quickly and at
any time.
Documentation More rigid methodology needs more documentation.
Training Effective training to key support staff and project managers is required.
Best practices/lessons Past lessons learned and good practices should be analyzed.
learned
Examination of existing The maturity of existing processes will influence the pace at which a
processes test project will progress.
Test tooling Right tooling can greatly contribute to the agility of the team.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
05
4 Mapping Agile and TestFrame
In this section of the paper we will investigate ways to combine TestFrame and Agile methods
and practices. TestFrame is a powerful tool to support common Agile practices. We will
demonstrate when and how to use it.
TF ‘fitting’ – TF is scalable and highly adaptable to the needs of different organisations. Thus
TestFrame practices you are able to select and apply the right components for each implementation.
supporting Agile Fast changing requirements – TestFrame’s simple Test Cases are easy to maintain and
modify in their nature.
Make next step possible – TestFrame’s simple Test Cases enable rapid scale-up to Test
Automation and TestFrame Integrator once requirements solidify.
Refactoring – TestFrame Automation framework supports Regression testing as well as the
structured scale down to manual testing to accommodate changing requirements. Once
requirements solidify again, it is easy to switch back to automated testing.
Rapid Quality Feedback – TestFrame provides continuous Product Quality Monitoring, even
throughout a different scaled mix of TestFrame implementations.
Travel light – Light versions of TestFrame allow for a jumpstart with a minimum of
documentation and overhead (within the regular TestFrame structure), yet supporting an Agile
approach with the ability to report accurately on product quality development.
TestFrame Action Words – provide a smooth transition to and from Automated Testing.
TestFrame Automation – supports continuous testing to keep the system stable as it is built
Agile practices and to support multiple Agile iterations and re-factoring.
supporting TestFrame Short feedback loops –supported by TestFrame test run system and PQM.
Test driven development – guarantees acceptance of the features built.
Joint Test ware Design (JTD) – guarantees the right test conditions are executed by
prioritizing on business value and business risk.
Frequent testing gives the customer and the developer’s confidence that the product is
evolving in the right direction in both functionality and quality.
Both SCRUM and XP (along with almost all Agile development techniques), mandate quick
Final Goal is and frequent feedback and there is no more important feedback than early indications of how
Acceptance well the system works. Acceptance tests allow the customer to assess that the system works
and how well the system works. This is why Agile methods advocate ongoing acceptance
testing by the customer as part of the iterative cycle. Once there is agreement between
Business, Development and Test that the developed software is approaching its final state then
a final acceptance test should be performed. These test activities of iteration acceptance and
the final and formal acceptance test level activities do complement each other. The test related
products such as test cases could well be reused in order to execute the final User Acceptance
Test (UAT). Iteration acceptance testing is preparation for the final UAT.
Now let us look into Agile TestFrame ways to reach the goal of final acceptance quickly,
effectively and accurate.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 06
A good practice is to begin creating Test Cases as soon as Use
From Use Case Cases (or the SCRUM equivalent of User Stories) are
to Test Case available, and well before any code is written.
Using Use Cases or User Stories to generate Test Cases can
help significantly to launch the test process quickly and early
in the development lifecycle. Use Cases identify and
communicate the conditions (however simple) that will be
implemented and that are necessary to verify the successful
and acceptable implementation of the systems requirements.
This gives you a lot of credit, because you do have something
to show quickly.
For agility purposes Use Cases and Test Cases should be closely linked. By doing so, the
software development and the test development remain synchronized if changes to the
requirements specifications occur.
Next we look at TestFrame Action words and how they fit in. A Test Case does include
Action Words
several steps, called ‘actions’ in TestFrame. These steps are documented in a structured and
reusable way by previously defined Action Words. An Action Word indicates which action
must be carried out for completing the particular step. Along with the Action Arguments test
data can be passed like parameters to carry out the Action. This is a very powerful mechanism.
It is good practice to name test case actions as an order, for example: ‘Enter client data’,
‘Check client data’. This prepares for the next step: identifying and introducing action words.
Action words can be executed manually but are also an important step towards Test
Automation. TestFrame uses Action Words to ensure smooth transition to Test Automation.
There are, however, limitations to using Use Cases for Test Case development. Use Cases are
Non-Functional used to model functional requirements. They are not used to model non-functional
requirements such as capacity and performance. In some cases however non-functional
requirements such as response times could be included in the Test Case.
A simple and effective way to start with Test Case design is to derive Test Cases directly from
Start with Use Cases. The Test Cases should cover the normal course
simple Test Cases of the Use Case, alternative courses, and the exceptions. In
almost every situation you will discover that it is sufficient
to start describing the individual steps and on appropriate
steps recording indication of Test Data requirements and
adding some expected results.
A good practice is to limit Test Cases to the essentials and
even leave out expected results if the expected results are
overly obvious. Most users with some support will
understand this format and easily grasp what is expected
from them. Additional information can be added as
required as the project progresses.
In situations where you need to do a Systems Integration
Test (verification) it is practical to add, at appropriate
steps, a reference to the document and/or paragraph where
the agreed functional detail can be found. This can speed
up both the verification process and the subsequent
addition of test conditions to the Test Case.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
07
Collaborative workshops can provide a valuable channel for speeding things up. In many
Build and refine circumstances collaborative workshops prove to be a lot more effective and efficient than
Test Cases individual meetings with people. This applies to the initial build and subsequent refinement of
using Workshops your Test Cases too. The Workshop approach can be used successfully in combination with
Joint Application Design (JAD), Joint Test Ware Design (JTD) and many other Agile
techniques.
Collaborative Test Case refinement workshops have proven to be remarkably successful in
reliably producing the desired results – usable Test Cases to begin testing with.
Collaborative Test Case refinement workshops will:
• Build trust among the workshop participants; business, development and test
• Commit users to the Test Case creation process
• Create a sense of ownership for deliverables, and ultimately the system, amongst users
• Discover both forgotten and non-essential requirements
• Shorten the Test Ware creation phase substantially
• Reinforce effective communication patterns
• Introduce an element of fun into a very serious process
Another proven way to refine your Test Cases is to use defects found. Look regularly at the
Refine Test Cases defects that have been detected. Where did they come from? How could Test Ware be
with defects found ammended to reveal similar problems in the future more quickly? Using this insight to tighten
up your Test Cases will make life easier. Often there is a part of the system where flaws seem
to cluster. Use this as an indication to do some thorough checking on that area.

Tracking Product Tracking product quality development is one of many features of TestFrame that can be used
to support Agile Testing. In most cases your Stakeholders are not interested in the exact
Quality Development
amount of defects found in the software. They are interested in the continuous build-up of
quality in the system they have sponsored. TestFrame can provide this even in the most agile
environments by tracking the quality per Use Case by simply (hyper)linking simple Test Cases
(as described above) into the PQM sheet at the row named TestFrame ‘Test Condition’.

Advantages of a lightweight version of the TestFrame methodology include:

• It is adaptable to change.
• It is people and expert user oriented rather than process oriented.
• It can be used to jumpstart, scale-up and formalise as the testing progresses and the
circumstances demand.
If you work in a SCRUM environment, it is a good practice to align Test Runs with SCRUM
Sprints. In doing this, you use the projects rhythm and naturally align to development cycles
and timing.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 08
5 Agile Test Ware
One of the most popular questions you come across about Agile Testing is: how much test
ware and documentation do I need? Every artefact that you create, and then decide to keep,
will need to be maintained over time, being a real burden if it is over-done or redundant for the
TOC for a document purpose. You need to consider the likelihood and impact of change on your test documentation
and anticipate accordingly. For example, the details of Use Cases and accordingly the details
of Test Cases are the most likely to change in the early iterations of the project. The burden of
maintenance therefore will be high in the beginning and lessen towards the final delivery of
the project. The decision to use simple Test Cases in the beginning will make you more agile
as you are travelling lighter. The more complex or detailed your test cases are, the more likely
it is that any given change will be harder to accomplish. You may sometimes even have to
down grade to a more simple form of test ware. This practice is particularly important for
gaining agility of Test Ware.
In most Agile projects you should consider creating at least a minimum set of Test Ware
because of:

Reasons to document Stakeholders require it - One increasingly important reason to document is the business need
for compliance. You need to work closely with your stakeholders to determine what they
actually need and more importantly, why do they need it? There may be an alternative that
will use less effort, or the documentation required for several stakeholders can be combined.
You should discuss the form in which specific documented proof is required.
Regulatory Compliancy - this may be driven by Sarbanes Oxley (SOX), BASEL-II, SAS70
or any other external compliancy obligation, as well as compliancy with internal agreed
standards. In these situations you need evidence that a defined process has been followed
correctly. You still want to create adequate and just good enough – and not more than this -
documentation to get the job done; this time including evidence for proof of compliance. A
sound piece of advice is to read the appropriate guidelines yourself, because they rarely
require what bureaucrats think they require. Be smart about compliance, often there is an
alternative that will cost much less in effort. Maybe the compliance officer will accept
something like a photograph of a sketch on a whiteboard as proof. Sometimes a digital photo
proves more to an auditor than an unsigned report using a lengthy and formal template issued
by means of an unsigned e-mail to the project manager.
Interfaces - It is important to define how systems interact with one another, because the
interfaces between them are hand-over points of responsibility. Interfaces are a contract that
both system owners should mutually agree to, document, and change over time if required.
Support of communication - It is not always possible to locate the test team, the development
team and the project stakeholders at the same place. Agile projects in particular need to find
ways to communicate Test Ware. However shared documentation is often part of the solution,
but it easily leads to misunderstandings, although, it is a good supporting mechanism. Always
combine written documentation with occasional face to face discussions, teleconferencing,
email and collaborative tools.
Knowledge Management - Another important reason to document is to record knowledge
(organizational memory) so passing on lessons learned and best practice to other projects and
personnel.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
09
Make the next step possible - Future business developments may require the testing of the
next major release of your system or to support operations and adaptive maintenance of the
version under development. The production of appropriate Test Ware at the start will make
both of these tasks easier. It could also be to facilitate the set up for automated regression
testing in the next iteration. To enable this, you will want to create just good enough
(adequate) quality test ware documentation and supporting materials so that the next step can
be effective. You should also consider whether the members of your existing team will be
involved in the next effort, the nature of the next phase of the system’s evolution, the nature of
that phase and its importance to the organization.
Compliancy Audits Auditing purposes - One more reason to document is for audit purposes, e.g. proof of
existence and that you have complied with mandatory standards. If you cannot prove it, it does
not exist. It is of high importance to keep in mind that you can view compliance in two
distinctive different ways as H. Gelinck points out in “The Science of Compliance” [Gelinck
2007]. The first way is to view compliance as a burden you have to take on. The other way
around is to view compliance rules as a way forward to prove and improve operational
excellence. To cash in on these, you should talk with the compliance officer and work with
him on creative and agile ways to create proof of compliance.
Test Ware to consider 2
What to document Test Test Plan – A test plan is required to communicate the test approach and get approval
management for it from the team and the sponsor. Agile projects often can do with one or two pages.
Test Environment specification – At a minimum record the differences between the
Agile development environment and the likely life operating environment. In Agile projects
this would fit on one single page.
Test On Agile projects this documents are mostly combined with development documents,
specification such as the SCRUM Product Backlog.
Test Design - identifies features to be tested, the test cases to test them and the
pass/fail criteria.
Test Case - documents the test cases in detail with the actual values used for input and
the anticipated outputs. This is considered a must whatever the methodology.
Test Scenario - describes the business operational actions. A useful tool to have for
Business Acceptance Testing.
Test Procedure specification - lists all steps required to operate the system and
exercise the specified test cases. If you need it, make it.
Test reporting Test item transmittal report - lists the test items being put into the testing environment.
This report only has its value when you do not have a source control system (SCS).
Otherwise rely on configuration management or a SCS.
Test log - used by the test team to keep track of what occurred during test execution.
This may be temporary and does not require maintenance after the test job is completed.
Agile projects may use a test log on a dedicated whiteboard.
Test incident report – report of findings during the test execution. The main purpose of
this report is to communicate issues found and enabling the programmer to resolve the
issue quickly. Anything that supports the findings should be included. In most cases
screenshots prove to be of great value.
Test summary report - summarizes the testing activities, matches outcomes to exit and
acceptance criteria and provides input to the management decision to proceed to go live.
This report proves particularly valuable if you can extract metrics from it.

2
See also: IEEE Std 829 Standard for Software Test Documentation [IEEE 829]

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 10
6 RRBT, TestFrame and Agile lifecycles
Within the Agile Community it is common to see teams mix best practices of different agile
methodologies, for example SCRUM and XP. As for the tester, the most important things to
be aware of in any agile project environment are its iterative lifecycle and the need for
frequent communication and feedback.
This requires some adjustments on the part of testers:
• Do not expect to base your testing on a requirements document signed off at the
beginning of the project. Instead, collaborate with other team members to find out what
should be tested. You can decide on the how
• Plan testing and set goals per Agile iteration only, but within the context of the overall
project budget and timeframe
• Get used to testing constantly within each iteration - rather than at the end of a
development activity
• Decide what and how to test when a product is still unfinished and before development
start
• Wait to automate acceptance testing till functionality is almost stable. Component testing
should be automated before code is written.
• Collaborate with other team members to find out what to test, rather than testing from
requirements documents only
• Participate actively in daily short status meetings and do not restrict your input to test
related issues.
This applies to all of RRBT/TestFrame phases.

RRBT/TestFrame Phases
1. Preparation Test Analysis 3. Test execution
Preparation TestManagement file Test Case execution
Standards and guidelines Regression testing
Definition of Test clusters and Prioritisation Test Runs and PCM
Reporting quality development
Issue management
2. Test design and Test Analysis 4. Transfer to maintenance
Test Techniques Regression test suite
Test cluster analysis

The role of the Tester The role of the tester in an Agile project is to add value by thinking about the system from the
viewpoint of those who are going to use it but with a grasp of details, possibilities, and
constraints of those building it.
A Tester:
• Brings a unique combination of business and technical viewpoints and specialized skills
• Supports risk management through direction and prioritisation of testing and provides
checks and priority schemes using RRBT
• Is more efficient at finding defects, because they focus on it
• Finds different types of defects than developers do
• Provides a critical view of the system, helping the team avoid complacency and remain
focussed on delivering business value
• Facilitates communication both inside and outside the team
The tester role and testing functions must be integrated with the requirements and
development teams. An isolated testing function in which testers are involved only at the end
of the development phase will not work.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
11
Both RRBT TestManagement and TestFrame can be successfully integrated with Agile
Software Development Methodologies, Scrum and eXtreme Programming (XP) being the
most popular.
Combining the best Scrum is an Agile software project management methodology aimed at maximizing the
business value of the product under development at every stage of that development. XP is a
software engineering methodology focusing on developing high-quality code and software
systems. XP it self doesn't have management practices.
Scrum and XP overlap at the planning game (XP) and Sprit planning (Scrum). Both encourage
similar values and provide complementary practices and rules and support increased
communication between project management and developers. Combined, they can provide a
structure to boost both the productivity of a team and the quality of its outputs.
RRBT TestManagement complements Scrum by delivering an appropriate, minimal Test
Strategy establishing development and test priorities based on both Business Risks and
Business Value. Progress reporting is based on risks mitigated and requirements realized.
RRBT also provides Test Goals for each Scrum Cycle. TestFrame complements XP by adding
Test Case development practices, specialized Test Analysis Techniques, reporting on product
quality development and Test Automation with TestFrame Integrator. TestFrame Action Word
Testing supports XP with full automation capabilities of both the functional and technical
tests.
Scrum Project Scrum project management is distinguished by rapid response to changing requirements,
management controls frequent feedback to the customer and the integration of the customer into the project.
supported by RRBT Scrum radically differs from traditional management because the planned product (backlog,
risk) can be changed at the end of any sprint (iteration), in response to the environment
(competition, changing requirements, new insights, new technology, etc.). This ensures that
the latest delivered product is always a usable and valuable product to the business.
A Scrum guided software project is controlled by establishing, maintaining, and monitoring
key control parameters. These controls are critical to contain areas of uncertainty,
unpredictable behaviour, and ultimately avoid the collapse of the team into chaos. Use of these
controls is the backbone of the Scrum development process.
Controls used in Scrum:
• Risks - the risk associated with a problem, issue, or backlog item
• Backlog – a list of all requirements and features that should be available in the completed
product
• Issues - Concerns that must be resolved prior to the release of a feature or a backlog item.
• Changes - the activities that are performed to resolve a problem or issue
• Components - self-contained reusable pieces of software
• Packet or Feature Group realisation - a group of software components that will
implement a feature or backlog item.

The main Scrum controls are Risk and Backlog (Requirements). RRBT TestManagement
complements and strengthens these controls with its Risk & Requirement based approach.
Also RRBT Test Monitoring and Test Progress and Quality Reports can be a substantial
support for Scrum projects.
Another important characteristic of Scrum is the constant testing and appropriate
documentation of a product-as it is built. This enables the team to demonstrate the tested
product increment to the customer at the end of each Sprint (Iteration) and getting valuable
feedback. This practice also builds confidence and trust across the team and between team and
sponsors.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

 12
RRBT and TestFrame SCRUM Initial planning phase is supported by
inside SCRUM • TestFrame Test Clusters enabling the testing of Scrum Feature Groups in line with
the Test Strategy
• RRBT risk analysis and the identification of relationships between features, risk and
quality
SCRUM Planning game is supported by
• Alignment of RRBT/TestFrame test clusters to the most valuable Backlog items
• RRBT continuous reassessment of risks
• RRBT/TestFrame reporting on risks mitigated , test accomplished and ultimately
product fit for going Live.

How does TestFrame phasing fit in?

SCRUM RRBT/TestFrame
1. Product Backlog Risk Analysis, Test preparation & Cluster analysis (Release planning)

2. Sprint backlog Cluster Analysis (individual User stories) and choice of appropriate test
techniques for cluster

3. Backlog tasks Initial Test Case development

4. SCRUM Cycle Test case execution, Report on Product Quality, Test Case refinement,
Test Runs and Regression testing

5. Product increment Test evaluation, Transfer to maintenance

TestFrame inside XP When doing Acceptance tests on traditional software projects it is common to find technical
faults at unit or integration level because of code that has never been tested at the unit or
integration level. This is a very expensive and inefficient way to find faults that were probably
introduced at quite an early stage of the project and can easily consume the time allocated for
system and acceptance testing. XP avoids this scenario by one hundred percent component test
automation and continuous integration practices.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
13
Therefore XP discerns two distinctive levels of testing:
Customer and Customer level: With the aid of the test members of the team, users develop a minimum set
Acceptance of acceptance tests that assess the overall behaviour of the system. This activity corresponds to
the ISTQB test level “Acceptance test’ (UAT and BAT). For testing at this level, using light
versions of TestFrame through to TestFrame Automation using TestFrame Integrator prove of
great value. Whilst developing adoptions are accommodated, after acceptance by the customer
the TestFrame light version of a Test Case can be quickly automated.
Programmer level: Component tests should be automated before component code is written.
This activity corresponds to ISTQB test level ‘component test and component integration test’.
Often component tests are created by using pair programming. One programmer concentrates
on developing the component test (for example in JUnit) while the other developer codes the
software component.
Testers Role The Testers Role in XP is to define, automate (if appropriate), and run acceptance tests.
When someone is focused on that role it allows focus in other areas as well. The XP Tester
also supports the whole team with his test expertise (for example specialized Test techniques).

How does TestFrame phasing fit in?

XP RRBT/TestFrame
1. Release Planning Risk Analysis, Test preparation & Cluster analysis (Release planning). The
Tester negotiates quality with the customer and helps to Estimate
acceptance-test time.

2. User Stories Planning Cluster Analysis (individual User stories) and choice of appropriate test
techniques for cluster testing.

3. User Stories Creation Initial Test Case development, create effective tests before coding begins.
Identify and make explicit hidden assumptions in the User Stories. The
Tester helps to clarify the proposed User Stories and adds Variants and
exceptions to it

4. Acceptance Tests Test case execution, Report on Quality, Test Case refinement, Test Runs
and Regression testing. The XP Tester makes sure acceptance tests meet
the quality negotiated with and specified by the customer.
This includes: functional, load, stress, performance, compatibility, security,
and anything not covered by unit and integration tests. Once an acceptance
test passes, it should never fail on re-tests.

5. Small Releases Test evaluation and Transfer to maintenance

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

14
7 Conclusion
It is clear to see that both RRBT TestManagement and TestFrame can be integrated to and add
enhance projects run using Agile methodologies, techniques and practices. RRBT
TestManagement and TestFrame are robust and scalable, thus addressing the needs of both plan-
driven and agile methods, as well as being adaptable to a wide range of environments and
situations.
RRBT adds strategy and reporting on risks. TestFrame supports Agile with Acceptance testing
and Product Quality Monitoring.
The combination of Scrum, RRBT TestManagement, XP and TestFrame provides the ‘just
enough’ structure, process and control that is a key pre-requisite for Agile success.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

 Combining Agile, RRBT & TestFrame
15
A Glossary of terms
Action word - a combination of action on the test object, defined by the test analyst, which
describes how test rules must be carried out (definition from TF course).
Agility – Agility is the ability to both create and respond to change in order to profit in a
turbulent business environment. Agility is the ability to balance flexibility and stability.
(Highsmith 2002)
Continuous integration - additions and modifications to the code are integrated into the build
system on at least a daily basis. Automated unit regression tests are used to ensure additions or
modifications do not introduce new defects to previously accepted code.
Refactoring – Removing duplication and needless complexity from the code during each
coding session without changing functionality, even when this requires modifying components
that are already complete and accepted by the user. Automated tests should be used to verify
that change does not break functionality.
SCRUM sprint - A time period (usually 2 to 4 weeks) in which development occurs on a set
of stories that the Agile team has committed to.
Test Case - a set of input values, execution preconditions, expected results and execution post
conditions, developed for a particular objective or test condition, such as to exercise a
particular program path or to verify compliance with a specific requirement. [ISTQB, after
IEEE 610]
Test Driven Design (TDD) - technique that involves repeatedly first writing a test case and
then implementing only the code necessary to pass the test.

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.

16
B Recommended reading
[Augano 2004] Managing Agile Projects, Kevin Augano, PMP, MAPM, The Project
Management Essentials Library, ISBN 1-895186-11-0
[Beck 2000] Beck, Kent, Extreme Programming Explained: Embrace Change
ISBN 0201616416
[Beck 2003] Beck, Kent, Test Driven Development: By Example,
ISBN 0321-14653-0
[Buwalda 2001] Integrated Test Design and Automation, Hans Buwalda, Dennis Janssen, Iris
Pinkster, ISBN 0-201-73725-6
[Cohn 2005] Agile Estimating and Planning, Mike Cohn, ISBN 0131479415
[Cohn 2004] User Stories Applied: For Agile Software Development, Mike Cohn , ISBN
0321205685
[Crispin 2002] Testing Extreme Programming, Lisa Crispin, ISBN : 0-321-11355-1
[Fowler ] Fowler, Martin, Refactoring: Improving the Design of Existing Code, ISBN 0-201-
48567-2
[Gelinck 2007] Gelinck, Henriette, The Science of Compliance, ISBN 90-78907860302-x
[Gottesdiener 2002] Requirements by Collaboration, Workshops for Defining Needs, Ellen
Gottesdiener, ISBN 0-201-78606-0
[IEEE 829] IEEE Std 829, IEEE Standard for Software Test Documentation.
[Koskela 2008] Test Driven, Practical TDD and Acceptance TDD, L. Koskela, ISBN 1-
932394-85-0
[Kroll 2006] Agility and Discipline Made Easy: Practices from OpenUP and RUP Kroll,
Bruce MacIsaac, ISBN 978-0-321-32130-5
[Pinkster 2006] Successful Testmanagement an integral approach, (2nd print 2006) Iris
Pinkster, Bob van der Burgt, Dennis Janssen, Erik van Veenendaal ISBN: 3-540-
22822-5
[Schwaber 2002], Agile Software Development with SCRUM Ken Schwaber, Mike Beedle ,
ISBN 0130676349
[Wiegers 2003] Software Requirements, Second Edition, Karl E. Wiegers, ISBN:0735618798
[Wiegers 2006] More about Software Requirements—Thorny Issues and Practical Advice,
Karl E. Wiegers, ISBN 0-73562-267-1
[Yourdon 2004] Death March, second edition, Edward Yourdon,
ISBN 0-13-143635-x

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved

LogicaCMG Nederland BV
Kralingseweg 241-249
3062 CE Rotterdam
T: +31 (0)10 253 7000
F: +31 (0)10 253 7001
www.logicacmg.com

© Peter Kalmijn, LogicaCMG, 2007. All rights reserved.