What Every Tester Needs to Know

to Succeed in the Agile World

Jean Tabaka

SQuAD January 2007

Copyright 2007, Rally Software Development Corp

Presenter Jean Tabaka

Customer Advisor and Agile
MentorRally Software
Development
25 years in software
development government,
consulting, ISVs, large IT
departments
Large XP rollout4000
distributed people at Qwest
Communications
Certified ScrumMaster (CSM),
CSM Trainer, CSM Practicing,
Certified Professional
Facilitator
Author of Collaboration

Explained

jean.tabaka@rallydev.com
Slide 2

Copyright 2007, Rally Software Development Corp

What Well Talk About

General view of testing in an agile world

10 tenets of testing in an agile environment
10 commitments to what you will NOT do as an agile
tester

Special thanks to my Rally Software colleagues/gurus:

Bob Cotton
David Demmer
Melissa Gallegos
Alex Pukinskis

Slide 3

Copyright 2007, Rally Software Development Corp

General View of Agile Development

Why agile loves testing and testers

Copyright 2007, Rally Software Development Corp

Why The Agile Manifesto Loves Testers

Welcome changing requirements, even late in

development. Agile processes harness change for the

customer's competitive advantage.

Working software is the primary measure of

progress.

Deliver working software frequently, from a couple

of weeks to a couple of months, with a preference to

the shorter timescale.

zhttp://agilemanifesto.org/principles.html

Slide 5

Copyright 2007, Rally Software Development Corp

What Does it Look Like to be Agile?

Slide 6

Copyright 2007, Rally Software Development Corp

We Pull Testing Forward

Agile Development
Waterfall

PM
Process

Critical
Path through
Phases

Iterative

Iterative and
Incremental

Parallel

Acceptance
Test Driven

Critical
Drop
Schedule

Time Boxes

Continuous
Flow

Automated
Flow

Requirements
& Design
Process

Phase 2

Phase 2

Detailed
Design
Inside
Iteration

Just-in-time
Elaboration

Acceptance
Test Cases

Development
Process

Phase 3
All Features
in Parallel

Multiple
Drops to QA

Increment
at a time

DesignBuild-Test

Test
Design Build

Phase 4
only

Drops in
Phase 3

Acceptance
Test Inside
the Iteration

Failing Test
Cases as
Defects

By
Exception

Test & Fix

Process

Slide 7

Copyright 2007, Rally Software Development Corp

An Agile IterationWe Test and Test and Test

Slide 8

Copyright 2007, Rally Software Development Corp

That Means Acceptance Testing Too!

Demo & Retro

Slide 9

Accept

Iteration N-1

Accept

Detailed Iteration
Planning & Design

Prepare for
features
and tests

Dev Feature
Dev Feature
Priority 1
Priority 4
Auto. Tests
Auto. Tests
Feature 1
Feature 4
Dev Feature
Dev Feature
Priority 2
Priority 5
Auto. Tests
Auto. Tests
Feature 2
Feature 5
Dev Feature
Priority 3
Auto. Tests
Feature 3

Iteration N+1

Iteration N

Copyright 2007, Rally Software Development Corp

A Classic Testing Architecture

Manual GUI
Acceptance
Tests

Automated
GUI Tests

Unit
Tests

Slide 10

Easy to Create
Very familiar what we always do
Typically tedious
How do we know coverage?
Automation good for performance
Need Automation specialists
Seems like we always rewrite
Sometimes fragile
What is Dev testing?
How do we know what these are?
Dev resistance to writing tests

Copyright 2007, Rally Software Development Corp

Better, But Not Perfect

Manual GUI
Acceptance
Tests
Automated
GUI Tests
& FitNesse

Unit Tests

Slide 11

Still too many here

FitNesse drives talking tests

Increased Coverage
Features under test more stable

Copyright 2007, Rally Software Development Corp

Mike Cohns Testing PyramidThis is Agile!

Small number
Automate as much as you can
GUI
Tests

Find the right ones

FitNesse
Do these the most
Unit Tests

Slide 12

Copyright 2007, Rally Software Development Corp

The 10 Tenets of Agile Testing

What drives our behaviors as testers in
agile

Copyright 2007, Rally Software Development Corp

1. The system always runs

Frequent builds
Continuous integration
Without continuous testing, we never truly know
where we stand, how much work is left to do

Slide 14

Copyright 2007, Rally Software Development Corp

2. In agile, we stop the line versus logging

defects

If the build is broken, no one can move forward until

the build is fixed
A CruiseControl fix is THE HIGHEST PRIORITY for the
entire team
Dev team immediately fixes the failing automated
test without logging a defect

When to log a defect?

GUI autotest may warrant a defect

Manual walkthrough

Slide 15

Copyright 2007, Rally Software Development Corp

3. If its not tested, its not Done

Untested code is worthless

Untested code is excess inventory
Must have a HIGH CONFIDENCE at all times about the
shippable state of the code
Have a test plan that declares DONENESS for:

Feature
Iteration
Release

Testers therefore impact the estimate for any DONE

item

Slide 16

Copyright 2007, Rally Software Development Corp

4. Testing comes first not last

We prevent (QA) defects versus finding them (QC)

If code is defective, it is not the developers fault; it
is the fault of the system, what the management
values and will tolerate

How to bake quality in:

Slide 17

Start writing acceptance tests immediately after the Planning

meeting
Dont allow defects to accumulate
Dont allow defects to get stale

Copyright 2007, Rally Software Development Corp

5. Finding defects after DEV is done is too late

Development has already moved on to other coding

tasks
It is harder to ferret out exactly where the defect
resides if conversations cant occur immediately
Using a spec for the conversation sets the stage for
miscommunication

Slide 18

Copyright 2007, Rally Software Development Corp

6. Dev Complete is meaningless

Only Running Tested Features (RTFs) matter

RTFs are the true definition of doneness and

contribute to always shippable code

Development writes both the code and the harness
(abstracts the details as a wrapper around the code
so that the tester doesnt have to know JavaScript
and HTML)
Good reason to go to open source test tools

Slide 19

Copyright 2007, Rally Software Development Corp

7. Use testing, not analysis, to explore

requirements

Test suite is run based on how long it takes to run:

Fastest: Run fast unit tests all the time

Fast: Conduct out of DB tests every 5 minutes

Slower: Conduct in DB tests every 20 minutes

Slide 20

Slower still: Conduct smoke test, GUI, load, and performance

nightly
Slowest: Use manual walkthrough for user experience prior
to final acceptance

Copyright 2007, Rally Software Development Corp

10

8. Test automation is a how not a whether

or when

Your code cant converse with you adequately

without a rich test suite
Set up a Test Harness:

Why automate? Ron Jeffries says:

Unit tests
Automated test suites
Full regression tests

1. You get good at it

2. You learn to work in small bites
3. You learn to work in a test scaffold that helps you find
mistakes early

4. You find ways to work that dont require such long delays
before development work is tested

Slide 21

Copyright 2007, Rally Software Development Corp

9. Tests are your second most detailed spec

Next to the code, you cant get a better depiction of

what is specified, working, defective, or done

You can make your tests talk more clearly than a text
spec
FitNesse enables customers, testers, and
programmers to learn what their software should do,
and to automatically compare that to what it actually
does do. It compares customers' expectations to
actual results.

www.fitnesse.org
Slide 22

Copyright 2007, Rally Software Development Corp

11

10. Testers are customer-developer liaisons

The 3-legged agile stool defines functions and

doneness criteria
Testing helps elaborate requirements and makes
sure they are testable
Testers find holes in the requirements prior to start
of development
Test and dev sit together to elaborate the tests
Lisa Crispin The whole team approach is critical.
XP Testers Bill of Rights http://home.att.net/~lisa.crispin/XPTesterBOR.htm
Assign a team to each story to determine testing

Slide 23

For this thing, well use JUnit.

For this thing, well use a FitNesse Fixture.
For this thing, well use a Selenium GUI test.

Copyright 2007, Rally Software Development Corp

The Agile Testers 10 Commitments

What we commit to NOT doing

Copyright 2007, Rally Software Development Corp

12

Agile Testers Ten Commitments

1. We commit to not moving forward if a hole is found

through root cause analysis without first writing a

test.
2. We commit to not relying solely on just automated
testing or just manual testing.
3. We commit to not sitting behind a QA wall (no
boundaries!)
4. We commit to not allowing a code complete without
test code harness complete.
5. We commit to not waiting for a test phase but rather
working in smaller and smaller pieces, sooner and
sooner.

Slide 25

Copyright 2007, Rally Software Development Corp

Agile Testers Ten Commitments

6. We commit to not testing one iteration after

development is Done.
7. We commit to not allowing surprises to accumulate
for large end-to-end testing (mock it now).
8. We commit to not leaving the riskiest tests to the
end.
9. We commit to being an equal participant with the
customer and the developer in defining Doneness.
10.We commit to not taking this oath lightly, i.e. no
AUI!

Slide 26

Copyright 2007, Rally Software Development Corp

13

Useful References for Agile Testing

Slide 27

Beck, Kent Test-Driven Development By Example, Addison Wesley,

2003
Cohn, Mike, User Stories Applied For Agile Software Development,
Addison Wesley, 2003
Cunningham, Ward and Rick Mugridge Fit for Developing Software,
Prentice-Hall, 2005
Crispin, House, Testing Extreme Programming, Addison Wesley, 2003
Poppendieck, Implementing Lean Software Development, Addison
Wesley, 2006
Tabaka, Collaboration Explained, Addison Wesley, 2006
www.testing.com Brian Marick
www.mountaingoatsoftware.com - Mike Cohn
www.fitnesse.org
www.stickyminds.com
www.opensourcetesting.org
Agiletesting Yahoo! Group

Copyright 2007, Rally Software Development Corp

Thank you
jean.tabaka@rallydev.com

Copyright
Slide 28 2003-2005, Rally Software
Copyright
Development
2007, Rally
CorpSoftware Development Corp

14