March 31 – April 2, 2009

The 5% Challenges
of Test Automation
Hans Buwalda, CTO, LogiGear Corporation
 Scope of this Workshop

• Raising the bar for automation

– the 5% challenges and their rationale

• Show techniques, concepts, ideas to

achieve the higher bar
– how could you achieve the challenges

• How feasible is all of this in practice . . .

© 2009 LogiGear Corporation. All Rights Reserved

 Testing Under Pressure
DEADLINE

specification development testing

© 2009 LogiGear Corporation. All Rights Reserved

 Testing Under Pressure
DEADLINE

specification development testing

Develop tests in time:

• Test design
• Auditing, acceptance
• Preparations
• Automation

© 2009 LogiGear Corporation. All Rights Reserved

 The 5% Challenges

• No more than 5% of all test cases should

be executed manually

• No more than 5% of all efforts around

testing should involve automating the tests

© 2009 LogiGear Corporation. All Rights Reserved

 Why a High Automation Degree ??

• The best way to prepare for efficiency in the crunch zone

– good manual test cases can help too, but marginally
• Buy time to do more "exploratory testing"
– valuable addition to testing with test cases, since it is likely to find other
and unexpected bugs
• Credible pay-off for the cost of introduction of automation
– initial costs are: tooling, learning curve, adaptation of existing tests
• Automation is better positioned to identify “bonus bugs”
– on average 15% of fixes cause new bugs
– many of these bugs are hard to find without integral testing
• often a result of violating overall architectures
• the bugs occur because data is left in inconsistent state
• Automated test cases have a better chance of being kept up to date
if they form the majority of the testware

© 2009 LogiGear Corporation. All Rights Reserved

 Why < 5% Automation Efforts ??

• Automation should not dominate testing automation

should deliver,
– it is not a goal in itself not dominate…
– may never be a bottleneck

• Testers should be able to focus on testing

– better tests (higher ambition level)
– communication with stake holders

• High automation efforts can aggravate the “crunch

zone”, instead of relieving it
– “invitation to Murphy’s law”

© 2009 LogiGear Corporation. All Rights Reserved

 Vision on Automation
• Have a good method
– integrated method for testing and automation
– testers in the driver seat
– maximum flexibility, best technology
– minimal technical efforts
– maximum maintainability
• Have the right tool
– specifically designed for keyword based testing
– flexible enough to allow for the right mix of high and low level testing
– allow the testers to quickly build keyword tests, without hassle
– is not overly complicated in the automation
• Succeed in the three "success factors for automation"
– test design
– automation solution
– organization

© 2009 LogiGear Corporation. All Rights Reserved

 Base Requirements

• Test development and automation must be fully separated

– testers are not / should not be programmers
– the automation people must implement the action words, and test them

• Test cases must have a clear and differentiated scope

– and not deviate from it

• The tests must be written in the right level of abstraction

– for example higher business level, lower UI level, or both
– the tooling must provide this flexibility

• The method must be simple, not add to the complexity of the

automation
– easy straightforward interpretation of tests, with minimal programming

© 2009 LogiGear Corporation. All Rights Reserved

 Approaches to Automation

1. Record and Playback

2. Scripting / programming

3. Keywords

it is my believe that an approach based on

keywords is a prerequisite to achieve a high
level of stable automation

(simplified, see also: "Software Test Automation", Mark Fewster and Dorothy Graham)
© 2009 LogiGear Corporation. All Rights Reserved
 Keywords, Action Based Testing
number name quantity
new product P-9009 Sledge Hammer 5

number quantity
add quantity P-9009 20
add quantity P-9009 3
add quantity P-9009 6

number quantity
check quantity P-9009 34

function action_AddQuantity
....
end
…
add quantity function action_CheckQuantity
check quantity ....
new product end
…
function action_NewProduct
....
end
© 2009 LogiGear Corporation. All Rights Reserved
 Keywords, Action Based Testing

• Describe test cases as a series of actions with

keywords and arguments, in a sheet like format
• Externalize those descriptions out of the
scripting
• Let the testers (not the automation engineer)
define the keywords, as by-product of the test
design
• Organize the test cases into "test modules",
each with a clear and differentiated scope
• Focus the automation efforts on the actions, not
on the test cases
© 2009 LogiGear Corporation. All Rights Reserved
 Example of a Test Module

test objective
test objective

© 2009 LogiGear Corporation. All Rights Reserved

 Example of a Test Module, "low level"

TEST MODULE Product Maintenance

VERSION 1.0
...

TEST CASE Test Case 03 Use an invalid number

test objective
requirement add 003 A number must be divisable by 9

source control destination

click main new new product

window control value

enter new product number P9001

source control
click new product process

text
check message Invalid product number.
© 2009 LogiGear Corporation. All Rights Reserved
 Achieving the Challenges...

Three major success factors:

1. Test design

2. Automation architecture

3. Organization

© 2009 LogiGear Corporation. All Rights Reserved

 Success Factor 1:
Test Design

© 2009 LogiGear Corporation. All Rights Reserved

 Test Design

• Effective test breakdown (into test modules)

– make sure every test module has a clear focus
– keep different kinds and levels of tests separate
• Right level of actions
– as “high level” if possible, hiding as many details as
much as possible
– but not if the details are relevant for the test

It is my believe that test design, not automation

or “the tool”, is the deciding factor in achieving
the 5% challenges
© 2009 LogiGear Corporation. All Rights Reserved
 Breakdown: Divide and Conquer
Mission, Requirements, System Design, Use Cases,
Priorities, Planning, Budget, Stakeholders, ...

Identify test modules

• priority and planning
• what, when, who, how

Test Module 1 Test Module 2 Test Module N

Analysis Analysis Analysis

Development Development Development

...
Execution Execution Execution

Maintenance Maintenance Maintenance

© 2009 LogiGear Corporation. All Rights Reserved

 Splitting into Test Modules
• UI oriented tests
– does function key F4 work
– does listbox xyz the right values
– is the tab order correct
• Do individual functions work
– like transactions in a financial system
• Alternate paths in use cases
– like cancel a transaction
• End-to-end tests
• Simulating business processes
• Tests with specific automation needs
– like multi station tests
• Tests of non-UI functions
• High ambition level tests (aggressive tests)
© 2009 LogiGear Corporation. All Rights Reserved
 Levels of Actions

• High level actions to hide details

– to hide unneeded details
• always hunt down details
– implemented in terms of lower level actions

• Low level actions to access the

automation technology
– either built-in in a tool like TestArchitect
– or create yourself in a “harness”
• this is the only real programming you would do

© 2009 LogiGear Corporation. All Rights Reserved

 Example of “Going High Level”

© 2009 LogiGear Corporation. All Rights Reserved

 Success Factor 2:
Automation Architecture

© 2009 LogiGear Corporation. All Rights Reserved

 Automation: Implementing Actions

• In a keyword driven approach the automation focuses on

the actions:
– as opposed to programming test cases
– in general this will lead to fewer and shorter scripts etc in the
automation

• In the Action Based Testing method actions are

implemented in three ways:
– built-in in the tool, like “click”, “enter”, “check” etc
– defined in an “action definition”, using other (lower level) actions
• similar to functions in a programming language
– programmed in a programming or scripting language

© 2009 LogiGear Corporation. All Rights Reserved

 "Action Definition"

© 2009 LogiGear Corporation. All Rights Reserved

 Automation Architecture
• Pay attention to where and how to implement actions
– using the “action definitions” (or something similar) for straightforward
actions
– use programming for
• complex actions (typically involving loops)
• wrap specific technical code to access the interface
• Select the right tools and technologies
– some of the considerations are subtle
– technical choices strongly influence the ease of the automation
– make sure the technology is organized as durable/re-usable as possible
– Go for a “mix” of technologies (UI, API’s, SQL, TCP/IP, ...)
• Manage the handling of the interface (UI, API, ...)
– make sure all dialogs, pages, controls etc get meaningful “logical”
names (mapped to the actual items in the UI etc)
– let the developers help you if necessary by providing easy to identify
properties of UI elements (for example the “id” in Web apps, and
“AccessibleName” in .Net”)

© 2009 LogiGear Corporation. All Rights Reserved

 Tips for Stable Automation

• Make the system under test automation-friendly

– developers are not always inclined to do that, but it pays off
– ask to add specific property values to the GUI interface controls for
automated identification
• like “accessible name” in .Net and Java, or “id” in Web controls
• Use "active" timing
– wait based on conditions in the system under test, avoid fixed “sleeps”
• Test your automation
– develop separate tests for the actions themselves
• Use automation to identify differences between versions of the
system under test
– test in a loop if all control can still be found
• Keep an eye on the test design
– unsuccessful automation virtually always can be traced back to poor
test design

© 2009 LogiGear Corporation. All Rights Reserved

 Success Factor 3:
Organization

© 2009 LogiGear Corporation. All Rights Reserved

 Organization
• Much of the 5% success is also gained or lost in how you organize
the process
– who does test design
– who does automation
– what to outsource, what to keep in-house
• Write a plan of approach:
– scope, assumptions, risks, planning
– methods, best practices
– tools, technologies, architecture
– stake holders, including roles and processes for input and approvals
– team
– ... see also:
“QA All-Stars: Building
• Assemble the right team Your Dream Team”, Hans
– testers, lead testers Buwalda, “Better
Software” September
– automation engineer(s) 2006
– managers, ambassadors, ...
© 2009 LogiGear Corporation. All Rights Reserved
 Separation of Automation

Plan of Approach

Test
Development
Plan
Automation:
intake - design
- research
Test Module
- development
Development
- testing
- support

Test Module Focus on:

Acceptance - actions
- interface handling
- technology
- never on tests

Execution and
Follow Up

© 2009 LogiGear Corporation. All Rights Reserved

 Are the 5% rules Achievable?

• To achieve to high automation coverage tends to be

fairly straightforward
– once the actions work all tests are automated
– don’t settle for too little
– be ware of "creative" complex solutions in test design and
automation, they tend to obscure readability and maintainability
• The automation efforts need to be well managed, but
then can be kept low as well
– need of automation specialization and intelligence in the team
– a method like ABT can provide a proper framework, but is only a
start
• The key to success is: think it through
– don’t hesitate to revisit your design and automation
– make sure the tests are easy to run
© 2009 LogiGear Corporation. All Rights Reserved
 Typical Time Allocation
TEST DEVELOPMENT
efforts

AUTOMATION

time
© 2009 LogiGear Corporation. All Rights Reserved
 What Does It Take More

• The method, tooling and success-factors

mentioned earlier in this presentation
• Focus on your objectives, and monitor:
– make sure stuff gets automated
– make sure the method is applied consistently
– make sure the test designs are good
– go aggressively after "can't do" claims
• Give clear direction

© 2009 LogiGear Corporation. All Rights Reserved

 Case Study

• Large IT provider
• New version of one of their major web-
sites
• Test scope was user acceptance test
(functional acceptance)
– the users were the “business owners”

• Development was off-shore

© 2009 LogiGear Corporation. All Rights Reserved

 Case Study

• Test development was done separate from

automation
– time-line for test development: May – Oct
– time-line for automation (roughly): Jan – Feb
• All tests were reviewed and approved by
the business owners
– acceptance was finished by the end of the
test development cycle

© 2009 LogiGear Corporation. All Rights Reserved

 Test Development Plan
Nr Module Business Owner Date to BO
1 Portal Navigation, Audience Robyn Peterson 05 / 23
2 Portal Navigation, Search Ted Jones 05 / 27
3 Membership, registration Steve Shao 06 / 03
4 Portal Navigation, Category Ted Jones 06 / 08
5 Portal Navigation, Topic and Expert Ted Jones 06 / 13
6 Access Control Mike Soderfeldt 06 / 17
7 Portal Navigation, Task Ted Jones 06 / 22
8 Contact DSPP Ted Jones 06 / 27
9 Portal search Mike Soderfeldt 07 / 01
10 Membership, review and update Steve Shao 07 / 05
11 Program contact assignment Alan Lai 07 / 11
12 Company, registration Steve Shao 07 / 14
13 Catalog, view and query Robyn Peterson 07 / 19
14 Site map Ted Jones 07 / 25
15 Membership, affiliation Steve Shao 07 / 28
16 Learn about DSPP Ted Jones 08 / 01
17 Products and services Steve Shao, Robyn Peterson 08 / 08
18 What's new Ted Jones 08 / 11
19 Company, life cycle Steve Shao, Alan Lai 08 / 17
20 Specialized programs Ted Jones, Steve Shao 08 / 22
21 Customer surveys Ted Jones 08 / 29
22 Software downloads Mike Soderfeldt 09 / 01
23 Newsletters Ted Jones 09 / 06
24 Internationalization and localization Ted Jones 09 / 13
25 Membership, life cycles Steve Shao 09 / 19
26 Collaboration, forums Ted Jones 09 / 23
27 Collaboration, blogs Mike Soderfeldt 09 / 28
28 Collaboration, mailing lists Ted Jones 10 / 03
© 2009 LogiGear Corporation. All Rights Reserved
 Review Process with Stake Holders

START

Stake Holder reviews:

Test Team sends draft
- coverage
Module to Stake Holder
- correctness

no Stake Holder returns

changes needed?
notice of approval

yes

Stake Holder returns

Test Team receives and notes: Test Team marks the
processes notes - additions Module as "Final"
- corrections

END

© 2009 LogiGear Corporation. All Rights Reserved

 Case Study, Results

• All tests were developed and reviewed on schedule

– many notes and questions during test development phase

• The automation was 100% of the tests

– all actions were automated, thus automating all test modules

• The test development took an estimated 18 person

months
– one on-shore resource, two off-shore resources

• The automation took between one and two months

– focused on actions
– most time was spent in handling changes in the interface (layout
of pages etc)

© 2009 LogiGear Corporation. All Rights Reserved

 Summary
• Good to set aggressive targets for the automation:
– it costs money and efforts
– it should not get in the way of the testers making good tests

• Elements for success:

– test design
– automation architecture
– organization

• Focus on the method first

– the “tool” is only a minor element
– use a method like Action Based Testing that is based on keywords and
addresses topics like test design

• The 5% targets can be achieved:

– automation coverage can commonly achieve, can even reach 100%
– to limit efforts (to 5% of all efforts) requires serious planning and
thinking ahead. It also does not include a learning curve for the method

© 2009 LogiGear Corporation. All Rights Reserved