Model-based Testing: Taking BDD/ATDD to the Next Level

Robert V. Binder System Verification Associates rvbinder@sysverif.com #MoreModelsLessTests Chicago Quality Assurance Association February 25, 2014

The Incredible Vastness of Testing

Eaten Alive by the Testing Backblob

Tell Me A Story

ModelBased Testing

More Models, Less Tests

Model Driven Development: Taking BDD/ATDD to the Next Level

The Incredible Vastness of Testing

THE INCREDIBLE VASTNESS OF TESTING

Saved by Revealing Sub-domains

Grains of sand to fill the Grand Canyon
~ 1.68 1025 ~ 10 septillion

Trivial Program
Data space: 21024 Sequences: 21024 Execution space: 22048 1.34 10616

Large subsets of any programs execution space are revealing: if a bug exists in any member of that space, the space is reached it will trigger an observable failure.
But, we rarely know exactly which sub-domains are revealing until we test.
Model Driven Development: Taking BDD/ATDD to the Next Level 4

Its release time. Do you know where your bugs are?

Aerospace 0.7 0.8 1.9 1.8 4.4 5.0 Other Agile

Medical
IT Mobile Best In Class

Average bugs per thousand equivalent source code lines, post-release

6.0 6.0

4.7
Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .
Model Driven Development: Taking BDD/ATDD to the Next Level 5

Its release time. Do you know where your bugs are?

325 375 925 875 Other Agile

Aerospace Medical IT Best In Class

Predicted total latent bugs, 500 KLOC Java

2511 2202
2350

Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .

Model Driven Development: Taking BDD/ATDD to the Next Level

Driven to Distraction
Test Driven Development
TDD

What is it?

Behavior Driven Development

BDD

How does it work?

What can it do for me?

Acceptance Test Driven Development

ATDD

Model-based Testing
MBT
Model Driven Development: Taking BDD/ATDD to the Next Level 7

Eaten Alive by the Testing Backblob

EATEN ALIVE BY THE TESTING BACKBLOB

Where does the time go?

Total Number of Test Cases Available Dev/Test Time Sprint 1 Sprint 2 Sprint 3 Sprint 4

Manual Test Cases not executed Automated Test Cases not maintained
Model Driven Development: Taking BDD/ATDD to the Next Level 9

The Attack of the BackBlob

Tell Me a Story

TELL ME A STORY

The Chat Server

Logon List active users Post a message Broadcast message Logoff
Client Endpoint

!!!
Client Endpoint Chat Server
13

Model Driven Development: Taking BDD/ATDD to the Next Level

TDD BDD ATDD

Test Driven Development Behavior Driven Development Acceptance Test Driven Development Programming strategies
Test suites are scaffolding
Test each method/feature at least once Rerun after any add/change

Not testing strategies

14

Model Driven Development: Taking BDD/ATDD to the Next Level

The TDD Red-Green Cycle

Imagine a class and its responsibilities

Code its API only

Logon Logon

GetStatus GetStatus

Logoff Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level

15

The Red-Green Cycle

Imagine a class and its responsibilities

Code its API only

Write and run a simple test it should fail

TestLogon Logon

GetStatus

Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level

16

The Red-Green Cycle

Imagine a class and its responsibilities

Code its API only

Write and run a simple test it should fail Code implementation Rerun all tests expect pass

TestLogon Logon

GetStatus

Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level

17

The Red-Green Cycle

Imagine a class and its responsibilities

Code its API only

Write and run a simple test it should fail Code implementation Rerun all tests expect pass Repeat until feature complete

TestLogon Logon TestGetStatus

GetStatus

Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level

18

The Red-Green Cycle

Imagine a class and its responsibilities

Code its API only

Write and run a simple test it should fail Code implementation Rerun all tests expect pass Repeat until feature complete

TestLogon Logon TestGetStatus

GetStatus

TestLogoff

Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level

19

TDD Limitations
Process, Activities
Components, Test Objects

No explicit linkage for user/customer needs, requirements, or stories Gaps

UI automation Feature testing

Model Driven Development: Taking BDD/ATDD to the Next Level

20

Behavior Driven Development

Process, Activities
Components, Test Objects Features, Scenarios
Feature Files *DD Tool
Scenario A Scenario B

Scenario C

Test Objects *Unit

Components
Model Driven Development: Taking BDD/ATDD to the Next Level 21

Acceptance Test Driven Development

Process, Activities
Features, Scenarios UI Test Cases
Feature Files *DD Tool
Scenario A Scenario B

Scenario C

Test Objects Scripts Test Test Tooling *Unit

System Under Test
Model Driven Development: Taking BDD/ATDD to the Next Level 22

Features and Scenarios

Feature: story Scenario: instance of a story Steps: Given: pre-condition When: trigger Then: verification condition
Feature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session Scenario: Submit Good Given A connection to When I send the Login Then I should see the Credentials the server is open message "Hi Chatter" message

Model Driven Development: Taking BDD/ATDD to the Next Level

23

Overview
http://www.specflow.org
Feature Files SpecFlow

Enter features, scenarios Generates skeleton test code for each step
Extracts step names Can run immediately

Scenario A Scenario B

Scenario C

Test Objects
Nunit Visual Studio Components

Developer adds step code

Test run reports

CI integration
Model Driven Development: Taking BDD/ATDD to the Next Level 24

1. Create a Feature As a I want to So that

Model Driven Development: Taking BDD/ATDD to the Next Level

25

2. Add Scenarios and Steps

Given
Precondition Data State

When
Trigger API call, GUI action

Then
Verification conditions (expected results)
Model Driven Development: Taking BDD/ATDD to the Next Level 26

3. Add code for steps

Model Driven Development: Taking BDD/ATDD to the Next Level

27

Modelbased Testing
THE VERY MODEL OF A MODERN

What is a Model?
Focus Smaller, simpler In IT systems
Behavior Structure Content
Model Driven Development: Taking BDD/ATDD to the Next Level 29

IT Models
Process, Activities
Behavior, Structure, Content

Technology

Model Driven Development: Taking BDD/ATDD to the Next Level

30

IT Models
Notional (design) As built (documentation)

Test Models
Focus on aspects that support testing goals

Behavior
Sequence and time

Transformation
Input, output function

Verification (oracle) Coverage Risk Management

Model Driven Development: Taking BDD/ATDD to the Next Level 32

Chat Server
Chat Server

Logon
Must be first Cant be repeated Logon

GetStatus
Only after logon Returns number of calls since logon

GetStatus

Logoff

Logoff
Only after logon

Typical RFC status codes

Model Driven Development: Taking BDD/ATDD to the Next Level 33

Simple Test Model

nc = 0
System Under Test (SUT)

Logoff GetStatus Logon assert (401) assert (401) nc++ GetStatus Logoff assert (200) assert (n == nc)
34

Logon assert (200)

Logon

GetStatus

assert (401)

Logoff

Logoff assert (200) assert (200)

Model Driven Development: Taking BDD/ATDD to the Next Level

Chat Server Message Sequence

Login a 1234 Hi Chatter a Login b secret Hi Chatter b

List Users
Users: a, b !!! Client Endpoint Client Endpoint Chat Server

Model Driven Development: Taking BDD/ATDD to the Next Level

35

Chat Server Message Sequence

List Users

Users: a, b
Post Pumpkin Latte #mypics

Ack a1 Pumpkin Latte #mypics

Bcast a1 Pumpkin Latte #mypics Logoff b Ack Logoff b Client Endpoint
Model Driven Development: Taking BDD/ATDD to the Next Level

!!! Client Endpoint

Chat Server

36

Spec Explorer Demo

Model-based testing tool
Microsoft Research Used to test all MSFT APIs Robust and stable Visual Studio power tool

C# code, not cartoons Generates standalone executable test suite

Model Driven Development: Taking BDD/ATDD to the Next Level 37

1. Create Test Model Actions [Rule] tags a method for exploration Requires defines a exploration precondition must be true to explore
Model Driven Development: Taking BDD/ATDD to the Next Level 38

2. Define Accepting State(s)

Tag condition(s) that terminate an exploration

Model Driven Development: Taking BDD/ATDD to the Next Level

39

3. Create Cord File

Set options

Define machines
Limits Actions used in an exploration

Similar to regular expression

Composable
Very powerful!
Model Driven Development: Taking BDD/ATDD to the Next Level 40

4. Explore the Model Select any explorable machine

Explore Generate test suite Run on-the-fly test

Model Driven Development: Taking BDD/ATDD to the Next Level

41

5. Review Exploration Graph

Shows action/event sequence

Ellipse: model state computed by model program machine Diamond: observation event get something from SUT (input to the model program) Right click to inspect any item
42

Model Driven Development: Taking BDD/ATDD to the Next Level

6. Define machine(s) to generate test suites

Set options to generate test code Use any machine or composite machine

Model Driven Development: Taking BDD/ATDD to the Next Level

43

7. Explore the Test Suite machine

Each root to leaf path becomes a test suite

Model Driven Development: Taking BDD/ATDD to the Next Level

44

8. Generate Test Code

Explores the test machine Writes out test code using VS testing framework

Model Driven Development: Taking BDD/ATDD to the Next Level

45

9. Generate Test Code

Not intended to be maintained Re-generate any time Add helper code in the model to extend/customize
Model Driven Development: Taking BDD/ATDD to the Next Level 46

10. Run the Test Suite

Uses built-in Visual Studio test runner

Adapters
start/stop chat clients and chat server

Other SUT setup/cleanup

Test code sends messages, checks replies

Model Driven Development: Taking BDD/ATDD to the Next Level

47

11. Review Test Results

Model State (test case inputs)

Detailed run log

VS test runner States Events Requirements

Event Handler (pass/fail)

Requirement Trace

Add any other programmable function

48

Model Driven Development: Taking BDD/ATDD to the Next Level

Spec Explorer
Exploration Interactive model development Generate test suites
Machines Cord Declarations Model Program Model Data Structures Rules Actions Events Adapters

Machines define composable Action sequences Options control exploration and data generation Actions are explored
Rules define exploration conditions
Events define expected SUT response

Options

Data structures support exploration Adapters abstract, control, and observe any interface, any platform
49

Model Driven Development: Taking BDD/ATDD to the Next Level

MBT Driven Development

Requirements, Stories, Scenarios...
Explore Test Model Run
Ambiguous, missing, contradictory, incorrect, redundant, incomplete Model error, omission Missing, incorrect
Coverage Requirements Model

Generate
Test Suite

Evaluate

Inputs (Test Sequences)

Expected Outputs (Test Oracle)

Control

SUT

Observe

Bug
50

Model Driven Development: Taking BDD/ATDD to the Next Level

More Models, Less Tests

MORE MODELS, LESS TESTS

Model-based Testing in the Real World

Windows protocol compliance (Web services, RPC, DCOM) Total effort: 250 person years (mostly junior SDETs) Saved 50 person years with model-based testing
Model-based Testing 1.4 Days/Requirement Traditional Testing 2.4 Days/Requirement Requirements Study Modeling Test Coding Adapter Coding

42% Less Time Per Requirement

Test Execution
52

Model Driven Development: Taking BDD/ATDD to the Next Level

Scope of Test Asset Maintenance

Model-based Testing
Model

Behavior Driven Development

Test Code

Adapters
Model Driven Development: Taking BDD/ATDD to the Next Level 53

More Models, Less Tests

More better testing-MBT
Rule-based
Systematic and deep exploration Generate 1000s of scenarios in seconds

Icing the BackBlob

Maintain model
Regenerate test suites

Models grow much more slowly than test code

Model Driven Development: Taking BDD/ATDD to the Next Level

54

Icing the Test BackBlob

Q&A
#MoreModelsLessTests rvbinder@sysverif.com

Model Driven Development: Taking BDD/ATDD to the Next Level

56

Learn More - MBT

UCCAT (User Conference on Advanced Automated Testing) Presentations 2012 Model-based Testing User Survey Open Source MBT tools http://ucaat.etsi.org/2013/program_conf.html

http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/

Overview of MBT, list of commercial MBT tools

Spec Explorer

http://www.slideshare.net/robertvbinder/model-basedtestingignite

http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx Microsoft Open Protocol Project
Model Driven Development: Taking BDD/ATDD to the Next Level 57

http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!

Learn More: BDD, etc.

SpecFlow http://www.specflow.org/ Using SpecFlow with WatiN http://msdn.microsoft.com/en-us/magazine/gg490346.aspx SpecFlow Quick Start

http://www.youtube.com/watch?v=SVwBz-ZTliw

Writing Cucumber Feature Files

https://github.com/cucumber/cucumber/wiki

Model Driven Development: Taking BDD/ATDD to the Next Level

58

Content and Image Sources

Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0k The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/ Defect densities: Capers Jones, private communication. Robert E. Park. Software Size Measurement: A Framework for Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon University, September 1992. SpecFlow http://www.specflow.org/ Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270 Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980 Spec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9956b-d9bfa4902745 Spec Explorer Chat Model. Microsoft. The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0w

All other content Copyright Robert V. Binder, 2014.

Model Driven Development: Taking BDD/ATDD to the Next Level

59