Lecture 6 - Test Design Techniques

DITP 3263: SOFTWARE
VERIFICATION &
VALIDATION
Lecture 6: Test Design Techniques
Overview
Design techniques
Black Box techniques
White Box Techniques
Experienced Based Techniques
Factors in Choosing Techniques
Fakulti Teknologi Maklumat dan Komunikasi
Terminology
Test case
a set of test inputs, execution conditions and expected

results developed for a particular objective;
smallest entity that is always executed as a unit, from

beginning to end
A test procedure are detailed instructions for the set-up,

execution, and evaluation of results for a given test
case.
Testware are any deliverable associated with the test
process, e.g. test cases, documentation, test results.
Example of test case
3 Types of Design Techniques
Structured base (White box)

Create tests primarily by analysis of the structure of the
component or system
Attempt to identify logical errors and verify test coverage which
typically reveal bugs the way system is built
Specification based (Black-box)
Create tests primarily by analysis of the test basis
Look for bugs in the way system behaves
Experience based (attacks, checklists, exploratory)
Create tests primarily based on understanding of the system,
past experience, and educated guesses about bugs
Look for bugs in the places other system have bugs
Black box tests are specification or experience based, functional
or non functional.
White-Box
Black Box
Test Approach
Test Approach
Structural testing approach

focuses on application internals.
Program based.
Functional testing approach

focuses on application externals.
Requirements or specification
based
Characteristics
Characteristics
Module design
Implementation
Do modules/functions meet
functional and design
specifications?
Do program structures meet
functional and design
specifications?
How does program work?
Functional
Requirements, standards
Correctness
Business forms
Does system meets business
requirements?
White-Box
Black Box
Type of Test
Type of Test
Unit Testing
Integration Testing
System Testing (menu structure,
web page structure)
System Testing
User acceptance testing
Test Design Techniques
Test Design Techniques
Fault Insertion
Error Handling
String testing
Statement Coverage
Decision Coverage
Condition coverage
Path coverage
Cyclomatic Complexity
Equivalence Analysis
Boundary Analysis
Path Analysis
Use case
Decision tables
White Box Structural Based
Important of structural based : Is a best practice to measure the

extent of structural coverage of specification based and experience
based, so that you can look for important pieces of systems
structure which are not being tested.
Used system structures to derived the test cases (e.g code, design,
database tables, queries, system design..etc.)
3 Main ways to design structural:1.

To analyze control flows in the code to achieve desirable code
coverage
2.
3.
To analyze data flows using code and data structure to

achieve desirable data flow coverage
To analyze interfaces, classes, call flows by looking at the
API,;hardware, software, network system design, database
tables, integrity constrains To achieve desirable design
coverage
White Box Structural Based

Structure-based tests are based on how the system works
inside
They guarantee that all independent paths within a

module have been exercised at least once
They exercise all logical decisions on their true and

false side
They execute all loops at their boundaries and within

their operational bounds
They exercise internal data structures to ensure their

validity
White Box Fault Insertion
Forcing return codes to indicate errors and

seeing how the code behaves.
Popular method
return NULL in a function.

Erroneous input testing (valid/invalid/values outside
the boundary range of parameters)
Offers a way to measure the effectiveness of

false tests being performed.
10
White Box Error Handling
The test engineer needs to ensure that the

application returns error message properly.
Example:
Common Object Request Broker Architecture

[COBRA] user exception error has little value to
the end user
Error handling tests seek to uncover instances

where the system application does not return a
descriptive error message, but instead crashes
and reports run time error.
11
White Box-String Testing
Involves the examinations of a related group of

modules that constitute a software functions
interactions of the functions
It determines whether modules work successfully

to form a cohesive unit and whether the software
unit produces accurate and consistence result.
Parameter passed from function to function within

the module
Correctness in term of data type and data validity
12
White Box- Statement Coverage
The technique requires that every statement in

the program be invoked at least once.
Weakness:
Does not verify decisions (path/results) Boolean

expression but is affected for computational
E.g logical operator (AND /OR)
E.g. comparison operator (> Vs >=)
13
White Box- Decision Coverage
The technique requires that every possible

decision in the program to be invoked at least
once.
Also known as branch coverage
14
Example of code coverage

IF A>B THEN
Read p
C=A-B
Read q
IF p+q > 100 THEN
Print Large
ENDIF
IF p>50 THEN
Print pLarge
ENDIF
ELSE
C=A+B
ENDIF
READ D
IF C=D THEN
Print Error
ENDIF
15
White Box - Basic Paths and Tests
16
White Box MC Cabes

Cyclomatic Complexity
Theory Complexity increases with the increase

number of decisions in the modules
Thus, prone to more bugs
Rule of thumb for quality risk analysis to say that

we expect higher likelihood of bugs in those
parts of the system which are high complexity
17
Black Box Specification Based
Concerned on the behavior or features of the software

product : the inputs and the outputs
Synonymous with system testing & acceptance

testing. BUT can also occur in unit or integration
testing
User participant is important to black box testing as

the users are more familiar with the results that can be
expected from the business functions ; Thus, during
data generation phase, it is imperative that the end
users give as much input as possible.
Sets of inputs are derived mainly from functional

requirements/specifications
18
Black Box Equivalence

Partitioning
Inputs of a software are divided into a groups

that are expected to exhibit similar behavior.
Can be found for valid and invalid data.

Partitions can also be identified for outputs,
internal values, time related values
The techniques helps tester to identify data

(input/output) for testing
19
Example :Equivalence Partitioning

Testing problem the specification for a software
system for validating expenses claims for hotel
accommodation includes the following
requirements :
There is an upper limit of RM 70 for accommodation

claims
Any claims above RM 70 should be rejected and cause

an error message to be displayed
All expense amounts should be greater than RM 0 and

an error message should be displayed if this is not the
case
20
Example :Equivalence Partitioning

70
0
Hotel
Charge<=0
0< Hotel Charge

<=70
Hotel Charge
>70
Test Case
ID
Hotel Charge
Partition Tested
Expected
Output
50
0< hotel charge <=70
OK
-25
Hotel charge <=0
Error Message
89
Hotel charge >70
Error Message
21
Black Box : Boundary Value

Analysis
A refinement of equivalence partitioning that selects the

edges or end points of each partition for testing.
Equivalence partitioning looks for bugs in the code

that handles each equivalent class
Boundary values are members of equivalence classes

that also look for bugs in the definitions of the edges.
Can only be used when the elements of the equivalence

partition are ordered
Non functional boundaries (capacity, volume, etc.) can

be used for non functional testing too.
Can be used for test data selection

22
Example :Boundary Analysis

0
70
-1 +1
69 71
Test Case
ID
Hotel Charge
-1
OK
69
OK
70
71
Boundary Tested
Expected
Output
Error Message
70
Error Message
OK
Error Message
23
Black Box : Use Cases/Scenario

Tests
Use case is used to show the user interaction

with system, any action which of tangible value
to the user
Design various cases that reflect challenging real

world uses of product
Each flow can be mapped to one test case.
Some object oriented design methodologies

include use cases, so this can be an easy source
of tests
24
Screen Prototype
25
Use Case
26
Black Box : Decision Tables
Helps to select actions/conditions/logic
Mainly apply for business rules-complex rules.
Deciding how to process an order based on size,

stock on hand, state to which to ship, and so forth is
often in business rules
These can be shown as flow charts or as tables
The decision tables can make for instant test cases
Strength it creates combinations that might not

otherwise have been exercised during testing
27
ATM Decision Tables

Condition
Valid card
N Y
Valid Pin
Invalid PIN=3
Balance OK
Reject Card
Reenter PIN
N Y
Keep Card
N N
Reenter Request N N
Dispense Cash
Action
N N
This decision table shows

the business logic for an
ATM, Notice that the
dashes indicate conditions
that arent reached as part
of this rule. The rules are
mutually exclusive, in that
only one rule can apply at
any one moment of time
Notice that the business
logic layer is usually under
the user interface layer, so
at this point the basic
sanity checking of the
inputs should have been
done.
28
Black Box State Transition
To view the system in terms of states, transition

between states, the inputs/events that triggers
state changes(transition) and the action which
result from those transitions
Much used within embedded software industry

and technical automation in general.
The technique may also suitable for modeling a

business object having specific states or testing
screen dialogue flows.
29
Black Box State Transition
Advantage:
Reveal undefined situation
Terms:
State situations that persists, and will persists until

some event occurs
Event is an occurrence, something that concludes

almost instantly or in some fixed constant period
Action is an occurrence; the result
30
Black Box - State Transition Diagram
31
Black Box- State Transition Tables

Current State
Event[Condition]
Action
New State
Logging In
Password[Invalid]
Error
Logging In
Logging In
Password[Valid]
Open register
Waiting
Logging In
Customer
[Undefined]
[Undefined]
Logging In
Scan[Any]
[Undefined]
[Undefined]
Logging In
Pay[Any]
[Undefined]
[Undefined]
Logging In
Shift end
[Undefined]
[Undefined]
A state-transition table can represent complex state-transitions that

wont fit on the graph. (However, complexity might indicate bad
design). It can also reveal undefined situations as does this portion of
the table of the graph on the preceding page.
** Each row represents 1 test case
32
Example : Path Analysis

A person who is married and has worked for 5 years
will follow the path B1/A2/B2/A3, resulting in a salary of 7000
EXAMPLE :
Specifications
Explanation
IF worked = Y AND number of

years > 10
Decision B1, consisting of

conditions C1 and C2
THEN salary := 10,000
Action A1
ELSE salary := 5,000
Action A2
Decision B2, consisting of

condition C3
Action A3
IF married = Y
THEN salary := salary + 2,000
33

Start
1
2
B1
A2
A1
B2
A3
5
End
34
Steps to determine test paths
Coded uniquely the individual decision points, conditions and

actions (B1,B2 for decision points, and A1,A2 for actions)
Translate the algorithm into flow chart to show the test path
Create action combinations for each decision points [B1:(1,2);

(1,3)],[B2: (2,4,5);(2,6);(3,4,5);(3,6)]
Put the action combination in ascending order :
(1,2) ; (1,3) ; (2,4,5) ; (2,6); (3,4,5) ; (3,6)
Create paths by linked action combinations from the start to

the end of the algorithm
Path 1 : ( 1,2,4,5 )
Path 2 : ( 1,3,4,5 )
Path 3 : ( 1,2,6 )
Path 4 : ( 1,3,6 )
35
Test
No.
Action
Test Case
C1
C2
C3
Expected
Result
Path 1 : (1,3,4,5)
11
A1 & A3
Path 2 : (1,2,4,5)
10
A2 & A3
11
10
Path 3 : (1,2,6)
A2
Path 4 : (1,3,6)
A1
Actual
Result (Y/N)
36
Experienced Based
Common elements include:
The knowledge and experience used to derive test

cases
Can consider knowledge and experience of the

software, its usage and its environment or
Knowledge about historical and likely defects and

their distribution
Example include:
Attacks
Checklists
Exploratory
37
Experience Based Techniques
Key concepts
Reasons for writing test cases based on intuition,

experience, and knowledge
Comparing experienced based techniques with

specification based techniques
38
Experience Based Tests
Experienced based tests are based on the testers ..
Skill and intuition
Experience with similar applications
Experience with similar technologies
Rather than being pre-designed, experience based

tests are often created during test execution
Tests are frequent time-boxed (i.e brief periods of

testing focused on specific test conditions)
Example includes error guessing, bug hunting,

breaking software, based on checklists or bugs
taxonomies and exploratory testing
39
Common Experience Based

Approaches
Most professional approaches to experience based

testing do not create tests entirely during execution
May be guided by:
Checklist - descriptions of area, characteristics and features to test
Bug taxonomy hierarchy of bugs
List of attacks list of bugs that we seek
Bug hunt approach catalog of common problem
Set of test charters description of test to be done, similar to checklists
These guidelines are prepared in advance to some level of detail
Purely on-the-fly testing (ad hoc testing) is common, but usually

(ineffective) manual random testing
40
Experience Based Advantages

and Disadvantages
Disadvantages
Advantages
Gappy coverage
especially under pressure
Effective at finding bugs
Resists pesticide paradox

due to high variance
Difficult to estimate
Efficient (light weight

record keeping)
No bug prevention
Extensive debriefing and

discussion doesnt scale
to large teams
Not all testers have the

necessary level of skill,
experience
Good check on prepared

tests
Fun and creative
41
Factors in Choosing Techniques
Type of system
Regulatory standards
Customer or contractual requirements
Level and type of risk
Test objectives
Documentation available
Knowledge of testers
Time and budget
Development life cycle
Previous experiences on type of defects found

42
The Dynamic/Pre-designed
Spectrum
No documentation; often
associated with amateur test
efforts.
Pros: Cheap and quick
Cons : No coverage
measurement
Pure Exploratory
Charted
Exploratory
Lightweight documentation; a
sophisticated technique
Pros: Inexpensive but
measureable
Cons: Requires skilled testers
and managers
Used for highly regulated

industries; follows specific
templates
Pros: Traceable and auditable
Cons: High initial and on going
costs; not always followed
Many test teams balance

precision and detail here,
IEEE 829 Style
getting adequate
reproducibility and
traceability within realistic
budgets
Pure Scripted
Standard documentation; a
widely described technique
Pros: Reproducible and
traceable
Cons: Significant resources to
develop and maintain
43
Thank You

Lecture 6 - Test Design Techniques

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 6 - Test Design Techniques

Uploaded by

Copyright:

Available Formats

DITP 3263: SOFTWARE

Lecture 6: Test Design Techniques

Black Box techniques

White Box Techniques

Experienced Based Techniques

Factors in Choosing Techniques

Fakulti Teknologi Maklumat dan Komunikasi

a set of test inputs, execution conditions and expected

smallest entity that is always executed as a unit, from

A test procedure are detailed instructions for the set-up,

Fakulti Teknologi Maklumat dan Komunikasi

Example of test case

Fakulti Teknologi Maklumat dan Komunikasi

3 Types of Design Techniques

Structured base (White box)

Structural testing approach

Functional testing approach

Fakulti Teknologi Maklumat dan Komunikasi

Test Design Techniques

Test Design Techniques

Fakulti Teknologi Maklumat dan Komunikasi

White Box Structural Based

Important of structural based : Is a best practice to measure the

3 Main ways to design structural:1.

To analyze data flows using code and data structure to

White Box Structural Based

They guarantee that all independent paths within a

They exercise all logical decisions on their true and

They execute all loops at their boundaries and within

They exercise internal data structures to ensure their

Fakulti Teknologi Maklumat dan Komunikasi

White Box Fault Insertion

Forcing return codes to indicate errors and

return NULL in a function.

Offers a way to measure the effectiveness of

Fakulti Teknologi Maklumat dan Komunikasi

White Box Error Handling

The test engineer needs to ensure that the

Common Object Request Broker Architecture

Error handling tests seek to uncover instances

White Box-String Testing

Involves the examinations of a related group of

It determines whether modules work successfully

Parameter passed from function to function within

Correctness in term of data type and data validity

Fakulti Teknologi Maklumat dan Komunikasi

White Box- Statement Coverage

The technique requires that every statement in

Does not verify decisions (path/results) Boolean

E.g logical operator (AND /OR)

E.g. comparison operator (> Vs >=)

Fakulti Teknologi Maklumat dan Komunikasi

White Box- Decision Coverage

The technique requires that every possible

Also known as branch coverage

Fakulti Teknologi Maklumat dan Komunikasi

Example of code coverage

Fakulti Teknologi Maklumat dan Komunikasi

White Box - Basic Paths and Tests

Fakulti Teknologi Maklumat dan Komunikasi

White Box MC Cabes

Theory Complexity increases with the increase

Thus, prone to more bugs