Testing

Motivation
People are not perfect

We make errors in design and code
Goal of testing: given some code, uncover
Software Testing as many errors are possible
Important and expensive activity
Not unusual to spend 30-40% of total
project effort on testing
For critical systems (e.g. flight control): cost
can be several times the cost of all other
activities combined
CSE 757 - Software Testing 1 CSE 757 - Software Testing 2
A Way of Thinking Testing Objectives
Design and coding are creative Testing is a process of executing

software with the intent of finding
Testing is destructive
errors
The primary goal is to “break” the software
Good testing has a high probability of
Very often the same person does both
finding as-yet-undiscovered errors
coding and testing
Need “split personality”: when you start Successful testing discovers unknown
testing, become paranoid and malicious errors
Surprisingly hard to do: people don’t like If did not find any errors, need to ask
finding out that they made mistakes whether our testing approach is good
Basic Definitions Testing Approaches
Test case: specifies In this course: small sample of

Inputs + pre-test state of the software approaches for testing
Expected results (outputs and state)

White-box testing
Black-box testing: ignores the internal Control-flow-based testing
logic of the software Data-flow-based testing
Given this input, was the output correct?
Black-box testing

White-box testing: uses knowledge of Equivalence partitioning
the internal structure of the software
e.g. write tests to “cover” internal paths
1
Control-flow-based Testing Example of a Control Flow Graph
Traditional form of white-box testing s:=0;

1
d:=0;
Step 1: From the source code, create a while (x<y) {
graph describing the flow of control 2
x:=x+3;
Called the control flow graph y:=y+2;
3
The graph is created (extracted from the if (x+y < 100)
source code) manually or automatically s:=s+x+y;
4
else
Step 2: Design test cases to cover d:=d+x-y; 5 6
certain elements of this graph } 7
Nodes, edges, paths
8
Elements of a Control Flow Graph IF-THEN, IF-THEN-ELSE, SWITCH
Three kinds of nodes: if (c) if (c) switch (c)

Statement nodes: single-entry-single-exit then … then … case 1: …
sequences of statements join point else … case 2: …
Predicate (decision) nodes: conditions for join point . . .
branching join point
Auxiliary nodes: (optional) for easier
understanding (e.g. “merge points” for IF)
Edges: possible flow of control ... ... ...

...........
Example Mapping for Loops

switch (position)
case CASHIER:
if (empl_yrs > 5) while (c) {
bonus := 1;
…
...
else
}
.....
bonus := 0.7;
break;
case MANAGER:
Note: other loops (e.g. FOR,
bonus := 1.5;
if (retiring_soon) DO-WHILE, …) are mapped similarly
bonus := 1.2 * bonus
break; Mini-assignment: figure out how this is done
case ...
endswitch CSE 757 - Software Testing 11 CSE 757 - Software Testing 12
2
Statement Coverage Example
Given the control flow graph, define a Suppose that we write and
execute two test cases 1
“coverage target” and write test cases to
achieve it Test case #1: follows path 2
1-2-exit
Traditional target: statement coverage 3
Test case #2: 1-2-3-4-5-
Test cases that cover all nodes 7-8-2-3-4-5-7-8-2-exit
(loop twice, and both times T 4 F
Code that has not been executed during take the true branch)
testing is more likely to contain errors 5 6
Problem: node 6 is never 7
Often this is the “low-probability” code executed, so we don’t have
100% statement coverage
8
Branch Coverage Branch Coverage
Target: write test cases that cover all Statement coverage does not imply
branches of predicate nodes branch coverage
True and false branches of each IF
Example: if (c) then s;

The two branches corresponding to the
By executing only with c=true, we will achieve
condition of a loop
statement coverage, but not branch
All alternatives in a SWITCH coverage
In modern languages, branch coverage Motivation: experience shows that many
implies statement coverage errors occur in “decision making”
Plus, it subsumes statement coverage
Example Achieving Branch Coverage

Same example as Branch coverage: a necessary minimum
1
before: two test cases Pick a set of start-to-end paths that cover
Path 1-2-exit 2 all branches, and write tests cases to
Path 1-2-3-4-5-7-8-2-3- execute these paths
4-5-7-8-2-exit 3
Basic strategy
Problem: the “false” T 4 F Add a new path that covers at least one edge
branch of 4 is never 5 6
that is not covered by the current paths
taken - don’t have Sometimes the set of paths chosen with this
100% branch coverage 7 strategy is called the “basis set”
8 Good example in Pressman, Sect. 17.4.3

3
Data-flow-based Testing Example
Test connections between variable assume y is already initialized

1 2 3
definitions (“write”) and variable uses 1 s:=0;
(“read”) 2 x:=0; DEF(1)={s} USE(1)=∅ 4
3 while (x<y) { DEF(2)={x} USE(2)=∅
Variation of the control flow graph 4 x:=x+3; DEF(3)=∅ USE(3)={x,y} 5
A node represents a single statement, not a 5 y:=y+2; DEF(4)={x} USE(4)={x}
single-entry-single-exit chain of statements 6 if (x+y<10) DEF(5)={y} USE(5)={y} 6
7 s:=s+x+y; DEF(6)=∅ USE(6)={x,y}
Set DEF(n) contains variables that are else DEF(7)={s} USE(7)={s,x,y} 7 8
defined at node n (i.e., they are written) 8 s:=s+x-y; DEF(8)={s} USE(8)={s,x,y} 9
} DEF(9)=∅ USE(9)=∅
Set USE(n): variables that are read
DEF(10)=∅ USE(10)=∅ 10
Reaching Definitions Def-Use Pairs

A definition of x at n1 A def-use (DU) pair for variable x is a
reaches n2 if and only if 1 2 3 pair of nodes (n1,n2) such that
there is a path between n1 x is in DEF(n1)
and n2 that does not 4
the definition of x at n1 reaches n2
contain a definition of x 5 x is in USE(n2)
DEF(1)={s} USE(1)=∅
Reaches DEF(2)={x} USE(2)=∅
6 i.e., the value that is assigned to x at n1
DEF(3)=∅ USE(3)={x,y}
nodes 2, 3, DEF(4)={x} USE(4)={x}
is used at n2
7 8
4, 5, 6, 7, DEF(5)={y} USE(5)={y} Since the definition reaches n2, the value is
9 not “killed” along some path n1…n2
8, but not DEF(6)=∅ USE(6)={x,y}
DEF(7)={s} USE(7)={s,x,y}
9, 10 DEF(8)={s} USE(8)={s,x,y} 10
Example of Def-Use Pairs Data-flow-based Testing

Reaches nodes 2, 3, 4, 5, Identify all DU pairs and construct test
6, 7, 8, but not 9, 10 1 2 3 cases that cover these pairs
For this 4 Variations with different “strength”
definition: All-DU-paths: for each DU pair (n1,n2)
two DU 5
DEF(1)={s} USE(1)=∅ for x, exercise all possible paths n1…n2
pairs DEF(2)={x} USE(2)=∅ that are clear of definitions of x
1-7, 1-8 DEF(3)=∅ USE(3)={x,y} 6
DEF(4)={x} USE(4)={x} 7 8 All-uses: for each DU pair (n1,n2) for x,
DEF(5)={y} USE(5)={y} exercise at least one path n1…n2 that is
DEF(6)=∅ USE(6)={x,y} 9
DEF(7)={s} USE(7)={s,x,y} clear of definitions of x
DEF(8)={s} USE(8)={s,x,y} 10
4
Data-flow-based Testing Black-box Testing
All-definitions: for each definition, cover Unlike white-box testing: don’t use any
at least one DU pair for that definition knowledge about the internals of the code
i.e., if x is defined at n1, execute at least
Test cases are designed based on

one path n1…n2 such that x is in USE(n2) and
the path is clear of definitions of x specifications
Motivation: see the effects of using the Example: search for a value in an array
values produced by computations Postcondition: return value is the index of
some occurrence of the value, or -1 if the
Focuses on the data, while control-flow-
value does not occur in the array
based testing focuses on the control
Equivalence Partitioning Equivalence Classes
Consider input/output domains and Examples

partition them into equivalence classes Input x in range [a..b]: three classes “x<a”,
For different values from the same class, “a<=x<=b”, “b<x”
the software should behave equivalently boolean: classes “true” and “false”
Some classes may represent invalid input
Test values from each class
Example for input range 2..5: “less than 2”, Choosing test values
“between 2 and 5”, and “greater than 5” Choose a typical value in the middle of the
class(es) that represent valid input
Testing with values from different
classes is more likely to find errors than Choose values at the boundaries of classes
e.g. for [a..b], use a-1, a, a+1, b-1, b, b+1
testing with values from the same class
Example Another Example
Spec says that the code accepts between Similarly for the output: exercise
4 and 24 inputs; each is a 3-digit integer boundary values
One partition: number of inputs Spec: the output is between 3 and 6
Classes “x<4”, “4<=x<=24”, “24<x” integers, each in the range 1000-2500
Chosen values: 3,4,5,14,23,24,25

Try to design inputs that produce
Another partition: integer values 3 outputs with value 1000
Classes: “x<100”, “100<=x<=999”, “999<x” 3 outputs with value 2500
Chosen values: 99,100,101,500,998,999,1000 6 outputs with value 1000
6 outputs with value 2500
5
Example: Searching Example: Searching
Search for a value in an array Array Value Output

Return: index of some occurrence of the empty 5 -1
value, or -1 if the value does not occur
[7] 7 0
One partition: size of the array
Programmer errors are often made for size [7] 2 -1
1: a separate equivalence class [1,6,4,7,2] 1 0
Classes: “empty array”, “array with one
[1,6,4,7,2] 4 2

element”, “array with many elements”
[1,6,4,7,2] 2 4
Another partition: location of the value
“first element”, “last element”, “middle [1,6,4,7,2] 3 -1
element”, “not found”
Testing Strategies
Many issues beyond what we discussed

Who does the testing?
Which techniques should we use and when?
Testing Strategies No universal strategies
Principles that have been useful in practice
e.g. the notions of unit testing and
integration testing
Scope and Focus Unit Testing
Unit testing: scope = individual component Scope: one component from the design
Focus: component correctness Often corresponds to the notion of
White-box and black-box techniques “compilation unit” from the prog. language
Integration testing: scope = set of Responsibility of the developer
interacting components
Focus: correctness of component interactions Both white-box and black-box techniques
Mostly black-box, some white-box May be necessary to create stubs: “fake”
System testing: scope = entire system code that replaces called modules
Focus: overall system correctness If not yet implemented, or not yet tested
Only black-box techniques
6
Basic Strategy for Unit Testing Test-First Programming
Create black-box tests Modern practices: the importance of unit

Based on the specification of the unit testing during development
Evaluate the tests using white-box Example: test-first programming
techniques (test adequacy criteria) Before you start writing any code, first
How well did the tests cover statements, write the tests for this code
branches, paths, DU-pairs, etc.? Write a little test code, write the
Many possible criteria; at the very least corresponding unit code, make sure it passes
need 100% branch coverage the tests, and then repeat
Create more tests for the inadequacies: E.g., used in Extreme Programming: any
e.g. to increase coverage of DU-pairs program feature without a corresponding
test simply doesn’t exist
Advantages of Test-First Programming System Testing
Developers don’t “skip” unit testing Goal: find whether the program does
Satisfying for the programmer: feeling of what the customer expects to see
accomplishment when the tests pass Black-box
Helps clarify interface and behavior From requirements analysis, there should
before programming be validation criteria
To write tests for something, first you need How are the developers and the customers
to understand it well going to agree that the software is OK?
Software evolution Many issues: functionality, performance,
After changing existing code, rerun the tests usability, portability, etc.
to gain confidence (regression testing)
System Testing (cont) System Testing (cont)
Initial system testing is done by the For multiple customers: two phases
software producer
Alpha testing: at the vendor’s site by a
Eventually need testing by the customers few customers
Customers are great testers
Beta testing: distributed to many end-

If the software is built for a single users
customer: series of acceptance tests Users run it in their own environment
Deploy in the customer environment and have Sometimes done by thousands of users
end-users run it
7
Stress Testing Stress Testing (cont)
Form of system testing: behavior under Find how the system deals with overload
very heavy load
Reason 1: determine failure behavior
e.g. normally there are 1-2 interrupts per
If the load goes above the intended, how
second; what if there are 10 interrupts?
“gracefully” does the system fail?
e.g. what if data sets that are an order of
magnitude larger than normal? Reason 2: expose bugs that only occur
e.g. what if our server gets 10 times more under heavy loads
client requests than normally expected? Especially for OS, middleware, servers, etc.
e.g. memory leaks, incorrect resource
allocation and scheduling, race conditions
Regression Testing
Rerun old tests to see if anything was

“broken” by a change
Changes: bug fixes, module integration,
maintenance enhancements, etc. Testing of Object-Oriented Software
Need test automation tools
Load tests, execute them, check correctness
Everything has to be completely automatic
Could happen at any time: during initial

development or after deployment
Object-Oriented Software One Difference: Unit Testing
Widespread popularity in the last decade Traditional view of “unit”: a procedure

Initial hopes: it would be easier to test In OO: a method is similar to a procedure
OO software than procedural software But a method is part of a class, and is
Soon became clear that this is not true tightly coupled with other methods and
fields in the class
Some of the older testing techniques are
still useful The smallest testable unit is a class
It doesn’t make sense to test a method as a
New testing techniques are designed separate entity
specifically for OO software
Unit testing in OO = class testing
8
Class Testing Example: Inter-method DU Pairs
class A {
Traditional black-box and white-box
private int index;
techniques still apply public void m1() {
E.g. testing with boundary values index = …;
Inside each method: at least 100% branch …
coverage; also, DU-pairs inside a method m2();
}
Extension: DU pairs that cross method private void m2() { … x = index; … }
boundaries public void m3() { … z = index; … }
Example: inside method m1, field f is }
assigned a value; inside method m2, this test 1: call m1, which calls m2 and
value is read reads the value of index
test 2: call m1, and then call m3
Possible Test Suite Polymorphism

public class MainDriver {
A variable may refer to objects of
public static void main(String[] args) {
A a = new A(); different classes at different points of
… time
a.m1(); e.g., in Java, if D is a subclass of C, a
a.m3(); variable of type C may refer to instances of
… C or to instances of D
}
An operation may be defined in several
Note: need to ensure that the actual execution classes and may have different
exercises definition-free paths for the two DU implementations in each class
pairs Method overriding
Example of Polymorphism Testing of Polymorphism
Example: class A with subclasses B and C During class testing of X: “drive” call site
class A { … void m() {…} …} a.m() through all possible bindings
class B extends A { … }

All-receiver-classes: execute with at
class C extends A { … void m() {…} … } least one receiver of class A, at least one
Suppose inside class X there is call a.m(), receiver of class B, and at least one
where variable a is of type A receiver of class C
Could potentially send message m() to an All-invoked-methods: need to execute
instance of A, instance of B, or instance of C
with receivers that cover A.m and C.m
The invoked method could be A.m or C.m
i.e. (A or B receiver) and (C receiver)
9
State-based Testing Example: Stack
Natural representation with finite-state States

machines Initial: before creation
States correspond to certain values of the Empty: number of elements = 0
attributes Holding: number of elements >0, but less
Transitions correspond to methods than the max capacity
Full: number elements = max
FSM can be used as basis for testing
Final: after destruction

e.g. “drive” the class through all transitions,
and verify the response and the resulting Transitions: starting state, ending state,
state action that triggers the transition, and
possibly some response to the action
Examples of Transitions Finite State Machine for a Stack
Initial -> Empty: action = “create” Initial

e.g. “s = new Stack()” in Java
create add,delete
Empty -> Holding: action = “add”
add
Empty -> Full: action = “add” Empty Holding
delete
if max_capacity = 1 add delete
Empty -> Final: action = “destroy” add delete
destroy
e.g. destructor call in C++, garbage collection destroy
in Java Final destroy Full
Holding -> Empty: action = “delete”
FSM-based Testing Inheritance
Each valid transition should be tested People thought that inheritance will
Verify the resulting state using a state reduce the need for testing
inspector that has access to the internals of Claim 1: “If we have a well-tested superclass,
the class we can reuse its code (in subclasses, through
inheritance) without retesting inherited
Each invalid transition should be tested code”
to ensure that it is rejected and the
Claim 2: “A good-quality test suite used for a
state does not change superclass will also be good for a subclass”
e.g. Full -> Full is not allowed: we should call
add on a full stack Both claims are wrong
10
Problems with Inheritance Testing of Inheritance
Incorrect initialization of superclass Principle: inherited methods should be

attributes by the subclass retested in the context of a subclass
Missing overriding methods Example 1: if we change some method m
Typical example: equals and clone is a superclass, we need to retest m
Direct access to superclass fields from inside all subclasses that inherit it
the subclass code
Example 2: if we add or change a
Can create subtle side effects that break
unsuspecting superclass methods subclass, we need to retest all methods
inherited from a superclass in the
A subclass violates an invariant from the context of the new/changed subclass
superclass, or creates an invalid state
Example Another Example

class A { class A {
int x; // invariant: x > 100 void m() { … m2(); … }
void m() { // correctness depends on void m2 { … } … }
// the invariant … } … } class B extends A {
class B extends A { void m2() { … } … }
void m2() { x = 1; … } … }
If inside B we override a method from A,
If m2 has a bug and breaks the invariant, this indirectly affects other methods
m is incorrect in the context of B, even inherited from A
though it is correct in A e.g. m now calls B.m2, not A.m2: so, we
cannot be sure that m is correct anymore
Therefore m should be retested on B objects and we need to retest it inside B
Testing of Inheritance (cont) Testing of Interacting Classes
Test cases for a method m defined in Until now we only talked about testing of
class X are not necessarily good for individual classes
retesting m in subclasses of X
Class testing is not sufficient
e.g. if m calls m2 in A, and then some
OO design: several classes collaborate to
subclass overrides m2, we have a completely
implement the desired functionality
new interaction
Still, it is essential to run all superclass A variety of methods for interaction

tests on a subclass testing
We will only consider testing based on UML
Goal: check behavioral conformance of the
interaction diagrams
subclass w.r.t. the superclass (LSP)
11
UML Interaction Diagrams for Testing Collaboration Diagram
UML interaction diagrams: sequences of prepare()

messages among a set of objects :Order 1.2.1:needsReorder:=
There may be several diagrams showing needToReorder()
1*[all order lines]:
different variations of the interaction prepare()
*
Basic idea: run tests that cover all :OrderLine 1.1:hasStock:=check() :StockItem
diagrams, and all messages and conditions 1.2[hasStock]:remove()
inside each diagram 1.3[hasStock]:
If a diagram does not have conditions and create()
1.2.2[needsReorder]:create()
iteration, it contains only one path
:DeliveryItem :ReorderItem
Coverage Requirements
Run enough tests to cover all messages

and conditions
test with 0 loop iterations and >=1 iterations
test with hasStock=true and hasStock=false
test with needsReorder=true and
needsReorder=false
To cover each one: pick a particular path

in the diagram and “drive” the objects
through that path
CSE 757 - Software Testing 69
12

Testing

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Testing

Uploaded by

Copyright:

Available Formats

Motivation

 People are not perfect

CSE 757 - Software Testing 1 CSE 757 - Software Testing 2

A Way of Thinking Testing Objectives

 Design and coding are creative  Testing is a process of executing

CSE 757 - Software Testing 3 CSE 757 - Software Testing 4

Basic Definitions Testing Approaches

 Test case: specifies  In this course: small sample of

 Traditional form of white-box testing s:=0;

Elements of a Control Flow Graph IF-THEN, IF-THEN-ELSE, SWITCH

 Three kinds of nodes: if (c) if (c) switch (c)

Edges: possible flow of control ... ... ...

CSE 757 - Software Testing 9 CSE 757 - Software Testing 10

Example Mapping for Loops

Branch Coverage Branch Coverage

CSE 757 - Software Testing 15 CSE 757 - Software Testing 16

Example Achieving Branch Coverage

8  Good example in Pressman, Sect. 17.4.3

 Test connections between variable assume y is already initialized

Reaching Definitions Def-Use Pairs

Example of Def-Use Pairs Data-flow-based Testing

CSE 757 - Software Testing 25 CSE 757 - Software Testing 26

Equivalence Partitioning Equivalence Classes

 Consider input/output domains and  Examples

Example Another Example

 Search for a value in an array Array Value Output

 Many issues beyond what we discussed

CSE 757 - Software Testing 33 CSE 757 - Software Testing 34

Scope and Focus Unit Testing

 Create black-box tests  Modern practices: the importance of unit

Advantages of Test-First Programming System Testing

System Testing (cont) System Testing (cont)

CSE 757 - Software Testing 41 CSE 757 - Software Testing 42

CSE 757 - Software Testing 43 CSE 757 - Software Testing 44

 Rerun old tests to see if anything was

 Could happen at any time: during initial

Object-Oriented Software One Difference: Unit Testing

 Widespread popularity in the last decade  Traditional view of “unit”: a procedure

Possible Test Suite Polymorphism

Example of Polymorphism Testing of Polymorphism

CSE 757 - Software Testing 53 CSE 757 - Software Testing 54

 Natural representation with finite-state  States

 Final: after destruction

Examples of Transitions Finite State Machine for a Stack

 Initial -> Empty: action = “create” Initial

FSM-based Testing Inheritance

CSE 757 - Software Testing 59 CSE 757 - Software Testing 60

 Incorrect initialization of superclass  Principle: inherited methods should be

Example Another Example

Testing of Inheritance (cont) Testing of Interacting Classes

 Still, it is essential to run all superclass  A variety of methods for interaction

 UML interaction diagrams: sequences of prepare()

CSE 757 - Software Testing 67 CSE 757 - Software Testing 68

 Run enough tests to cover all messages

 To cover each one: pick a particular path

CSE 757 - Software Testing 69

You might also like

People are not perfect

Design and coding are creative Testing is a process of executing

Test case: specifies In this course: small sample of

Traditional form of white-box testing s:=0;

Three kinds of nodes: if (c) if (c) switch (c)

8 Good example in Pressman, Sect. 17.4.3

Test connections between variable assume y is already initialized

Consider input/output domains and Examples

Search for a value in an array Array Value Output

Many issues beyond what we discussed

Create black-box tests Modern practices: the importance of unit

Rerun old tests to see if anything was

Could happen at any time: during initial

Widespread popularity in the last decade Traditional view of “unit”: a procedure

Natural representation with finite-state States

Final: after destruction

Initial -> Empty: action = “create” Initial

Incorrect initialization of superclass Principle: inherited methods should be

Still, it is essential to run all superclass A variety of methods for interaction

UML interaction diagrams: sequences of prepare()

Run enough tests to cover all messages

To cover each one: pick a particular path