Testing: - Verification of Programs (Do Program Runs Correctly - Different Roles Within Lifecycle

Testing
Verification of programs (do program runs correctly Different roles within lifecycle
Developers testing Users testing
Different techniques
Blackbox Whitebox
Regression testing
does program works after a change?
1 3/9/2013
Spectramindsolutions.com
Testing Purpose
Developers testing
Done by developer during development (coding) Purpose: verify direction of development
Acceptance testing
Last phase before delivery of software
Program correctness
Algorithmically unsolvable problem Halting problem Testing is based on heuristics We solve a similar solvable problem
2 3/9/2013
Cost to Fix Faults
60* to 100*
1.5* to 6* Cost 1*
Definition 3 3/9/2013
Development
Post Release
The V-model of Development
Requir ements specification
System specification
System design
Detailed design
Acceptance test plan
System integration test plan
Sub-system integration test plan
Module and unit code and tess
Service
Acceptance test
System integration test
Sub-system integration test
4 3/9/2013
Testing Techniques
Testing in the small
testing of individual module
Testing in the large

address the issue of decomposing and organizing the testing activity according to the modular structure of complex program
How to choose the test case?

Convince ourselves and customers that system work correctly not only for test cases, but also for other input
5 3/9/2013
Testing Techniques
Number of test case does not necessarily contribute to the significance of the test set A fragment to compute the maximum of two numbers if x>y then max=x; else max =x; Test set {<x=3, y=2>, <x=2, y=3> } is able to detect the error, {<x=3, y=2>, <x=4,y=3>, <x=5, y=1> } is not
6 3/9/2013
Testing Techniques
Two approaches in testing in the small
White box testing (Glass Box testing, structural coverage testing)
use info about the internal structure implementation-based testing
black box testing (functional testing)

operates with no knowledge about design and coding test what the program is supposed to do specification-based testing 7 3/9/2013
Spectramindsolutions.com 7
Criteria for white-box testing

Statement coverage
every elementary statement is executed at least once
Branch coverage
each branch of the control graph is traversed at least once
Path coverage
all paths leading from the initial to the final node of ps control flow graph are traversed
8 3/9/2013
Statement coverage criterion

Select a test set T such that every statement in P is executed at least once by some t in T
an error cannot be discovered if the parts of the program containing the error are not executed an input datum executes many statements
try to minimize the number of test cases while preserving the desired coverage 9 3/9/2013
Example
read (x); read (y); if x > 0 then write ("1"); else write ("2"); end if; if y > 0 then write ("3"); else write ("4"); end if;
10 3/9/2013
{<x = 2, y = 3>, <x = - 13, y = 51>, <x = 97, y = 17>, <x = - 1, y = - 1>} covers all statements {<x = - 13, y = 51>, <x = 2, y = - 3>} is minimal
10
Weakness of the criterion

if x < 0 then x := -x; end if; z := x; {<x=-3} covers all statements it does not exercise the case when x is positive
11 3/9/2013
11
Branch coverage criterion

Select a test set T such that every branch is exercised at least once by some d in T
nodes represent decisions in the program branches are sequences of statements (I/O, assignment, procedure call) can be empty!
12 3/9/2013
12
Weakness of branch coverage

if x 0 then y := 5; else z := z - x; end if; if z > 1 then z := z / x; else z := 0; end if;
13 3/9/2013
{<x = 0, z = 1>, <x = 1, z = 3>} causes the execution of all branches, but fails to expose the risk of a division by zero
13
Path-coverage criterion
Select a test set T which traverses all paths from the initial to the final node of Ps control flow
it is finer than previous kinds of coverage however, number of paths may be too large, or even infinite (while loops)
additional constraints must be provided
14 3/9/2013
14
Exhaustive testing
If-then-else
There are 520=1014 (approx.) possible paths!
loop < 20x
15 3/9/2013
15
The feasibility problem

The coverage is often impossible
unreachable code, etc.
Adequacy criteria
adequacy scores based on coverage
example: 95% statement coverage
16 3/9/2013
16
Further problem
What if the code omits the implementation of some part of the specification? White box test cases derived from the code will ignore that part of the specification!
17 3/9/2013
17
Black box testing

also called functional testing based on the definition of what a program is intended to do derives test cases from specifications rather than on program structure
18
Black box testing (Functional Testing)

Example
The program receives as input a record describing an invoice. The invoice must be inserted into a file of invoices that is sorted by date. If other invoices exist in the file with the same date, then the invoice should be inserted after the last one.
19 3/9/2013
19
Test cases
An invoice whose date is the current date An invoice whose date is before the current date (This might be even forbidden by law) An invoice whose date is the same as that some existing invoice An invoice whose date does not exist in any previously recorded invoice Several incorrect invoices, checking different types of inconsistencies
20 3/9/2013
20
Systematic black-box techniques

decompose spec into a set of relevant features analyze behavior of the program in at least one case for each feature
21 3/9/2013
21
Testing boundary conditions

For both black box testing and white box testing Some typical programming errors are at the boundary between different classes of values
if x>y then S1 else S2 Coverage: just choose two test cases: one for x>y and one for x<= y Boundary: test for x>y, x<y, x=y 22 3/9/2013
Testing in the large

Module testing
testing a single module
Integration testing
integration of modules and subsystems
System testing
testing the entire system
Acceptance testing
performed by the customer
23 3/9/2013
Module testing
Verify: given module has been implemented correctly (it expected external behavior) Example
Mi uses Mh and Mk Mh and Mk not available how to test Mi in isolation? Provide temporary context that simulates the real context to be provided by Mh and Mk
24 3/9/2013
Module testing
In general, a module may
use an operation (calls a process) that does not belong to it access non-local data structure be used by another module
25 3/9/2013
25
Testing a functional module
STUB CALL
PROCEDURE UNDER TEST CALL
DRIVER
ACCESS TO NONLOCAL VARIABLES
26 3/9/2013
26
driver
module activating the module under test simulates use of module being tested by others sets shared data as they would be set in the real application by other modules that are yet to be designed
27 3/9/2013
27
Example driver
procedures insertTable inserts an item in a table in a proper order initTable to initialize a table printTable to print out the content of the table procedure input to read values via suitable interactions with the user
a driver initTable; input (elm1); insertTable (elm1); printTable ; ---//check whether insertion is done properly input (elm2); insertTable (elm2); printTable ; ---//check whether insertion is done properly ---Spectramindsolutions.com 28
28 3/9/2013
Integration testing
Incremental approach
modules are progressively integrated and tested Advantages
Easy to discover errors at early stages easy to localize errors incrementally
integration of n modules, then n+1
reduce the need for stubs or drivers
can proceed both top-down and bottom-up according to the USES relation
29 3/9/2013
29
System Testing
A series of tests to verify that all system elements have been properly integrated. Purpose of System Testing: to ensure that the system does what the customer wants
30 3/9/2013
30
Function Testing
Check whether integrated system performs functions as specified in requirements Example: A bank account package enter a withdrawal, calculate interest Example: A word processing system
document creation document modification document deletion 31 3/9/2013
Performance Testing:
stress Testing:
Executes the system in a manner that demands resources in abnormal quantity, frequency or volume. If requirements state that a system is to handle a specified number of devices or users, activate all those devices or users simultaneously
configuration testing
analyze various s/w, h/w configurations
robustness testing
test sunder unexpected conditions, like erroneous user commands, power failure
32 3/9/2013
32
Performance Testing
Recovery Testing:
addresses response to 1) presence of faults 2) loss of data, devices, service loss system resources and see if it recovers properly
Overload testing
evaluate when goes beyond limit ex. A transaction system is required to execute 20 tran. Per second. Test when 30 tran. Required in a second
Quality testing
evaluate systems reliability, maint. Availability include calculation of mean time to failure and mean time to repair
33 3/9/2013
33
Installation testing
Users exercise system functions and document problems that result from being the actual site Example: A system designed to work aboard a ship must demonstrate that it is not affected by severe weather, ships motion
34 3/9/2013
34
Acceptance testing
assures customers that they are getting the system they requested testing is done by the customers according to their understanding of requirements
35 3/9/2013
35
Alpha & Beta test

Alpha test
software developer site

Beta test
customer tests
customer site
developer reviews
customer tests
software customer site
developer site
36 3/9/2013
36
regression testing
identifies new faults that may have been introduced as current ones are being corrected
37 3/9/2013
37
Walkthroughs
Team of 4-6 members, chaired by SQA (why?) spec writer & manager, client, designers, SQA more experienced & senior better why? Distribute info in advance, each reviewer creates list of confusing items list of items in error Process detect suspected faults, dont correct (why?) document-driven & interactive, not participant-driven verbalization leads to fault finding spontaneously by presenter no performance appraisal
38 3/9/2013
38
Software inspections
Involve people examining the source representation with the aim of discovering anomalies and defects Do not require execution of a system so may be used before implementation May be applied to any representation of the system (requirements, design, test data, etc.) Very effective technique for discovering errors
39 3/9/2013
39
Inspection success
Many different defects may be discovered in a single inspection. In testing, one defect ,may mask another so several executions are required The reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly arise
40 3/9/2013
Inspections and testing

Inspections and testing are complementary and not opposing verification techniques Inspections can check conformance with a specification but not conformance with the customers real requirements Inspections cannot check non-functional characteristics such as performance, usability, etc.
41 3/9/2013
Inspection pre-conditions
A precise specification must be available Team members must be familiar with the organisation standards Syntactically correct code must be available An error checklist should be prepared Management must accept that inspection will increase costs early in the software process Management must not use inspections for staff appraisal
42 3/9/2013
42
Inspections
Five stage process oral overview document distributed afterwards preparation participants try to understand in detail aided by statistics of fault types from recent inspections inspection one participants walks through the document thoroughly fault finding moderator must produce a report within one day rework responsible individuals resolve all faults follow-up moderator ensures document corrected if more than 5% reworked, reconvene a 100% reinspection
43 3/9/2013
43
The inspection process
Planning Overview Individual preparation Rework Inspection meeting Follow-up
44 3/9/2013
44
Inspection procedure
System overview presented to inspection team Code and associated documents are distributed to inspection team in advance Inspection takes place and discovered errors are noted Modifications are made to repair discovered errors Re-inspection may or may not be required
45 3/9/2013
45
Inspection teams
Made up of at least 4 members Author of the code being inspected Inspector who finds errors, omissions and inconsistencies Reader who reads the code to the team Moderator who chairs the meeting and notes discovered errors Other roles are Scribe and Chief moderator
46 3/9/2013
46
Inspection checklists
Checklist of common errors should be used to drive the inspection Error checklist is programming language dependent The 'weaker' the type checking, the larger the checklist Examples: Initialisation, Constant naming, loop termination, array bounds, etc.
47 3/9/2013
47
Inspection rate
500 statements/hour during overview 125 source statement/hour during individual preparation 90-125 statements/hour can be inspected Inspection is therefore an expensive process Inspecting 500 lines costs about 40 man/hours effort
48 3/9/2013
48
Fault Statistics
Recorded by severity & fault type Metrics
fault density (faults / page or faults / KLOC)
by severity (major vs. minor) or phase
fault detection rate (faults detected / hour)
Several uses
help inspectors focus their inspections help warn of problems
compare to previous products at same stage disproportionate # of faults in particular module disproportionate # of faults of certain type
49 3/9/2013
49
Success of Inspections
Inspections can find many of the faults
82%, 70%, 93% of all detected faults (IBM, 76, 78, 86)
Inspections decrease the cost of finding faults

90% decrease in cost of detecting (Switching system, 86) 4 major faults, 14 minor faults per 2 hours (JPL, 1990)
savings of $25,000 per inspection
Inspections can decrease cost of fixing

# of faults decreased exponentially by phase (JPL, 1992)
50 3/9/2013
50
Strengths & Weaknesses of Reviews

Strengths
effective way of detecting a fault faults are detected early in the process
Weaknesses
if system has poor architecture, unwieldy to review
OO paradigm makes this easier small, independent modules
must have updated documentation from previous phase
51 3/9/2013
51

Testing: - Verification of Programs (Do Program Runs Correctly - Different Roles Within Lifecycle

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Testing: - Verification of Programs (Do Program Runs Correctly - Different Roles Within Lifecycle

Uploaded by

Copyright:

Available Formats

Testing

Cost to Fix Faults

The V-model of Development

Requir ements specification

Acceptance test plan

System integration test plan

Sub-system integration test plan

Module and unit code and tess

System integration test

Sub-system integration test

Testing in the large

How to choose the test case?

black box testing (functional testing)

Criteria for white-box testing

Statement coverage criterion

Weakness of the criterion

Branch coverage criterion

Weakness of branch coverage

There are 520=1014 (approx.) possible paths!

loop < 20x

The feasibility problem

Black box testing

Black box testing (Functional Testing)

Systematic black-box techniques

Testing boundary conditions

Testing in the large

Testing a functional module

PROCEDURE UNDER TEST CALL

ACCESS TO NONLOCAL VARIABLES

reduce the need for stubs or drivers

Alpha & Beta test

software developer site

software customer site

Inspections and testing

The inspection process

Planning Overview Individual preparation Rework Inspection meeting Follow-up

fault detection rate (faults detected / hour)

Inspections decrease the cost of finding faults

Inspections can decrease cost of fixing

Strengths & Weaknesses of Reviews

must have updated documentation from previous phase

You might also like