CH 11 Lect 1

UsingUML,Patterns,andJava
ObjectOrientedSoftwareEngineering
Chapter11,Testing
Outline
Terminology
Typesoferrors
Dealingwitherrors
QualityassurancevsTesting
ComponentTesting
Unittesting
Integrationtesting
Systemtesting
Functiontesting
StructureTesting
Performancetesting
Acceptancetesting
Installationtesting
TestingStrategy
DesignPatterns&Testing
BerndBruegge&AllenH.Dutoit
ObjectOrientedSoftwareEngineering:UsingUML,Patterns,andJava
SomeObservations
Itisimpossibletocompletelytestanynontrivialmoduleorany
system
Theoreticallimitations:Haltingproblem ??
Practiallimitations:Prohibitiveintimeandcost
Testingcanonlyshowthepresenceofbugs,nottheirabsence
(Dijkstra)
totalnumberofexecutionpaths?
loop 200 times
TestingActivities
Subsystem
Code
Subsystem
Code
Unit
Test
Tested
Subsystem
Unit
Test
Tested
Subsystem
Requirements
Analysis
Document
System
Design
Document
Integration
Test
Integrated
Subsystems
Functional
Test
User
Manual
Functioning
System
TestedSubsystem
Subsystem
Code
Unit
Test
Alltestsbydeveloper
Alltestsbydeveloper
Cf.levelsoftesting
TestingActivitiescontinued
Clients
Understanding
ofRequirements
Global
Requirements
Validated
Functioning
System PerformanceSystem
Test
Accepted
System
Acceptance
Test
User
Environment
Installation
Test
Usable
System
Testsbyclient
Testsbyclient
Testsbydeveloper
Testsbydeveloper
Usersunderstanding
Tests(?)byuser
Tests(?)byuser
Systemin
Use
system
requireme
nts
system
integration
software
requireme
nts
acceptance
test
in
unit
test
n
sig
de
code &
debug
te
g
r
at
nd
ea
detailed
design
software
integratio
n
componen
t
test
t
e
st
preliminar
y
design
an
d
z
aly
an
Level of abstraction
LevelsofTestinginVModel
Time
N.B.:componenttestvs.unittest;acceptancetestvs.systemintegration
TestPlanning
ATestPlan:
coversalltypesandphasesof
testing
guidestheentiretestingprocess
who,why,when,what
developedasrequirements,
functionalspecification,andhigh
leveldesignaredeveloped
shouldbedonebefore
implementationstarts
[Pressman]
Atestplanincludes:
testobjectives
scheduleandlogistics
teststrategies
testcases
procedure
data
expectedresult
proceduresforhandling
problems
FaultHandlingTechniques
FaultHandling
FaultAvoidance
Design
Methodology
Verification
FaultDetection
FaultTolerance
Atomic
Transactions
Reviews
Modular
Redundancy
Configuration
Management
Debugging
Testing
Unit
Testing
Integration
Testing
System
Testing
Correctness
Debugging
Performance
Debugging
QualityAssuranceencompassesTesting
QualityAssurance
UsabilityTesting
Scenario
Testing
FaultAvoidance
Verification
Prototype
Testing
Product
Testing
FaultTolerance
Configuration
Management
Atomic
Transactions
Modular
Redundancy
FaultDetection
Reviews
Walkthrough
Inspection
Unit
Testing
Debugging
Testing
Integration
Testing
System
Testing
Correctness
Debugging
Performance
Debugging
TypesofTesting
UnitTesting:
Individualsubsystem
Carriedoutbydevelopers
Goal:Confirmthatsubsystemsiscorrectlycodedandcarriesoutthe
intendedfunctionality
IntegrationTesting:
Groupsofsubsystems(collectionofclasses)andeventuallytheentire
system
Goal:Testtheinterfaceamongthesubsystem
10
SystemTesting
SystemTesting:
Terminology:
systemtestinghere=validationtesting
Theentiresystem
Goal:Determineifthesystemmeetstherequirements(functional
andglobal)
AcceptanceTesting:
2kindsofAcceptancetesting
Evaluatesthesystemdeliveredbydevelopers
Carriedoutbytheclient.Mayinvolveexecutingtypical
transactionsonsiteonatrialbasis
Goal:Demonstratethatthesystemmeetscustomerrequirements
andisreadytouse
Implementation(Coding)andtestinggohandinhand
11
UnitTesting
Informal:
Incrementalcoding
Writealittle,testalittle
StaticAnalysis:
Handexecution:Readingthesourcecode
WalkThrough(informalpresentationtoothers)
CodeInspection(formalpresentationtoothers)
AutomatedToolscheckingfor
syntacticandsemanticerrors
departurefromcodingstandards
DynamicAnalysis:
Blackboxtesting(Testtheinput/outputbehavior)
Whiteboxtesting(Testtheinternallogicofthesubsystemorobject)
Datastructurebasedtesting(Datatypesdeterminetestcases)
Whichismoreeffective,staticordynamicanalysis?
12
BlackboxTesting
Focus:I/Obehavior.Ifforanygiveninput,wecanpredictthe
output,thenthemodulepassesthetest.
Almostalwaysimpossibletogenerateallpossibleinputs("test
cases")
why?
Goal:Reducenumberoftestcasesbyequivalencepartitioning:
Divideinputconditionsintoequivalenceclasses
Choosetestcasesforeachequivalenceclass.(Example:Ifanobject
issupposedtoacceptanegativenumber,testingonenegative
numberisenough)
If x = 3 then
If x > -5 and x < 5 then
What would be the equivalence classes?

13
BlackboxTesting(Continued)
Selectionofequivalenceclasses(Norules,onlyguidelines):
Inputisvalidacrossrangeofvalues.Selecttestcasesfrom3
equivalenceclasses:
Belowtherange
Withintherange
Abovetherange
Are these complete?
Inputisvalidifitisfromadiscreteset.Selecttestcasesfrom2
equivalenceclasses:
Validdiscretevalue
Invaliddiscretevalue
Anothersolutiontoselectonlyalimitedamountoftestcases:
Getknowledgeabouttheinnerworkingsoftheunitbeingtested=>
whiteboxtesting
14
WhiteboxTesting
Focus:Thoroughness(Coverage).Everystatementinthecomponentisexecutedatleast
once.
Fourtypesofwhiteboxtesting
StatementTesting
LoopTesting
PathTesting
BranchTesting
15
WhiteboxTesting(Continued)
StatementTesting(AlgebraicTesting):Testsinglestatements
LoopTesting:
Causeexecutionofthelooptobeskippedcompletely.(Exception:
Repeatloops)
Looptobeexecutedexactlyonce
Looptobeexecutedmorethanonce
Pathtesting:
Makesureallpathsintheprogramareexecuted
BranchTesting(ConditionalTesting):Makesurethateach
possibleoutcomefromaconditionistestedatleastonce
if(i=TRUE)printf("YES\n");
elseprintf("NO\n");
Testcases:1)i=TRUE;2)i=FALSE
16
WhiteBoxTesting
LoopTesting
[Pressman]
Simple
loop
Nested
Loops
Concatenated
Loops
Why is loop testing important?
Unstructured
Loops
17
WhiteboxTestingExample
FindMean(float Mean, FILE ScoreFile)
{ SumOfScores = 0.0; NumberOfScores = 0; Mean = 0;
Read(ScoreFile, Score); /*Read in and sum the scores*/
while (! EOF(ScoreFile) {
if ( Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
NumberOfScores++;
}
Read(ScoreFile, Score);
}
/* Compute the mean and print the result */
if (NumberOfScores > 0 ) {
Mean = SumOfScores/NumberOfScores;
printf("The mean score is %f \n", Mean);
} else
printf("No scores found in file\n");
}
18
WhiteboxTestingExample:DeterminingthePaths
FindMean (FILE ScoreFile)
{ float SumOfScores = 0.0;
int NumberOfScores = 0;
1
float Mean=0.0; float Score;
2 while (! EOF(ScoreFile) {
3 if (Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
NumberOfScores++;
}
5
}
/* Compute the mean and print the result */
7 if (NumberOfScores > 0) {
Mean = SumOfScores / NumberOfScores;
printf( The mean score is %f\n, Mean);
} else
printf (No scores found in file\n);
9
}
19
ConstructingtheLogicFlowDiagram
Start
1
F
2
T
3
F
5
4
6
7
8
Exit
20
FindingtheTestCases
Start
1
a(Coveredbyanydata)
2
b (Datasetmustcontainatleastonevalue)
(Positivescore) d
c
4
(Datasetmust
f
beempty)
6
7
(Totalscore<0.0) i
8
e (Negativescore)
5
h (Reachedifeitherfor
g
eisreached)
j (Totalscore>0.0)
9
Exit
21
Selfreading
ComparisonofWhite&BlackboxTesting25.1.2002
WhiteboxTesting:
Potentiallyinfinitenumberof
pathshavetobetested
Whiteboxtestingoftentests
whatisdone,insteadofwhat
shouldbedone
Cannotdetectmissingusecases
BlackboxTesting:
Potentialcombinatorical
explosionoftestcases(valid&
invaliddata)
Oftennotclearwhetherthe
selectedtestcasesuncovera
particularerror
Doesnotdiscoverextraneous
usecases("features")
Bothtypesoftestingareneeded
Whiteboxtestingandblackbox
testingaretheextremeendsofa
testingcontinuum.
Anychoiceoftestcaseliesin
betweenanddependsonthe
following:
Numberofpossiblelogicalpaths
Natureofinputdata
Amountofcomputation
Complexityofalgorithmsand
datastructures
22
The4TestingSteps
1.Selectwhathastobe
1.Selectwhathastobe
measured
measured
3.Developtestcases
3.Developtestcases
2.Decidehowthetestingis
2.Decidehowthetestingis
done
done
4.Createthetestoracle
4.Createthetestoracle
Analysis:Completenessof
Analysis:Completenessof
requirements
requirements
Design:testedforcohesion
Design:testedforcohesion
Implementation:Codetests
Implementation:Codetests
Codeinspection
Codeinspection
Proofs(DesignbyContract)
Proofs(DesignbyContract)
Blackbox,whitebox,
Blackbox,whitebox,
Selectintegrationtesting
Selectintegrationtesting
strategy(bigbang,bottom
strategy(bigbang,bottom
up,topdown,sandwich)
up,topdown,sandwich)
Atestcaseisasetoftest
Atestcaseisasetoftest
dataorsituationsthatwill
dataorsituationsthatwill
beusedtoexercisetheunit
beusedtoexercisetheunit
(code,module,system)being
(code,module,system)being
testedorabouttheattribute
testedorabouttheattribute
beingmeasured
beingmeasured
Anoraclecontainsofthe
Anoraclecontainsofthe
predictedresultsforasetof
predictedresultsforasetof
testcases
testcases
Thetestoraclehastobe
Thetestoraclehastobe
writtendownbeforethe
writtendownbeforethe
actualtestingtakesplace
actualtestingtakesplace
Next module
23
Selfreading
GuidanceforTestCaseSelection
Useanalysisknowledge
Useanalysisknowledge
aboutfunctional
aboutfunctional
requirements(blackbox
requirements(blackbox
testing):
testing):
Usecases
Usecases
Expectedinputdata
Expectedinputdata
Invalidinputdata
Invalidinputdata
Useimplementation
Useimplementation
knowledgeaboutalgorithms:
knowledgeaboutalgorithms:
Examples:
Examples:
Forcedivisionbyzero
Forcedivisionbyzero
Usesequenceoftestcasesfor
Usesequenceoftestcasesfor
interrupthandler
interrupthandler
Usedesignknowledgeabout
Usedesignknowledgeabout
systemstructure,algorithms,
systemstructure,algorithms,
datastructures(whitebox
datastructures(whitebox
testing):
testing):
Controlstructures
Controlstructures
Testbranches,loops,...
Testbranches,loops,...
Datastructures
Datastructures
Testrecordsfields,
Testrecordsfields,
arrays,...
arrays,...
24
Selfreading
UnittestingHeuristics
1.Createunittestsassoonasobject
1.Createunittestsassoonasobject
designiscompleted:
designiscompleted:
Blackboxtest:Testtheuse
Blackboxtest:Testtheuse
cases&functionalmodel
cases&functionalmodel
Whiteboxtest:Testthe
Whiteboxtest:Testthe
dynamicmodel
dynamicmodel
Datastructuretest:Testthe
Datastructuretest:Testthe
objectmodel
objectmodel
2.Developthetestcases
2.Developthetestcases
Goal:Findtheminimal
Goal:Findtheminimal
numberoftestcasestocover
numberoftestcasestocover
asmanypathsaspossible
asmanypathsaspossible
3.Crosscheckthetestcasesto
3.Crosscheckthetestcasesto
eliminateduplicates
eliminateduplicates
Don'twasteyourtime!
Don'twasteyourtime!
4.Deskcheckyoursourcecode
4.Deskcheckyoursourcecode
Reducestestingtime
Reducestestingtime
5.Createatestharness
5.Createatestharness
Testdriversandteststubsare
Testdriversandteststubsare
neededforintegrationtesting
neededforintegrationtesting
6.Describethetestoracle
6.Describethetestoracle
Oftentheresultofthefirst
Oftentheresultofthefirst
successfullyexecutedtest
successfullyexecutedtest
7.Executethetestcases
7.Executethetestcases
Dontforgetregressiontesting
Dontforgetregressiontesting
Reexecutetestcaseseverytime
Reexecutetestcaseseverytime
achangeismade.
achangeismade.
Big cost -> what should be done?
8.Comparetheresultsofthetestwiththe
8.Comparetheresultsofthetestwiththe
testoracle
testoracle
Automateasmuchaspossible
Automateasmuchaspossible
25
OOTStrategy
[Pressman]
classtestingistheequivalentofunittesting
if there is no nesting of classes
operationswithintheclassaretested
thestatebehavioroftheclassisexamined
integrationappliedthreedifferentstrategies/levelsof
abstraction
threadbasedtestingintegratesthesetofclassesrequired
torespondtooneinputorevent
usebasedtestingintegratesthesetofclassesrequiredto
respondtooneusecase
this is pushing
clustertestingintegratesthesetofclassesrequiredto
demonstrateonecollaboration
Recall: model-driven software development
26
OOTTestCaseDesign
[Pressman]
Berard [BER93] proposes the following approach:

1. Each test case should be uniquely identified and should be explicitly
associated with the class to be tested,
2. A list of testing steps should be developed for each test and should
contain [BER94]:
a. a list of specified states for the object that is to be tested
b. a list of messages and operations that will be exercised as a
consequence of the test
how can this be done?
c. a list of exceptions that may occur as the object is tested
d. a list of external conditions (i.e., changes in the environment external
to the software that must exist in order to properly conduct the test)
{people, machine, time of operation, etc.}
This is a kind of data structure testing
27
OOTMethods:BehaviorTesting
The tests to be
designed should
achieve all state
coverage [KIR94].
That is, the operation
sequences should
cause the Account
class to make
transition through all
allowable states
open
empty
acct
[Pressman]
setup Accnt
set up
acct
deposit
(initial)
deposit
balance
credit
accntInfo
working
acct
withdraw
withdrawal
(final)
dead
acct
close
nonworking
acct
Figure 14.3 State diagram for Account class (adapted from [ KIR94])
This can act as an oracle
28
WhoTeststheSoftware?
developer
independent tester
Understands the system

but, will test "gently"
and, is driven by "delivery"
[Pressman]
Must learn about the system,

but, will attempt to break it
and, is driven by quality
29
NFRs:Reliability
[Chung, RE Lecture Notes]]
CountingBugs
Sometimesreliabilityrequirementstaketheform:
"ThesoftwareshallhavenomorethanXbugs/1KLOC"
Buthowdowemeasurebugsatdeliverytime?
BebuggingProcessbasedonaMonteCarlotechniqueforstatisticalanalysisofrandomevents.
1.beforetesting,aknownnumberofbugs(seededbugs)aresecretlyinserted.
2.estimatethenumberofbugsinthesystem
3.remove(bothknownandnew)bugs.
#ofdetectedseededbugs/#ofseededbugs=#ofdetectedbugs/#ofbugsinthesystem
#ofbugsinthesystem=#ofseededbugsx#ofdetectedbugs/#ofdetectedseededbugs
Example:secretelyseed10bugs
anindependenttestteamdetects120bugs(6fortheseeded)
#ofbugsinthesystem=10x120/6=200
#ofbugsinthesystemafterremoval=2001204=76
But,deadlybugsvs.insignifantones;notallbugsareequallydetectable;(Suggestion[Musa87]:
"NomorethanXbugs/1KLOCmaybedetectedduringtesting"
"NomorethanXbugs/1KLOCmayberemainafterdelivery,
ascalculatedbytheMonteCarloseedingtechnique"
30
Summary
Testingisstillablackart,butmanyrulesandheuristicsare
available
Testingconsistsofcomponenttesting(unittesting,integration
testing)andsystemtesting,and
OOTandarchitecturaltesting,stillchallenging
Userorientedreliabilitymodelingandevaluationnotadequate
Testinghasitsownlifecycle
31
AdditionalSlides
32
Terminology
Reliability:Themeasureofsuccesswithwhichtheobserved
behaviorofasystemconfirmstosomespecificationofits
behavior.
Failure:Anydeviationoftheobservedbehaviorfromthe
specifiedbehavior.
Error:Thesystemisinastatesuchthatfurtherprocessingby
thesystemwillleadtoafailure.
Fault(Bug):Themechanicaloralgorithmiccauseofanerror.
Therearemanydifferenttypesoferrorsanddifferentwayshow
wecandealwiththem.
33
ExamplesofFaultsandErrors
FaultsintheInterface
FaultsintheInterface
specification
specification
Mismatchbetweenwhatthe
Mismatchbetweenwhatthe
clientneedsandwhatthe
clientneedsandwhatthe
serveroffers
serveroffers
Mismatchbetween
Mismatchbetween
requirementsand
requirementsand
implementation
implementation
AlgorithmicFaults
AlgorithmicFaults
Missinginitialization
Missinginitialization
Branchingerrors(toosoon,
Branchingerrors(toosoon,
toolate)
toolate)
Missingtestfornil
Missingtestfornil
MechanicalFaults(very
MechanicalFaults(very
hardtofind)
hardtofind)
Documentationdoesnot
Documentationdoesnot
matchactualconditionsor
matchactualconditionsor
operatingprocedures
operatingprocedures
Errors
Errors
Stressoroverloaderrors
Stressoroverloaderrors
Capacityorboundaryerrors
Capacityorboundaryerrors
Timingerrors
Timingerrors
Throughputorperformance
Throughputorperformance
errors
errors
34
DealingwithErrors
Verification:
Assumeshypotheticalenvironmentthatdoesnotmatchreal
environment
Proofmightbebuggy(omitsimportantconstraints;simplywrong)
Modularredundancy:
Expensive
Declaringabugtobeafeature
Badpractice
Patching
Slowsdownperformance
Testing(thislecture)
Testingisnevergoodenough
35
AnotherViewonHowtoDealwithErrors
Errorprevention(beforethesystemisreleased):
Usegoodprogrammingmethodologytoreducecomplexity
Useversioncontroltopreventinconsistentsystem
Applyverificationtopreventalgorithmicbugs
Errordetection(whilesystemisrunning):
Testing:Createfailuresinaplannedway
Debugging:Startwithanunplannedfailures
Monitoring:Deliverinformationaboutstate.Findperformancebugs
Errorrecovery(recoverfromfailureoncethesystemisreleased):
Databasesystems(atomictransactions)
Modularredundancy
Recoveryblocks
36
Whatisthis?
Afailure?
Anerror?
Afault?
Needtospecify
thedesiredbehaviorfirst!
37
ErroneousState(Error)
38
AlgorithmicFault
39
MechanicalFault
40
HowdowedealwithErrorsandFaults?
41
Verification?
42
ModularRedundancy?
43
DeclaringtheBug
asaFeature?
44
Patching?
45
Testing?
46
Testingtakescreativity
Testingoftenviewedasdirtywork.
Todevelopaneffectivetest,onemusthave:
Detailedunderstandingofthesystem
Knowledgeofthetestingtechniques
Skilltoapplythesetechniquesinaneffectiveandefficientmanner
Testingisdonebestbyindependenttesters
Weoftendevelopacertainmentalattitudethattheprogramshould
inacertainwaywheninfactitdoesnot.
Programmeroftensticktothedatasetthatmakestheprogram
work
"Dontmessupmycode!"
Aprogramoftendoesnotworkwhentriedbysomebodyelse.
Don'tletthisbetheenduser.
47
TestCases
Testcase1:?(Toexecuteloopexactlyonce)
Testcase2:?(Toskiploopbody)
Testcase3:?,?(toexecuteloopmorethanonce)
These3testcasescoverallcontrolflowpaths
48

CH 11 Lect 1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CH 11 Lect 1

Uploaded by

Copyright:

Available Formats

UsingUML,Patterns,andJava

loop 200 times

If x > -5 and x < 5 then

What would be the equivalence classes?

Are these complete?

if there is no nesting of classes

Recall: model-driven software development

Berard [BER93] proposes the following approach:

This is a kind of data structure testing

This can act as an oracle

Understands the system

Must learn about the system,

[Chung, RE Lecture Notes]]

You might also like