Requirements-Based Testing

Dr. Mats P. E. Heimdahl

University of Minnesota Software Engineering Center

Dr. Steven P. Miller

Dr. Michael W. Whalen
Advanced Computing Systems
Rockwell Collins
400 Collins Road NE, MS 108-206
Cedar Rapids, Iowa 52498
spmiller@rockwellcollins.com

Advanced Technology Center Slide 1

 Outline of Presentation

Motivation

Validation Testing

Conformance Testing

What’s Next

Advanced Technology Center Slide 2

 How We Develop Software

SW High-Level HW/SW Integration

Reqs. Development Testing

SW Design Description
SW Integration
Dev. (SW Low-Level
Testing
Reqs. & SW Arch.

SW Source SW Low-
Code Dev. Level Testing

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 3

 How we Will Develop Software
(From V to a Y)

SW High-Level HW/SW Integration

Reqs. Development Testing

Software SW Integration
Model Testing

Validation
How do we know
Testing Can we trust
our model is Conformance
Formal the code
correct? Testing
Verification generator?

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 4

 Outline of Presentation

Motivation

Validation Testing

Conformance Testing

What’s Next

Advanced Technology Center Slide 5

 How we Will Develop Software
(From V to a Y)

SW High-Level HW/SW Integration

Reqs. Development Testing

Software SW Integration
Model Testing

How do we know
our model is
correct?

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 6

 Modeling Process

High-Level SW High-Level Desired Model

Requirements Reqs. Development Properties

Low-Level Software
Requirements Model

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 7

 Problem—Modeling Frenzy
i ng
d el
o
SW High-Level Desired n om
t
Model
i
Reqs. Development st
Properties
i r
d f
e a
H
Software
Model

How do we know
the model is
“right”?
How do we test the
model?

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 8

 One Solution: Redefine Requirements

SystemDevelopment
System Reqs. HW/SW Integration
Processes(ARP4754)
Processes(ARP 4754) Development Testing

The model is the Software SW Integration

requirements Model Testing

Use Engineering
Judgment when
Testing
SoftwareDevelopment
Processes(DO-178B)

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 9

 One Solution: Redefine Requirements
M
y
C
HW/SW Integration om
SystemDevelopment
System Reqs.
Processes(ARP4754)
Processes(ARP 4754) Development Testing m
en
t
The model is the Software SW Integration
requirements Model Testing

Use Engineering
Judgment when
Testing
SoftwareDevelopment
Processes(DO-178B)

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 10

 Testing Does not go Away

System Reqs. HW/SW Integration

Development Testing

Software SW Integration
Model Testing

Extensive Testing
(MC/DC)

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 11

 It Simply Moves

System Reqs. HW/SW Integration

Development Testing

Software SW Integration
Model Testing

Extensive Testing
(MC/DC)

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 12

 Do it Right!

SW High-Level Desired Model

Reqs. Development Properties

Software
Model
Analysis
(Model Checking,
Specification Test – Theorem Proving)

Is the Model
Right?

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 13

 How Much to Test?

State Masking
Coverage MC/DC?
MC/DC
Transitio Decision
n Coverage
Coverage ?
? Somethin
Def-Use Where Do the Tests g New??
Coverage Come From?
?
Advanced Technology Center Slide 14
 Requirements Based Testing

Properties are
SW High-Level Desired Model Requirements…
Reqs. Development Properties

Software
Model

Cover the
Properties!

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 15

 Properties are Requirements

Advanced Technology Center Slide 16

 Requirements Based Testing
Advantages

 Objective Measurement of Model Validation Efforts

– Requirements Coverage in Model-based Development
– Help Identify Missing Requirements
• Measure converge of model
 Basis for Automated Generation of Requirements-based
Tests
– Even If Properties Are Not Used for Verification, They Can Be
Used for Test Automation

How Are Properties “Covered”

with Requirements-based Tests?

Advanced Technology Center Slide 17

 Property Coverage

“If the onside FD cues are off, the onside FD cues

shall be displayed when the AP is engaged”
– G(((!Onside_FD_On & !Is_AP_Engaged) ->
X(Is_AP_Engaged -> Onside_FD_On))

 Property Automata Coverage

– Cover a Synchronous Observer Representing the
Requirement (Property)
 Structural Property Coverage
– Demonstrate Structurally “Interesting” Ways in Which
the Requirement (Property) Is Met
Advanced Technology Center Slide 18
 Property Automata Coverage

 Cover Accepting State Machine As Opposed to

Structure of Property
 Büchi Coverage
– State Coverage, Transition Coverage, Lasso
Coverage…
Onside_FD_On
4
not Is_AP_Engaged ^
S0 S1 not Onside_FD_On

1 3
Is_AP_Engaged v
Onside_FD_On 2
not Is_AP_Engaged ^
not Onside_FD_On

Advanced Technology Center Slide 19

 Alternative Machine

 Different synthesis algorithms

a
Init

give different automata a

b
b
b

– Will affect the test cases S0

a
S1
b
S3

required for coverage a

b b

a !a, !b !a, !b

a, b

!a, !b a, b

!a, !b

!a, !b

b
S4 S2 S5
!a, b

!a, b !a, !b

b b
!a, b !a b

b
S6 S7

a, b
!a
!a
b

Advanced Technology Center Slide 20

 Structural Property Coverage

 Define Structural Coverage Criteria for the

Property Specification
– Traditional Condition-based Criteria such as MC/DC
Prime Candidates
 Property Coverage Different than Code Coverage
– Coverage of Code and Models
• Evaluate a decision with a specific combination of truth values in the
decision
– Coverage of Properties
• Run an execution scenario that illustrates a specific way a
requirement (temporal property) is satisfied

Advanced Technology Center Slide 21

 Example

– G(((!Onside_FD_On & !Is_AP_Engaged) ->

X(Is_AP_Engaged -> Onside_FD_On))
 Demonstrate That Somewhere Along Some Execution Trace Each MC/DC
Case Is Met
– Only the “positive” MC/DC cases
• The negative cases should have no traces
 In the Case of G(p)—Globally p Holds—we Need to Find a Test Where
– in the prefix the requirement p is met
– we reach a state of the trace where the requirement p holds because of the
specific MC/DC case of interest – let us call this case a
– then the requirement p keeps on holding through the remainder of the trace

p p a p p p

 p U ( a U X(G p))

Advanced Technology Center Slide 22

 Summary

 Objective Measurement of Model Validation Efforts

– Requirements Coverage in Model-based Development
– Help Identify Missing Requirements
 Basis for Automated Generation of Requirements-based Tests
– Even If Properties Are Not Used for Verification, They Can Be Used for
Test Automation and Test Measurement
 Challenges
– How Are Properties Specified?
• Combination of Observers and Temporal Properties
– What Coverage Criteria Are Suitable?
– How Is Automation Achieved?
– How Do We Eliminate “Obviously” Bad Tests? Should We?
– How Do We Generate “Realistic” Test-cases?
– Rigorous Empirical Studies Badly Needed

Advanced Technology Center Slide 23

 Outline of Presentation

Motivation

Validation Testing

Conformance Testing

What’s Next

Advanced Technology Center Slide 24

 How we Will Develop Software
(From V to a Y)

SW High-Level HW/SW Integration

Reqs. Development Testing

Software SW Integration
Model Testing

Can we trust
the code
generator?

SW Integration
(Executable Code Production)

Advanced Technology Center Slide 25

 “Correct” Code Generation—How?

 Provably Correct
Compilers
– Very Hard (and Often Not Generate Specification/Model Output
Convincing)

 Proof Carrying Code Specification

Based Tests
 Generate Test Suites
From Model
– Compare Model Behavior Implementation Output
With Generated Code
– Unit Testing Is Now Not
Eliminated, but Largely
Automated

Advanced Technology Center Slide 26

 Existing Capabilities

 Several Commercial and Research Tools for

Test-Case Generation
– TVEC
• Theorem Proving and Constraint Solving techniques
– Reactis from Reactive Systems Inc.
• Random, Heuristic, and Guided Search
– University of Minnesota
• Bounded Model Checking
– NASA Langley
• Bounded Model Checking/Decision Procedures/Constraint Solving
 Tools Applicable to Relevant Notations
– In Our Case Simulink

Advanced Technology Center Slide 27

 An Initial Experiment

 Used a Model of the Mode Logic of a Flight

Guidance System As a Case Example
 Fault Seeding
– Representative Faults
– Generated 100 Faulty Specifications
 Generate Test Suites
– Selection of Common (and Not So Common) Criteria
 Fault Detection
– Ran the Test Suites Against the Faulty Specifications
– Recorded the Total Number of Faults Detected

Advanced Technology Center Slide 28

 Fault Finding Results

Same Effort
100
90
80
70
60
50
40
30
20
10
0
ain n se) )
m s itio ision ( u /D C
( us e) /DC (use do
m
o c C n
D
Tr
an De ion MC /DC M /D C Ra
ble c is
i ng C MC
r ia De sk M
Va Ma ki ng
s
Ma

Advanced Technology Center Slide 29

 Model “Cheats” Test Generator

Control
FCS Surfaces

Architecture
Autopilot
PFDL PFDR

FGSL FGSR
Mode Mode
Logic Logic
FMSL FMSR

Air DataL Control Control Air DataR

Laws Laws

FCP

Advanced Technology Center Slide 30

 Effect of Test Set Size

Full Reduced

100
90
80
70
60
50
40
30
20
10
0
ain n n e) C )
om s itio c isio ( us
C/D (use
D a n e n M C
le Tr D isio C /D
r iab De
c M
V a

Advanced Technology Center Slide 31

 Summary

 Automated Generation of Conformance Tests

– Current Technology Largely Allows This Automation
 Challenges
– Development of Suitable Coverage Criteria
– Effect of Test Set Size on Test Set Effectiveness
– Effect of Model Structure on Coverage Criteria
Effectiveness
– Traceability of Tests to Constructs Tested
– Empirical Studies of Great Importance

Advanced Technology Center Slide 32

 Outline of Presentation

Motivation

Conformance Testing

Validation Testing

What’s Next

Advanced Technology Center Slide 33

 New Challenges for Testing

 Model Validation – Requirements-based Testing

– How Do We Best Formalize the Requirements?
– What Coverage Criteria Are Feasible?
– Which Coverage Criteria Are Effective (If Any)?
– How Do We Generate “Realistic” Tests?
– Will This Be a Practical (Tractable) Solution?
 Conformance Testing
– What Coverage Criteria Are Effective?
• Detecting Faults From Manual Coding
• Detecting Faults From Code Generation
– Relationship Between Model Structure and
Criteria Effectiveness
– Traceability From Tests to Model
– Relationship Between Model Coverage and Code Coverage
• Optimizations in Code Generator Will Compromise Coverage

Advanced Technology Center Slide 34

 Discussion

Advanced Technology Center Slide 35

 Perfection is Not Necessary

≥
Missed Faults

 Tools and Models Only Need To Be

Better Than Manual Processes…
I Think Many
– How Do We Demonstrate This? Already Are
• Empirical Studies Are of Great
Importance

Advanced Technology Center Slide 36

 DO-178B Test Objectives

1. The executable code complies with the high-level

requirements.
2. The executable code complies with the
specification (low-level requirements).
3. Test coverage of high-level requirements is
achieved
4. Test coverage of specification (low-level
requirements) is achieved
5. Test coverage of the executable code is achieved

Requirements-Based Testing
Conformance Testing
Advanced Technology Center Slide 37