IEC Certification Kit

Simulink Test Reference Workflow

R2015b

How to Contact MathWorks

Latest news:

www.mathworks.com

Sales and services:

www.mathworks.com/sales_and_services

User community:

www.mathworks.com/matlabcentral

Technical support:

www.mathworks.com/support/contact_us

Phone:

508-647-7000

The MathWorks, Inc.

3 Apple Hill Drive
Natick, MA 01760-2098
IEC Certification Kit: Simulink TestTM Reference Workflow

COPYRIGHT 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the governments needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a
list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective
holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more
information.

Revision History
September 2015

New for IEC Certification Kit Version 3.6 (Applies to Release 2015b)

Contents
1 Introduction ...................................................................................................................................... 1-1
1.1 Overview ................................................................................................................................. 1-2
2 Integration of Simulink Test into the Model-Based Design Process ................................................ 2-1
2.1 Workflow Overview ................................................................................................................ 2-2
2.2 Tool Use Cases ........................................................................................................................ 2-5
[SLTEST_UC1] Development and execution of tests for Simulink models ................................ 2-5
[SLTEST_UC2] Development and execution of tests for back-to-back testing between model and
code .............................................................................................................................................. 2-5
[SLTEST_UC3] Assessment of test results ................................................................................. 2-5
[SLTEST_UC4] Generation of test reports .................................................................................. 2-5
[SLTEST_UC5] Identification of traceability between requirements and tests cases.................. 2-6
2.3 Error Prevention and Detection Measures ............................................................................... 2-7
[SLTEST_M1] Requirements-based testing ................................................................................ 2-7
[SLTEST_M2] Tool installation integrity checks ........................................................................ 2-7
[SLTEST_M3] Configuration management ................................................................................. 2-7
[SLTEST_M4] Input data integrity checks .................................................................................. 2-7
[SLTEST_M5] Competency of project team ............................................................................... 2-7
[SLTEST_M7] Manual comparison of test results to expected results ........................................ 2-7
[SLTEST_M8] Manual review of test report content .................................................................. 2-8
3 Additional Considerations ................................................................................................................ 3-1
3.1 Configuration Management and Revision Control .................................................................. 3-2
3.2 Competency of the Project Team ............................................................................................ 3-3
3.3 Installation Integrity and Release Compatibility ..................................................................... 3-4
3.4 Bug Reporting ......................................................................................................................... 3-5
3.5 Deviation from the Reference Workflow ................................................................................ 3-6
3.6 Integration with the Software Safety Life Cycle ..................................................................... 3-7
4 Workflow Overview......................................................................................................................... 4-1
5 Conformance Demonstration Template ........................................................................................... 5-3
6 References ........................................................................................................................................ 6-1

vi

1 Introduction

1.1 Overview
Simulink Test provides tools for authoring, managing, and executing systematic, simulationbased tests of the Simulink models. You can create nonintrusive test harnesses to test models
and subsystems. Simulink Test includes a test sequence block that lets you construct complex
test sequences and assessments, and a test manager that lets you manage and execute tests. It
enables functional, baseline, equivalence, and back-to-back testing, including software-in-theloop (SIL) and processor-in-the-loop (PIL). You can generate reports, archive and review test
results, rerun failed tests, and debug the component or system under test.
The test harness in Simulink Test lets you test components without creating a separate test
model. You can apply pass and fail criteria that include absolute and relative tolerances, limits,
logical checks, and temporal conditions. Test execution can be automated or customized with
setup and cleanup scripts. Simulink Test stores test cases and their results, creating a repository
for reviewing and investigating failures. You can link requirements to a test case captured in
Microsoft Word, IBM, Rational, DOORS, and other documents (with Simulink
Verification and Validation).
Key capabilities of Simulink Test include:

Development of test harness for subsystem or model testing

Specifying sequence of tests using Test Sequence block
Specifying pass-fail criteria, including tolerances, limits, and temporal conditions
Implementation of baseline, equivalence, and back-to-back testing
Development setup and cleanup scripts for customizing test execution
Authoring, executing, and organizing test cases and their results using Test manager
Automatic report generation to document test outcomes

This document provides a reference workflow for Simulink Test. In particular, it describes how
to:

Leverage the Simulink Test capabilities in the software life cycle

Check that these capabilities are functioning as expected

Users of the Simulink Test tool seeking to leverage the certification or qualification of the tool
shall carry out this workflow as part of the overall ISO 26262, IEC 61508, or EN 50128
software safety lifecycle.
The document is organized as follows:

1-2

 Chapter 2, Integration of Simulink Test into the Model-Based Design Process provides a
reference workflow for the Simulink Test tool. It describes reference use cases and measures
to prevent or detect potential tool errors.
Chapter 3, Additional Considerations describes tool options that impact verification results,
and other considerations such as tailoring and bug reporting.
Chapter 4, Workflow Overview summarizes the workflow in a tabular way.
Chapter 5, Conformance Demonstration Template references a template that can be used to
demonstrate conformance with this reference workflow.
Chapter 6, References lists the standards and guidelines referenced in this document
Disclaimer While adhering to the recommendations in this document will reduce the risk that
an error is introduced in development and not be detected, it is not a guarantee that the system
being developed will be safe. Conversely, if some of the recommendations in this document are
not followed, it does not mean that the system being developed will be unsafe.

1-3

2 Integration of Simulink Test into

the Model-Based Design Process

2.1 Workflow Overview

The following capabilities support the use cases described in the Simulink Test Reference
Workflow:

Development of test harness for subsystem or model testing

Specifying sequence of tests using Test Sequence block
Specifying pass-fail criteria, including tolerances, limits, and temporal conditions
Implementation of baseline, equivalence, and back-to-back testing
Development setup and cleanup scripts for customizing test execution
Authoring, executing, and organizing test cases and their results using Test manager
Automatic report generation to document test outcomes

Note: For details about the capabilities and definitions of the terms, see the Simulink Test Users
Guide
During the development of embedded application software, you can use graphical modeling with
Simulink, Fixed-Point Designer, and Stateflow to conceptualize the functionality. Using
this modeling paradigm, the application software is modeled using time-based block diagrams
and event-based state machines. The model of the application software is simulated (executed)
within the Simulink environment. The model serves as the primary representation of the
application software throughout the development process, specifying functionality and design
information, and serving as a source for automated code generation with Embedded Coder. In
practice, this model elaboration is characterized by a step-wise transformation of the application
software model from an early executable specification into a model suitable for production code
generation, and then finally into C or C++ code. To accomplish the transformation, the model is
enhanced by adding design information and implementation details. The development process
becomes the successive refinement of models, followed by automatic code generation and
compilation and linking, as shown in Figure 1.
Note: For details about the parts of the workflow related to other tools and corresponding error
detection and mitigation measures, see the following documents:

Embedded Coder Reference Workflow

Simulink Verification and Validation Reference Workflow
Simulink Design Verifier Reference Workflow
Polyspace Bug Finder Reference Workflow
Polyspace Code Prover Reference Workflow
Simulink PLC Coder Reference Workflow

2-2

Figure 1: Model-Based Design Process1

You can use the Simulink Test capabilities to test:

Models for production code generation.

Interim models created during the modeling phase.
Executable object code using the Software-in-the-Loop (SIL) and Processor-in-theLoop (PIL) features of Simulink.

Simulink test features integrated into the Model-Based Design Process are shown on Figure 2.
Note: all types of testing shown on the Figure 2 (module, integration, equivalence and back-toback testing) include the activities described in the use cases (development and execution of
tests, assessment of test results, generation of test reports, traceability identification)

Solid arrows in the figure indicate the succession of software development activities.

2-3

Figure 2: Integration of Simulink Test features into Model-Based Design Process

2-4

2.2 Tool Use Cases

It is assumed that the Simulink Test tool is used as described by one or more of the following
use cases. Uses cases are integrated into the workflow as described in the paragraph Workflow
Overview.

[SLTEST_UC1] Development and execution of tests for

Simulink models
Simulink Test is used to create and execute tests for Simulink models. Testing of Simulink
models can be leveraged to implement the following verification and testing methods:

Simulation of dynamic parts of the software architectural design, including mechanisms

for error detection and handling at the architecture level (ISO 26262-6 Table 6 method
1c).

Verification of software unit design (ISO 26262-6 Table 9 method 1c).

Implementation model testing (ISO 26262-6 Table 10 methods 1a 1c).

[SLTEST_UC2] Development and execution of tests for

back-to-back testing between model and code
Simulink Test is used to create and execute tests for back-to-back testing between model and
code using equivalence test capability (ISO 26262-6 Table 10 method 1e and Table 13 method
1e).

[SLTEST_UC3] Assessment of test results

Simulink Test is used to evaluate test results by comparing them to expected results. Applicable
for all testing activities identified in the use cases SLTEST_UC1 and SLTEST_UC2.

[SLTEST_UC4] Generation of test reports

Simulink Test is used to generate test reports. Applicable for all testing activities identified in
the use cases SLTEST_UC1 and SLTEST_UC2.

2-5

[SLTEST_UC5] Identification of traceability between

requirements and tests cases
Simulink Test is used to establish bidirectional links between textual requirements and test cases
(ISO 26262-6 Tables 11 and 14 methods 1a and 1b).

2-6

2.3 Error Prevention and Detection Measures

It is assumed that the user carries out the following measures to check the seamless functioning
of the verification and analysis capabilities provided by Simulink Test and to verify their results.

[SLTEST_M1] Requirements-based testing

The test cases and expected results are derived from requirements independent of the model
under test and the test environment. The independence provides a high degree of confidence that
errors will be detected using the actual results from the model under test in the test environment.

[SLTEST_M2] Tool installation integrity checks

Integrity of tool installation can be insured by re-running the validation test suite provided with
Simulink Test in the IEC Certification Kit.

[SLTEST_M3] Configuration management

Configuration of the life cycle data shall be managed by applicant in accordance with Clause 7
of ISO 26262.

[SLTEST_M4] Input data integrity checks

Simulink Test verifies the integrity of input files using checksum.

[SLTEST_M5] Competency of project team

Training of users can be performed to ensure correct usage of tool.

[SLTEST_M6] Analysis of Available Bug Report

Information
Assess and analyze bug report information for Simulink Test provided by MathWorks and
comply with the recommendations and workarounds, if applicable.

[SLTEST_M7] Manual comparison of test results to

expected results
Test results are manually compared to expected results to determine whether test passed or
failed.

2-7

[SLTEST_M8] Manual review of test report content

Test report content is manually reviewed to verify that it corresponds to the actual test results.

2-8

3 Additional Considerations
When implementing this reference workflow, consider the following topics:

3.1 Configuration Management and Revision Control

Configuration management shall be applied to the artifacts to be tested, as well as to other work
products specified in the respective standard or in this document.

3-2

3.2 Competency of the Project Team

Those carrying out model testing and the related error prevention and detection measures shall
be competent for the activities undertaken.

3-3

3.3 Installation Integrity and Release Compatibility

The tool user shall adhere to the installation instructions for Simulink Test (including dependent
tools).
The tool user shall verify the version of Simulink Test and the integrity of the tools installation
(including dependent tools).
Note You can use the ver command in MATLAB to display the current versions of
MATLAB, Simulink Test, Polyspace Code Prover, and other MathWorks products.
The tool user shall validate modifications or additions to shipping product(s), if applicable.

3-4

3.4 Bug Reporting

The tool user shall assess bug report information provided by the tool vendors and comply with
the recommendations and workarounds, if applicable.
After deployment of the application under development, bug report information shall also be
assessed by the tool user on a regular basis.
The tool user shall carry out corrective actions if deployed applications are affected by bugs in
the tools identified after deployment.
Issues with Simulink Test shall be reported.
Note

You can use the bug reports section of the MathWorks web site

www.mathworks.com/support/bugreports to view and report bugs related to Simulink

Test.

Note You can use the IEC Certification Kit Model Advisor check Display bug reports for
Simulink Test to display bug report information for this product.
The tool user shall validate modifications or additions to shipping product(s), if applicable.

3-5

3.5 Deviation from the Reference Workflow

In some instances, deviation from the reference workflow explained in this document might
occur. In these cases, a defined deviation procedure shall be used to document and justify
deviations from the workflow.

3-6

3.6 Integration with the Software Safety Life Cycle

The application-specific model testing activities be integrated with the overall software safety
life cycle for the application under consideration.
The applicable safety standard provides additional guidance on additional objectives and
requirements for the overall software safety life cycle.

3-7

4 Workflow Overview

Table A.1 Objectives, Prerequisites, and Work Products

Activity

Objective

Prerequisites

Verification
and testing of
Simulink
models

Back-to-back
testing
between
model and
code

To verify that dynamic parts of

software architectural design,
including mechanisms for error
detection and handling at the
architecture level, are complete
and correct (ISO 26262-6 Table
6 method 1c)
To verify that software unit
design is complete and correct
(ISO 26262-6 Table 9 method
1c)
To demonstrate that
implementation model complies
with its specification (ISO
26262-6 Table 10 methods 1a
1c)
To demonstrate that execution
semantics of the model are being
preserved during code generation
(ISO 26262-6 Table 10 method
1e and Table 13 method 1e).

Work Products

Simulink model to

be verified/tested
Software
requirements and
architectural
specifications for the
development model
to be tested

Test cases and

procedures, including
Simulink Test manager
setup files and Simulink
test harnesses, integrated
into Simulink models
Test results (available in
the Simulink Test GUI)
and reports (in PDF,
HTML or Microsoft
Word docx format)

Simulink model to

use for production

code generation
Source code (for SIL
testing)
Object code (for PIL

testing)

Equivalence test cases

and procedures, including
Simulink Test manager
setup files and Simulink
test harnesses, integrated
into Simulink models
Equivalence test results
(available in the Simulink
Test GUI) and reports (in
PDF, HTML or Microsoft
Word docx format)

4-2

5 Conformance Demonstration
Template
To justify that the requirements outlined in this document have been satisfied, you must provide
evidence for the activities that have been carried out.
The IEC Certification Kit product provides an editable Conformance Demonstration Template that can
be used to demonstrate conformance with the parts of ISO 26262-6, IEC 61508-3, or EN 50128
covered in this document.
To access the conformance demonstration template, on the MATLAB command line, type
certkitiec to open the Artifacts Explorer. The template is in Simulink Test.
For each technique or measure:

In the third column, state to what degree you applied the technique or measure for the
application under consideration by using one of the phrases Used, Used to a limited degree,
or Not used.

In the fourth column, state how you used the technique or measure in the application under
consideration. If the reference workflow includes alternative means for compliance, indicate
what variant you used. In addition, enter a reference to the document (for example, test
report or review documentation) that satisfies the requirement.

5-3

5-4

6 References
[1] IEC 61508-3:2010. International Standard IEC 61508 Functional safety of electrical / electronic /
programmable electronic safety-related systems Part 3: Software requirements. Second edition,
2010.
[2] ISO 26262-6:2011. Road vehicles Functional safety Part 6: Product development: software
level. International Standard, 2011.
[3] EN 50128:2011. Railway applications - Communication, signaling and processing systems Software for railway control and protection systems. International Standard 2011.
[4] The MathWorks. Software Quality Objectives for Source Code. Version 3.0, 2012.