Software Structural

Testing Methods
By George N. Brower
Analex Corporation
❖

S
tructural testing encompasses
three critical phases of soft-
ware development and test-
ing; yet, one or more of these phas- testing assures the levels of testing.2 (As used herein, a
es is often deliberately bypassed,
overlooked, or performed in a less
than rigorous manner because either
the technical advantages are not
fully considered or, more often, the
cost and schedule benefits are not
appreciated. While structural test-
ing is required by the FDA for
medical devices of moderate and
major level of concern, this testing
1
levels discussed below.
❝Structural Structural testing should be done
at the unit, integration, and system
“unit” is the smallest separately
program’s compilable – or equivalent – ele-
statements and ment of code, such as a procedure,
subroutine, class, method, or data-
decisions are base table.) Structural testing
assures the program’s statements
fully exercised and decisions are fully exercised by
code execution. For example, it
by code confirms that program loop con-
execution.❞ structs behave as expected at their

should be done for all software, data boundaries. For configurable

regardless of the level of concern. software, the integrity of the data in
Also, it should be noted that there is no fundamental configuration tables are evaluated for their impact
difference between structural testing of software on program behavior. At the unit level, structural
used in a medical device and that used in a manu- testing also includes the identification of “dead
facturing process or a manufacturer’s quality system code,” which is code that cannot be reached for exe-
(or, for that matter, in any other software). An under- cution by any code pathway.
standing of these considerations and, thus, the Integration structural testing should be performed
importance of performing structural testing, are dis- after all verification testing of the units involved and
cussed below. before system-level structural testing. Figure 1 illus-
trates the general relationship between software ver-
Definition of Software Structural Testing ification and validation. Software verification con-
firms that the output of each phase of software
Software structural testing is meant to challenge development is true to (i.e., is consistent with) the
the decisions made by the program with test cases inputs to that phase. Performance qualification test-
based on the structure and logic of the design and ing confirms the final software product, running in
source code. Complete structural testing exercises its intended hardware and environment, is consistent
the program’s data structures (such as configuration with the intended product as defined in the product
tables) and its control and procedural logic at the test specifications and software requirements.

Journal of Validation Technology 37

George N. Brower

Figure 1 Figure 2
Software Verification Versus Configuration for a
Software Validation Testing Structural Unit Test

Driver Simulating
Concept “Compute Y”

(Unit-Under-
Product Test)
Specifications Get Formatted
ABC
Verification

Software
Requirements
Stub Simulating
Verification Stub Simulating “Notify Operator of
“Format ABC” Invalid Data”
Design Performance
Qualification

Verification Unit-level structural tests can be conducted on

Source Executable
Hardware
Code Code
Verification
(Test)
Maintenance
Unit-Level Structural Testing
Figure 2 illustrates a typical configuration for a
structural unit test. The unit is compiled and linked
with a driver and stubs, as needed. The driver is a
substitute for any actual unit that will eventually call
the unit-under-test, and if the driver passes data to
the unit-under-test, it is set up to pass test case vari-
able values such as maximum, minimum, and other
nominal and “stress-test” values. The stubs are sub-
stitutes for any units called by the unit-under-test. As
with the driver, if the stubs return data to the unit-
under-test, they also pass “stress test” and nominal
data values, as appropriate. The interface of the driv-
ers and stubs, including their names, are the same as
the true units’ interfaces, allowing the set of units to
be linked without altering the unit-under-test.
the actual target hardware, on an emulator or sim-
ulator of the actual hardware, or, if the situation
requires, on a totally different processor. The latter
case may occur, e.g., if the actual hardware has no
provision for determining the results of a unit’s
tests, but where the code is written in a higher
order language. Thus, the higher order source code
(such as C or C++) can be compiled and linked to
run on another computer that supports reading the
test results where the target computer (for exam-
ple, an embedded microprocessor) could not sup-
port the tests.
Structural testing (a.k.a. white-box testing) is per-
formed with the item-under-test, in this case the unit,
being viewed internally for purposes of determining
how the item should behave – for example, in deter-
mining all possible code branches. The primary pur-
pose of unit-level structural testing is to verify the
code complies with the design, including logic,
algorithms’ correctness, and accuracy (versus paral-
lel code or hand calculations), and the correctness of
the data’s engineering units. This requires, for each
unit, complete branch tests, complete code tests
(including verifying there is no dead code), and
stress tests (to detect, e.g., overflow and underflow
conditions as well as nominal and maximum loop-
control data values). The detailed design is used to
develop the acceptance criteria.

38 Journal of Validation Technology


The Environment for Both
Integration-Level and System-Level
Structural Tests
It is best to set up the integration and system
structural tests using the actual hardware and envi-
ronment to the extent practical. There are several
reasons for this, but the two most significant are (a)
the software may have subtle conditions, both good
and bad, that will only show up when running on the
actual hardware, and (b) the final computerized sys-
tem, including the intended hardware and software,
must be qualified running on that hardware, and the
structural tests should advance the software devel-
opment towards that end. However, there are also
good and sufficient reasons to perform structural
tests partially or wholly in a simulated environment.
In considering establishing simulation capabilities,
the two most common configurations are to either
emulate the computer and simulate the environment
(used most often when the actual computer is an
embedded microprocessor and it is difficult to stim-
ulate known inputs and/or read the outputs of a test)
or to simulate both the computer and the environ-
ment (used, e.g., if an emulator of the target com-
puter is not available). The principal advantages,
then, in using a simulation of the environment and,
at times, the computer include the following: (a) the
ability to set up absolutely known input values, such
that the results can be predetermined to establish the
acceptance criteria of each test; (b) a simulator
makes it easy to establish inputs that are over, under,
and at the exact limits of critical data values; (c) it is
easy to set up illegal inputs to test all error and fail-
ure conditions; and, finally, (d) the results of each
test can be readily seen.
Integration-Level Structural Testing
Integration structural testing combines functional-
ly cohesive units of verified code (which includes
unit-level structurally tested code) by compiling
and/or assembling and linking the code, along with
any drivers and stubs needed. The structure is then
loaded into the actual or simulated environment for
execution. This allows the tester to focus on that one
functional package to confirm its correct operation,
including all internal and external interfaces.
George N. Brower
Following completion of each functional package’s
test, the next functional package may be either sepa-
rately tested or added to (i.e., linked with) the previ-
ously tested package(s). Regression testing (i.e., run-
ning a selected subset of previous, successfully run
test cases) must be performed on the previously test-
ed packages to confirm they are not adversely affect-
ed by the newly introduced functional package.
Figure 3 illustrates a building-block, or incre-
mental, approach to structural testing. While the fig-
ure’s illustration is related to a structured design, the
same approach is used for all other design methods,
including flowchart and object-oriented design. In
Figure 3, the first function needs two stubs, “Get
Formatted ABC” and “Output Y to Device X,”
(where a stub is a simple software dummy needed to
link successfully, but is not part of the software
being tested). The stubs in the figure have no calls to
additional units. The second and third functions
require no driver (because they use the actual
“Compute Y”) and they require no stubs (because
they use the actual “Output Error Messages” unit).
The Incremental Approach
The incremental approach to integration-level
structural tests is the best for software developers (as
opposed to third party, validation testers – see below),
especially if the program is large or complex. In this
approach, selected small, functionally cohesive por-
tions of the software are compiled, linked, and test-
ed. This approach is used regardless of the software
life cycle development method being employed,
including any of the following three methods. In the
waterfall method, all of the requirements are devel-
oped, then the design is completed, and, finally,
selected “threads” are coded and structurally tested.
In the spiral method, a major element of the software
system is discussed and then the requirements,
design, and code are developed, and the element’s
structural test is performed prior to going on to dis-
cuss and develop the next major element. Finally, in
the incremental software development method, all of
the specifications and requirements may be devel-
oped, but the design and implementation are devel-
oped one function at a time.
In any case, if the operating system was unique-
ly developed for the system-under-test, that operat-
Journal of Validation Technology 39
George N. Brower

Figure 3
Incremental Integration Structural Testing

Compute Y

Get Formatted Output Y to

Initialize Data
ABC Device X

Notify Operator
Open Files if Device X
Format Notify Operator is Off-Line
ABC of Invalid Data
Notify Operator
2nd Function 3rd Function
of Any Errors

1st Function

Output Error
Messages

ing system should be structurally tested first. This
portion of the code itself should be broken down
into functionally cohesive packages if it is large
and/or complex; otherwise, it can be structurally
tested as an entity. The second portion of the code
to be tested is normally the unique input/output
section. If there are diverse input/output devices,
these may be structurally tested separately. But it is
often best to select a thread that includes both the
ability to input data and to see the resulting output
for each structural test. The third step is to select
and structurally test a functionally cohesive portion
of the application and the utilities needed to sup-
port that application. (Remember: The units select-
ed for integration structural testing, and this is
especially true for the utilities, should be verifica-
tion tested first – which includes unit testing – prior
to any integration-level structural testing.) Then,
select the next functionally cohesive portion of the
application software and associated application
utilities for the next structural test, and so on. All
previously tested functions should be regression
tested, as appropriate.
The Nonincremental Approach
As used herein, a nonincremental approach is one
in which the complete software package, as it is
intended to be deployed, is totally built and then
structurally tested. The nonincremental approach to
integration-level structural tests may reasonably be
used for small programs and is often the best ap-
proach for third party, independent testers. Although
in most circumstances integration-level structural
testing is the purview of the software developers and
not, for example, that of the computer system vali-
dation team, at times these third-party testers are
asked to support a program in this manner. An
immediate problem that most often surfaces, howev-
er, is due to budget, schedule, and/or personnel con-

40 Journal of Validation Technology


straints, testers independent of the software develop-
ment team cannot, in a cost-effective manner, select,
compile, and link individual software functions.
However, they can build and load the fully integrat-
ed program, select and test a function (ignoring the
remainder of program while maintaining a structur-
al-tester’s point-of-view), and then select the next
function for test. One advantage to this approach is
since a majority of the code may be executing dur-
ing each test, the need for regression testing of the
previous functions is diminished. A disadvantage to
this nonincremental approach is if a function is
found to not work properly, finding the cause of the
problem may be more difficult precisely because so
much of the code is executing.
At the integration level, structural testing uses the
architectural high-level design to develop the test
cases and to derive the acceptance criteria. (During
the integration test phase, functional, black-box test-
ing should also be performed.)
System-Level Structural Testing
The system-level structural tests must be run on
the fully integrated software package. However,
whether the software functions for the integration
structural tests discussed above are built incremen-
tally or all at once prior to conducting the integration
structural tests, the system-level structural tests must
still be run. The primary differences between inte-
gration level and system level structural tests are the
tester’s frame of reference for developing the test
inputs and determining the acceptance criteria of the
test outputs. An important secondary difference in
integration structural testing is that the code may be
altered (e.g., to create test drivers and stubs) or the
normal code execution flow may be altered (e.g.,
running the software in a de-bug compiler mode
where the execution can be halted, a data value
changed, and the execution restarted, allowing for a
variable’s maximum or minimum value, or a failure
or other perturbation to be inserted). In system struc-
tural testing, the code must not be changed, and the
program execution flow must not be altered. Rather,
all test inputs must be accomplished by control of
the actual or simulated environment.
At the system level, structural testing is also best
performed along with functional, black-box testing,
George N. Brower
both using the product specifications (which in turn
may include the user’s manual and other labeling)
and software requirements for the test development
and the acceptance criteria. Although, at the system
level, a knowledge of the design may be necessary
to determine how to set up a test for a specific func-
tionality (for example, an output error message that
a file cannot be found versus an output error mes-
sage indicating there is invalid data in a file), that
design knowledge should not be used in construct-
ing the acceptance criteria of the tests – such as the
boundaries of legitimate data. That information
should be derived from the specifications and
requirements.
The Importance of Each Testing Phase
As with each verification, integration, and qual-
ification activity, each structural testing phase finds
unique problems that otherwise may be either very
difficult to find at best or, at worst, may be left
undetected during pre-release testing, waiting for
the error to surface in the field under actual use.
Using a step-wise approach to testing finds prob-
lems and solutions faster and with a greater guar-
antee of producing a solid software product than if
any phase is skipped or not performed with rigor.
In this context, rigor means (a) planning all tests
and reviewing the plans for completeness, correct-
ness, and consistency; (b) writing the procedures
sufficiently well so they may be reviewed and,
when necessary, rerun (e.g., following software
updates); and (c) maintaining a written record of
the tests performed and the results so those may
also be reviewed for confirmation that all tests
have been completed. 3 In performing the above
three tasks, program management is supported in
that all of the activities, from planning through
execution and reporting, can be scheduled, budget-
ed, and, therefore, managed.
Figure 4 illustrates, for each structural test phase,
the documentation against which the software is test-
ed.1 In that figure, software development implementa-
tion is shown on the left side, going from product
specifications to code implementation. Software
(structural) test is shown on the right side, going from
unit test to system test. The documentation for each
test phase is illustrated by the horizontal arrows. This
Journal of Validation Technology 41
George N. Brower
general approach is followed for any software devel-
opment and testing life cycle approach, be it waterfall,
spiral, or iterative. For example, if the spiral approach
is used, a piece of the product specification is deter-
mined, the software requirements for that piece are
developed, the high-level and detailed design, and
then the code for that piece are developed, and then the
testing is accomplished starting at the unit level, work-
ing through the integration level, and finally working
up to the system level.
Figure 4
Structural Test Verification
Documentation
Product Specifications System-Level
and Structural
Software Requirements Test
Integration-Level
Top-Level Design
Structural Test
Unit-Level
Detailed Design
Structural Test
Implementation
(Coding)
Again, the preceding discussion, as well as all
aspects of this paper, applies to all software applica-
tions, whether the software is for a process control
system, medical device, software supporting a qual-
ity system, or any other software, such as that used
for simulators or even business systems.
Summary
Structural testing allows a step-by-step, rigorous
test approach that easily uncovers problems at each
phase that may otherwise be very difficult to detect
during other software verification and validation
activities. Therefore, structural testing not only adds
value from a technical point of view, but it also sup-
ports programmatic issues, mitigating the possibility
42 Journal of Validation Technology
of hidden, difficult-to-find software problems in the
succeeding test phases that can easily disrupt any
schedule and budget. ❏
About the Author
George N. Brower has published several articles
and has given presentations at numerous confer -
ences on his areas of expertise: software develop -
ment and software Independent Verification and
Validation (IV&V) testing. For the past 11 years, he
has served as Deputy Director of the Denver office
of Analex Corporation, providing software engineer -
ing services for computer systems validation and
outsource software development. Analex-Denver is
an ISO 9001 4 registered company that is the recipi -
ent of the 1995 Supplier of the Year Award for IV&V
testing and the 1996 recipient of the James S.
Cogswell Industrial Achievement Award. In 1998
Analex received a second award for IV&V testing of
critical software. Brower can be reached by phone
toll free at 1-888-262-5391, by fax at 303-730-2057,
or by e-mail at brower@analex.com.
References
1. Guidance for FDA Reviewers and Industry, Guidance for the
Content of Premarket Submissions for Software Contained in
Medical Devices, CDRH, May 29, 1998.
2. Guidance for Industry, General Principles of Software Val-
idation (Draft Guidance) Version 1.1, June 9, 1997.
3. FDA Current Good Manufacturing Practice (cGMP) Final
Rule; Quality System Regulation, 21 CFR Part 820.30, Design
Control, fully effective June 1, 1998.
4. International Organization for Standardization, 1994. (Analex
is registered for independent software verification and valida-
tion, software development, systems analysis, and hardware
prototyping.)