c




 

 

 
 
  
 
  
  

 p
pp
Statistical Process Control (SPC) can be applied to software development processes. A process
has one or more outputs, as depicted in the figure below. These outputs, in turn, have measurable
attributes. SPC is based on the idea that these attributes have two sources of variation: natural
(also known as common) and assignable (also known as special) causes. If the observed
variability of the attributes of a process is within the range of variability from natural causes, the
process is said to be under statistical control. The practitioner of SPC tracks the variability of the
process to be controlled. When that variability exceeds the range to be expected from natural
causes, one then identifies and corrects assignable causes.
pp

p
pp
SPC is a powerful tool to optimize the amount of information needed for use in making management
decisions. Statistical techniques provide an understanding of the business baselines, insights for process
improvements, communication of value and results of processes, and active and visible involvement.
SPC provides real time analysis to establish controllable process baselines; learn, set, and dynamically
improve process capabilities; and focus business on areas needing improvement. SPC moves away from
opinion-based decision making.
These benefits of SPC cannot be obtained immediately by all organizations. SPC requires defined
processes and a discipline of following them. It requires a climate in which personnel are not punished
when problems are detected, and strong management commitment.

 !"!#$ c%

p?

?
  p

Statistical Process Control (SPC) can be applied to software development processes. A process has one
or more outputs, as depicted in the first figure below. These outputs, in turn, have measurable attributes.
SPC is based on the idea that these attributes have two sources of variation: natural (also known as
common) and assignable (also known as special) causes. If the observed variability of the attributes of a
process is within the range of variability from natural causes, the process is said to be under statistical
control. The practitioner of SPC tracks the variability of the process to be controlled. When that variability
exceeds the range to be expected from natural causes, one then identifies and corrects assignable
causes.

The key steps for implementing Statistical Process Control are:

appppppppIdentify defined processes
appppppppIdentify measurable attributes of the process
appppppppCharacterize natural variation of attributes
 appppppppTrack process variation
appppppppIf the process is in control, continue to track
appppppppIf the process is not in control:
-ppppppppppIdentify assignable cause
-ppppppppppRemove assignable cause
-ppppppppppReturn to ³Track process variation´


 ?
 
Statistical Process Control (SPC) can be applied to software development processes. A process has one
or more outputs, as depicted in Figure 1(refer page 1). These outputs, in turn, have measurable
attributes. SPC is based on the idea that these attributes have two sources of variation: natural (also
known as common) and assignable (also known as special) causes. If the observed variability of the
attributes of a process is within the range of variability from natural causes, the process is said to be
under statistical control. The practitioner of SPC tracks the variability of the process to be controlled.
When that variability exceeds the range to be expected from natural causes, one then identifies and
corrects assignable causes. Figure 2 depicts the steps in an implementation of SPC.



?


In practice, reports of SPC in software development and maintenance tend to concentrate on a few
software processes. Specifically, SPC has been used to control software (formal) inspections, testing,
maintenance, and personal process improvement. Control charts are the most common tools for
determining whether a software process is under statistical control. A variety of types of control charts
are used in SPC. Table 1, based on a survey(radiace 2000)of SPC usage in organizations attaining
Level 4 or higher on the SEI CMM metric of process maturity, shows what types are most commonly used
in applying SPC to software. The combination of an Upper Control Limit (UCL) and a Lower Control Limit
(LCL) specify, on control charts, the variability due to natural causes. Table 2 shows the levels commonly
used in setting control limits for software SPC. Table 3 shows the most common statistical techniques,
other than control charts, used in software SPC. Some of these techniques are used in trial applications
of SPC to explore the natural variability of processes. Some are used in techniques for eliminating
assignable causes. Analysis of defects is the most common technique for eliminating assignable
causes. Causal Analysis-related techniques, such as Pareto analysis, Ishikawa diagrams, the Nominal
Group Technique (NGT), and brainstorming, are also frequently used for eliminating assignable causes.

 !"#"
  
&c '(    
)' * +++,
(*  -++,
)' .++,
*  /0,
1*  /0,
"


 2 ./0,

From Ron Radice¶s survey of 25 CMM Level 4 and Level 5 organizations [Radice 2000]
$"! %!!"#"
-    
 *
 ./,
*
 3,
!* 4,
'  ./,
"&" -3,



From Ron Radice¶s survey of 25 CMM level 4 and level 5 organizations [Radice 2000]
& "#?"" "$$#!' 

 

5(   
( 
--4,
$
 
-..,
 c


-..,
    
.67,
 

c


06,
 
+7,
2 &8  
+7,
! .67,

From Ron Radice¶s survey of 25 CMM level 4 and level 5 organizations [Radice 2000]
Control charts are a central technology for SPC. Figure 3 shows a sample control chart constructed from
simulated data. This is an X-chart, where the value of the attribute is graphed. Control limits are
graphed. In this case, the control limits are based on a priori knowledge of the distribution of the attribute
when the process is under control. The control limits are at three sigma. For a normal distribution, 0.2%
of samples would fall outside the limits by chance. This control chart indicates the process is out of
control. If this control chart were for real data, the next step would be to investigate the process to
identify assignable causes and to correct them, thereby bringing the process under control.


&!"#"

Some have extended the focus of SPC in applying it to software processes. In manufacturing, the
primary focus of control charts is to bring the process back into control. In software, the product is also a
focus. When a software process exceeds the control limits, rework is typically performed on the product.
In manufacturing, the cost of stopping a process is high. In software, the cost of stopping is lower, and
few shutdown and startup activities are needed [Jalote and Saxena 2002].
SPC is one way of applying statistics to software engineering. Other opportunities for applying statistics
exist in software engineering. Table 4 shows, by lifecycle phase, some of these uses of statistics. The
National Research Council recently sponsored the Panel on Statistical Methods in Software Engineering
[NRC 1996]. The panel recommended a wide range of areas for applying statistics, from visualizing test
and metric data to conducting controlled experiments to demonstrate new methodologies.
(?))$"! ?"" "$ !?"!!!

 
 


5( 
Specify performance goals that can be measured statistically, e.g., no more
than 50 total field faults and zero critical faults with 90% confidence.

 Pareto analysis to identify fault-prone modules. Use of design of experiments
in making design decisions empirically.
2 Statistical control charts applied to inspections.

 Coverage metrics provides attributes. Design of experiments useful in
creating test suites. Statistical usage testing is based on specified operational
profile. Reliability models can be applied.
Based on [Dalal, et. al. 1993]
Those applying SPC to industrial organizations, in general, have built process improvements on top of
SPC. The focus of SPC is on removing variation caused by assignable causes. As defined here, SPC is
not intended to lower process variation resulting from natural causes. Many corporations, however, have
extended their SPC efforts with Six Sigma programs. Six Sigma provides continuous process
improvement and attempts to reduce the natural variation in processes. Typically, Six Sigma programs
use the ³Seven Tools of Quality´ (Table 5). The Shewhart Cycle (Figure 4) is a fundamental idea for
continuous process improvement.
*#?+! ,"-

9 


: To count occurrences of problems.
$
  To identify central tendencies and any skewing to one side or the
other.
   To identify the 20% of the modules which yield 80% of the issues.
 (
 2 For identifying assignable causes.
  
     For identifying correlation and suggesting causation.
 
  For identifying processes that are out of control.
;  For visually displaying data, e.g., in a pie chart.

p
p

(?##"-$
p
c"c<"#!#-"c!"%

SPC is a powerful tool to optimize the amount of information needed for use in making management
decisions [Eickelmann and Anant 2003]. Statistical techniques provide an understanding of the business
baselines, insights for process improvements, communication of value and results of processes, and
active and visible involvement. SPC provides real time analysis to establish controllable process
baselines; learn, set, and dynamically improve process capabilities; and focus business on areas needing
improvement. SPC moves away from opinion-based decision making [Radice 2000].
These benefits of SPC cannot be obtained immediately by all organizations. SPC requires defined
processes and a discipline of following them. It requires a climate in which personnel are not punished
when problems are detected. It requires management commitment [Demmy 1989].
  ? ?p
The processes controllable by SPC are unlimited in application domain, lifecycle phase, and design
methodology. Processes need to exhibit certain characteristics to be suitable for SPC (as shown in the
table below). In addition, a process to which SPC is applied should be homogeneous. For example,
applications of SPC to software inspections have found that inspections must often be decomposed to
apply SPC effectively. Florence [1999] found, for instance, that the inspection of human machine
interface specifications should be treated as a process separate from the inspection of application
specifications in the project he examined. Weller [2000] found that inspections of new and revised code
should be treated as separate processes in the project he examined. A trial application of SPC is useful
in identifying homogeneous sub processes. Issues other than identifying defined homogeneous
processes arise in implementing SPC. A second table presents such implementation issues.
"
$  ?"?


ÊppppppppWell-defined
ÊppppppppŒave attributes with observable
measures
ÊppppppppRepetitive
ÊppppppppSufficiently critical to justify monitoring
effort
(Based on [Demmy 1989])

?
 )!""!  

 

Consistent measurements cannot be expected from software processes that

are not documented and generally followed.

 Measures need not be exhaustive. One or two measures that provide insight
c    into the performance of a process or activity are adequate, especially if the

( 
measures are related to the process or activity goal. Measures that can be
tracked inexpensively are preferable.
#(
 Control charts should be constructed so as to detect process trends, not
 

 2
individual nonconforming events.

(
 Straightforward formulas exist for calculating control limits and analyzing
 
-
 distributions. Introductory college courses in statistics usually do not address
 
 process-control techniques in detail.

  2 SPC only signals the possible existence of a problem. Without detailed
c investigations, as in an audit, and instituting corrective action, SPC will not
provide any benefit.
 2 Problems in following the above recommendations for implementing SPC can
  be decreased with effective training. SPC training based on examples of
software processes is to be preferred.
(Based on [Card 1994])



 -c!"$!!$  c%

The Figure below represents a high-level process architecture for the subject practice, depicting
relationships among this practice and the nature of the influences on the practice (describing how other
practices might relate to this practice). These relationship statements are based on definitions of specific
³best practices´ found in the literature and the notion that the successful implementation of practices may
³influence´ (or are influenced by) the ability to successfully implement other practices. A brief description
of these influences is included in the table below.



$ $#"$""#.?"" "$
$ !"./0
$"$

?-"! #)$" 

"!$ c
   '(
  Statistical Process Control can only be effective if the most critical
 



(
2' processes are identified and addressed using the technique.
 

2 Practices that help establish clear goals and decision points, and
 are based on meaningful metrics and attributes based on specific
program or technical goals, stand to gain the most payback from
using SPC. SPC techniques need not be restricted to the present,
i.e., planning for the insertion of new technology later in the life
cycle should also plan for the use of SPC to ensure that processes
are controlled and reliability of the resulting software artifacts is
optimized.
  
 Practices such as Performance-Based Specifications and
    

 
 Commercial Specifications/Open System can imply the generation

 and collection of data. Such data may serve as appropriate input
to SPC techniques such as control charts. Therefore, an
environment that is data-rich provides an excellent opportunity to
leverage the benefit statistical process control techniques.
 22  (  An initial step in applying SPC is often to discover controllable and
2

 ''
2 homogeneous processes. Past performance data can be used for
 this purpose. Formal inspection processes and processes for
leveraging COTS/NDI explicitly call for metrics to be collected.
These metrics can be used as the basis for SPC.
!# !$ c
 c

  

 2
 SPC can be used not only to control processes, but also to


 
determine if quantitative requirements on software processes are

 being met. The results of SPC, then, provide valuable data and
information that can be used to manage progress towards
achievement of software requirements. Part of this ability to
manage progress is supported by the quantitative progress
measurements that are inherent in the SPC process, primarily in
the form of defect tracking and correction against specific,
quantitative quality targets.
(   

 2
 Management go/no-go decisions can be based on whether
 

 


development processes are under control. SPC presents
 graphical displays supporting such decisions. The number and
 types of available graphical formats that can be used as part of the
SPC process provide accessible visibility into progress being
made to all program stakeholders. Demonstration-based reviews
provide an excellent vehicle for communicating the progress being
made in controlling processes through the use of SPC.
 
 2 By providing control over software development processes, SPC


will result in more predictable and more reliable software.

 Rigorous testing that is guided by specifications and supported by
well-documented and accurate operational and usage-based
models will be much more effective under the controlled processes
resulting from SPC.
p