Metrics for Managing

G
S
T

CMM Based
Metric Workshop
 Metrics for Managing
G
S
T

• Metrics for Management

– Metric program should be simple
– Metric program should be vertically and
horizontally consistent
– Metrics should be factual
– Metrics serve as the catalyst for effective
management decisions at all organization
levels
 Metrics for Managing
G
S
T

CMM Review
– Good Business Model
– L2/Tracking to plan (compliance to spec)
– L3 Tailoring to client needs (efficiency)
– L4 Performance and Goals at project level
– L5 Performance and Goals at organization level
 Metrics for Managing
G
S
T

Measurement improves Visibility

-Need for timeliness
-Need for accuracy
-Need for consistency
-Need for relevance
-Need for completeness
Measurement data sources are varied
-quality reviews and audits; peer and technical reviews;
phased, management, and project reviews; testing
 Metrics for Managing
G
S
T

CMM Review
Status; used in each KPA (Why? What is
implication?)
Quality; part of most KPAs (Cost of Quality?)
Management Reviews; part of all KPAs

All KPAs are made up of key practices

Execution of practice requires resources. Resources
need to be controlled and managed.
 Metrics for Managing
G
S
T

• Simplicity of Measures:
– Measures for managing activity
– Measures for managing process quality
– Measures for managing product quality
– Measures for managing performance
 Metrics for Managing
G
S
T

•Vertical and Horizontal Consistency
 Metrics for Managing
G
S
T

Activity Activity

Developer Loop Developer Loop

Project Management Loop

Quality  Management Loop

Organization Management Loop
 Metrics for Managing
G
S
T

• Factual (accurate, complete, consistent)

– Metrics are derived from factual data
• Test data
• Review and Audit data
• Accounting and Timekeeping data
• Project Status Data
• Fact-based metrics facilitate timely and
management decisions.
 Metrics for Managing
G
S
T

• Metrics Act as a Catalyst

– Provides management visibility
– Eliminates guessing, removes subjectivity
from decision making process
– Meetings are focused on facts
– Consensus on status leads to consensus to
the actions to be taken
 Metrics for Managing
G
S
T

• Who needs metrics

– Decision makers
– Analytical support activities like SEPG, Quality,
Tools Group
– All levels of management
• Task Manager
• Project Leader
• Program Manager
• Group (Business Unit) Manager
• Organization Manager
 Metrics for Managing
G
S
T

• Basic Premise

“ If the metric does not support a management

decision process, then it is not necessary and
should not be collected.”
 Metrics for Managing
G
S
T

• What Metrics should be collected?

– Metrics that show the status of work activities
in the organization (Time, Effort, Current
State, number of occurrances)
– Metrics that detail performance of the activities
that are performed (actual vs plan)
– Metrics that detail the quality attributes of the
products (error density, error rate, MTTR,
MTBF, Reliability)
– Metrics that detail the quality attributes of the
process (error density, error rate)
 Metrics for Managing
G
S
T

• Metrics that show status allow manage-

ment to take effective actions to meet
project requirements
– Schedule status (effort of work items completed
less the effort of work scheduled to date) shows
the effort that will need to be added to complete
on schedule
– Effort status (effort of work items completed less
the effort expended to date) shows whether there
is overage or underage on budget
 Metrics for Managing
G
S
T

• Other status metrics

– Number of changes (opened, received, closed, etc)
– Elements of system (requirements, design objects,
units of software code, tests, etc) work/time
– Comparison of planned activities vs actual
activities
• Value of these metrics is that they assist
management in allocation of resources
to achieve the project objectives
 Metrics for Managing
G
S
T

• Quality of product metrics

– Number and type of defects found
– Distribution and value of defects in product
– Value of actions to remove and repair defects
• Value of product quality metrics is they:
– Identify problems so that they can be fixed
– Provide insight into the product being developed
– Assist in identifying product quality issues
– Enable project to enhance overall product quality
 Metrics for Managing
G
S
T

• Quality of process metrics

– Number and type of defects found
– Distribution and value of defects in process
– Value of actions to remove and repair defects
• Value of process quality metrics is, they:
– Identify problems so that they can be fixed
– Provide insight into the process being utilized
– Assist in identifying process quality issues
– Enable organization to enhance overall process quality
 Metrics for Managing
G
S
T

• Executive Organization Metrics

Process Lifecycle Phases
Work Scheduled
Work Completed
Work Accomplished
Errors (value)
Error ratio
Moving Averages (3 Month Window)
New Projects
Active Projects
Project man/mth
 Metrics for Managing
G
S
T

• Executive Organization Metrics (continued)

Moving Averages (3 Month Window)
Total Staff
Managers
Project Pers
Support
Turnover

Moving Averages (3 Month Window)

Training Ratio
Systems Ratio
HR Ratio
 Metrics for Managing
G
S
T

• Executive Organization Metrics (continued)

Moving Averages (4 Week Window, Weekly)
Productivity
Delivery
Defect Rate
Removal Eff.
Defect Ratio

Moving Averages (3 Month Window)

Performance
Technology
 Metrics for Managing
G
S
T

• Project metrics ( all stated in effort)

DEFECT DATA (In effort lost)
Schedule Effort REQ HLD DD C&T INT Other Total

Norm/Plan
Task 1
Task 2
Task 3
Task 4
Task 5
Project Total
 Metrics for Managing
G
S
T

Organization  Total

Business Unit Head Total
Business Unit Head Total
Business Unit Head Total

Program Manager Total
Program Manager Total
Program Manager Total

Project Total
Project Total
Project Total
 Metrics for Managing
G
S
T

• Project level metrics

– Status: schedule and effort, planned vs
actual
– Quality: product and process, allocation of
defects/nonconformance to elements of the
product being built and to process by
lifecycle phase
– Performance: relationship between
accomplishments, time, and quality
 Metrics for Managing
G
S
T

• Clarification of Schedule and Effort

Metrics (Schedule only as example)
– Schedule: Determine variance to schedule in
precise quantitative terms
(Effort of items completed-Effort of scheduled items)= Variance
– Example: All effort allocated to tasks of a project
scheduled to be complete after three weeks total
200 man hours (10 tasks of 20 hours each). The
actual effort of tasks completed is 160 (only 8
tasks are complete)…value is -40 hours (variance)
 Metrics for Managing
G
S
T

• Defect Data Characteristics

– Project, task, WBS element, product or process
element
– Activity Type, evaluator, sample size (if audit)
– Defect Data
• Location in product (or facility)
• Description
• Defect Code
• Priority
• Effort (if manually collected)
• Disposition
• Status
 Metrics for Managing
G
S
T

• At this point, will all of the data and

metrics, you are still a level 3 company
– You have consistency of process and
consistency of metrics used across and
vertically in organization
– Information helps projects plan better and
assists in understanding processes and
ensuring product quality
 Metrics for Managing
G
S
T

• Transition from L1 to L2/3 to L4/5

– L1 Metrics are known at the end of a
project…no value to management
– L2 Metrics used on the project plan vs
actual
– L3 Metrics consistent across organization
and used by others
– L4 Metrics used to Manage Projects
– L5 Metrics used to Manage Organization
 Metrics for Managing
G
S
T

Earned Value Metrics

Budgeted Cost of work Performed (BCWP)

Actual Cost of work Performed (ACWP)

Budgeted Cost of work Scheduled (BCWS)
 Metrics for Managing
G
S
T

% of  ACWP
Work or 
units of  BCWP:
effort
(cost or BCWS Original or 
Effort)
Cost Variance Updated 
Estimate
Schedule 
Variance

Time
 Metrics for Managing
G
S
T

Types of metrics
• Management
– Performance, timeliness, status, efficiency,
profitability, return on investment, staff turnover,
training days, growth rate, productivity, % of rework,
etc.
• Quality
– Error Rates, Cost of Quality, Number and value of
errors by type, seepage rate, distribution of errors by
projects, by life-cycle phases, etc.
 Metrics for Managing
G
S
T

Organization    Sub­organization         Project              Activity

Management

Each element has a unique set 
metrics that are important to 
that organization element
Quality
 Process Improve
Metrics for Managing
Management Process
G
S
T
Define Control Measure
Process Process Process

Execute
Process

Requirements Activity

Execute Product
Plan
Performance
Management
Define Adjust  Measure
Plan Plan Plan/Actual
 Metrics for Managing
G
S
T

Developing Metrics…a 4-Step Process

• Establish goals for measurements
• Define the attributes of the goals
• Describe metric measures
• Identify data required
 Metrics for Managing
G
S
T

• Goals Examples:
– “We need to measure our productivity”
– “We need to measure defect frequency”
– “Philips needs to measure project
accomplishments”
– “We need to measure the accuracy and
completeness of testing”
 Metrics for Managing
G
S
T

• Verification:
Establish how metric will be used to
improve or manage performance or
quality. Example: Productivity will be
used to determine how projects are
performing to their plan so that actions
can be taken when significant deviations
are noted.
 Metrics for Managing
G
S
T

• Attributes of the Goals: Productivity

– What is existing productivity?
– What is the expected productivity?
– What is the improvement desired?
– What was the productivity at the end of
each of the last 3 quarters?
– What is our productivity trend?
 Metrics for Managing
G
S
T

• Metrics of Productivity
Effort Expended (Man-hours)
Product Units

Estimated Hours per Phase

Actual Hours per Phase
 Metrics for Managing
G
S
T

• Data needed for measures: Productivity

– Units of work
– Actual effort expended on work
– Estimated effort of work
– History of units of work produced and effort
consumed in production
– Distribution of work by phase
– Actual work distribution by phase
 Metrics for Managing
G
S
T

• Exercise One:
– Groups of 5 persons grouped by function:
– Each group is assigned a level of management for metric
definition
– Each group will identify at least 10 goals relative to the level
of management of the group
– At least 6 goals should be “Management” goals and at least 4
should be “quality” goals
– Time is 1 hour: Then each group will present its goals to the
class, 10 Minutes for each presentation
– Discussion by Class
 Metrics for Managing
G
S
T

• Exercise Two:
– Each Group will verify each of its goals by
defining how the goal will be helpful in
improving performance
– 20 Minutes for verification, then 5 minutes per
group to explain verification to class
 Metrics for Managing
G
S
T

• Exercise Three:
– Each Group will identify the attributes needed for so
that the Measurement Goals can be met. Each group
will also define at least one metric method (Calculation)
for each Measurement Goal.
– Each Group will take 45 minutes to complete; then they
10 minutes to report.
 Metrics for Managing
G
S
T

• Exercise Four
– Each Group will identify the data elements
needed for each goal/attributes/measurement
and determine if the data is currently available.
– 30 minutes. Then present need and availability.