CORPORATE LEADERSHIP COUNCIL

MARCH 2004
www.corporateleadershipcouncil.com

LITERATURE K EY FINDINGS

Practical Examples of Measuring Training

Effectiveness
TRAINING ROI CALCULATIONS

Cost-benefit analysisVulcan Materials, a US building materials and construction

chemicals supplier, conducts cost-benefit analysis for many training programmes
through the following methods: 1

Before trainingPrior to the start of a training programme, the organisations training staff
asks middle managers to estimate the savings that will result from a proposed training
intervention in their departments.

The managers also assess, on a scale of 0 to 1.00, their confidence that the training will
produce s avings.

Trainers then multiply savings by confidence ratings to obtain an estimate of total savings
they can expect from the course.

After trainingFollowing training, evaluators review actual cost savings or revenue

increases.

Line managers then estimate, by assigning a percentage, the extent to which they believe
the improvement(s) can be attributed to training.

The organisation multiplies actual savings or increases by line managers estimates, and
compares the resulting figure to the forecast of total saving from the course.

To calculate the ROI of training

companies can use the following
methods:

Business impact analysis

Cost benefit-analysis
Impact of training on performance
improvement
Revenue increases through
training
Value added per participant

Training ROI FormulaMerck & Company uses a unique formula, detailed below, to
measure the impact of specific training programmes. With this model, Merck determined
that its average ROI for training programmes is 84 percent and terminated 53
programmes that were not producing high enough returns.2
GAIN = D x SD$ x JSI x N
D=shift in performance by average individual undergoing training expressed in standard deviations
from pre-training average
SD$=the value in dollars of one standard deviation of performance shift
JSI=percentage of job skills affected by training
N=number of participants who underwent training

Training expenditure calculationsFinancial services companies profiled in past

Council research use the following metrics to evaluate their training functions:3

Training expenditures as a percentage of total payrollIndicates the amount of training

investment relative to labour costs.
Training expenditures per employeeIdentifies the average dollar amount spent on training
per employee. Allows HR professionls to compare a companys investment in human
development to industry competitors.
Training hours per employeeMeasures the temporal amount of training each employee
receives on average. Provides organisations with a non-financial quantity measure, allowing
them to adjust for different uses of training investment while benchmarking.
Trainer-to-employee ratioMeasures the number of employees that each training employee
supports in the organisation.

Specific training metrics at profiled financial services companies follow in the table
below.
TRAINING STATISTICS AT PROFILED FINANCIAL SERVICES COMPANIES
Profiled
Company
A
B
C

Training
Expenditure as
a Percentage of
Total Payroll
2 percent
2 percent
3.9 percent

Training
Expenditure Per
Employee

Training Hours
Per Employee

Trainer-toEmployee Ratio

1,515
457
1,355

n/a
31.5
131.6

1:200
1:144
1:88

2004 Corporate Executive Board

CATALOGUE NUMBER: CLC11W5Y7H

CORPORATE LEADERSHIP COUNCIL

PAGE 2

PRACTICAL EXAMPLES OF M EASURING T RAINING EFFECTIVENESS

KEY FINDINGS

TRAINING ROI CALCULATIONS (CONTINUED )

By assigning numerical values to
pre- and post training employee
performance, companies can
calculate the value that training
interventions add to each
participant.

Value added per participant calculationA medical products company profiled in past
Learning and Development Roundtable research uses the following process to assess
the ROI of training:4

Employee assessmentPrior to any training intervention, HR and business unit managers

prepare job-specific performance criteria, assigning different importance scores to specific
dimensions of a given job. Next, business unit managers assess their employees current
levels of performance along individual job dimensions, using a scale from 1 to 5. The scale
requires that managers assess employees relative to their peer group, where 1 equals worst
performance, 3 equals average performance, and 5 equals the best performance that the
manager has ever experienced.

Performance improvement ROI surveyThen business unit and HR managers complete a

performance improvement ROI survey to jointly place a dollar amount on the value of one
standard deviation in employee performance (e.g., if a sales representative currently assessed
at 2 were to move up to 3 after sales training, what would be the dollar value of that
improvement?). The survey takes into consideration the relative value of different jobs in the
corporation to overall financial results as well as the relative value of individual skills.

Assessment of performance changes after trainingOnce employees complete their

training, HR re-surveys business unit managers to assess employee performance changes.
HR then calculates the value-added per participant based on their performance changes and
the previously determined dollar value of those performance changes. In the final step, HR
completes the ROI calculation, considering the total number of training participants, their
post-training performance change, the value of that performance change, the duration of
performance gain in years, and the total cost of the training activity.

As a result, the company realised a 1,200 percent ROI for its technical training
curriculum, and 600 percent for sales training. However, the analysis revealed negative
return on most corporate management training. The company eliminated its courses
with low or negative results, saving the company millions in training.

Isolation of training impactAccenture analysed the ROI of all its training initiatives
for 261,000 employees over the history of the company to learn that the ROI of training
was 353 percent.5

Accenture used a patent-pending technique which forced out the effects of inflation, market
cycle, experience, and employee level to isolate the training impact on a per person margin
Accenture determined a return of 2.91 in net benefits for every euro invested in training.

Business impact of trainingIn addition to conducting end-of-course or training

programme questionnaires , Boots The Chemists measures the impact of training on
the bottom line. For instance, sales staff training in product knowledge is measured by
its impact on sales, while the success of systems and operations training is measured in
terms of reduction in time spent by staff on routine tasks.6

KIRKPATRICK MODEL EVALUATIONS

Training evaluations with the
Kirkpatrick model provide insight
into the impact of training on
employee satisfaction, knowledge,
behavioural change, and business
results.

All Kirkpatrick levelsConsignia (now Royal Mail) uses the Kirkpatrick model of
training evaluation to monitor its e-learning programme. The company assesses
different levels of the model as follows:7

Level OneOnline tool asks questions such as what participants thought of the product and
how they accessed it. The company also employs this tool to assess levels of usage.
Level TwoTool incorporates tests of level of knowledge during course
Level Three and FourSponsors and training professionals are responsible for measuring
these levels. In some courses, such as the marketing excellence programme, the company
undertakes follow -up work to measure the level of content understanding and how participants
have applied their learnings within their role.

2004 Corporate Executive Board

CORPORATE LEADERSHIP COUNCIL

PAGE 3

PRACTICAL EXAMPLES OF M EASURING T RAINING EFFECTIVENESS

KEY FINDINGS

KIRKPATRICK MODEL EVALUATIONS (CONTINUED )

Level Three evaluationArthur Andersen (now Accenture) uses video cameras to

conduct a Level Three evaluation of a two-day course for executives on the essentials of
giving good business presentations . Executives were first taped giving presentations
prior to the course. Next, they were taped at the conclusion of the course. Finally, the
executives were taped six months later as they delivered presentations to clients.
Evaluators graded each tape against a set list of performance indicators.8

Level Three evaluationFor a leadership training course, evaluators at Motorola use

360-degree evaluations to measure Kirkpatricks Level Three. Evaluators send surveys
to trainees, trainees supervisors and their subordinates that ask all three groups to rate
the trainees on the frequency with which they demonstrate certain behaviors associated
with leadership. The survey is conducted four times a year for two years, and evaluators
attempt to identify an increase in scores.9

Level Three evaluationHutchinson Technology conducted Level Three evaluations

by assessing employees skill levels prior to training by surveying customers, supervisors
and peers on how frequently employees displayed particular behaviors. After training,
the company surveyed the same individuals using a similar survey. The survey revealed
that employees used the selected skills approximately 75 percent of the time, as
opposed to 50 percent of the time prior to undergoing the training.10

ALTERNATIVE TRAINING EVALUATION MODELS

Training waste analysisJeffers Company seeks alignment by ensuring that

development investments produce the greatest possible benefit. Jefferss learning
function developed a cost analysis process for identifying the largest categories of waste
in internal training administration. For example, Jeffers identified no shows to training
as a prominent cost driver, since excessive registrations led, in the past, to overscoping
of capacity and unnecessary cost outlays. Addressing this issue helped Jeffers
substantially reduce the cost of training delivery. The company tracks metrics such as
cost of classes, number of participants, empty seats, and cancelled classes.11

Participant reaction assessmentThe Travelers Companies use various qualitative

and quantitative methods to evaluate training programmes. For example, participants
reactions are immediately captured after a training programme through evaluator
observations and post-course group discussion. If more than 25 percent of participants
express dissatisfaction with some aspect of the programmes as it relates to the job, a
committee consisting of line managers, instructional designers, an evaluator, and
instructor review the evaluation data to determine a course of action. 12

Qualitative training evaluationTraining evaluation at Johnson & Johnson focuses

on training participants change of behaviour on the job. The company bases most of its
training decisions on qualitative data acquired from worldwide employee surveys , such
as six-month post-training self-reports, annual surveys, and supervisor questionnaires.
For example, to assess training programmes for first-line supervisors participants have
to define their perceptions of change in relation to a set of critical supervisory skills. In
addition, participants receive evaluations from their supervisors as well as from five to
eight subordinates via a survey. 13

Telephone surveys and e-mailsThe Defence Evaluation Research Agency

(DERA) (now Defence Science and Technology Laboratory) evaluates its training
courses through various methods. Systems for feedback include the usual end-ofcourse questionnaires prepared by participants as well as the following:14

Measuring training waste such as

number of empty seats and
cancelled classes can help
organisations to optimise their
training resource allocation.

Qualitative training evaluations

such as telephone surveys, trainer
observations, and longer e-mail
responses can help organisations to
gain a perspective on employees
feelings about their training
offerings.

Evaluation of outcome forms completed by resource managers and business group managers
for members of their team to ensure that training offerings meet and satisfy key areas
More reflective longer e-mailed responses from participants
Telephone surveys by training staff of 10 percent of participants to ask whether original
objectives have been met and the desired learning outcomes been achieved

2004 Corporate Executive Board

CORPORATE LEADERSHIP COUNCIL

PAGE 4

PRACTICAL EXAMPLES OF M EASURING T RAINING EFFECTIVENESS

KEY FINDINGS

ALTERNATIVE TRAINING EVALUATION MODELS (CONTINUED )

Online training databaseMCI Communications maintains a corporate training

database that combines information from the companys training sites. Using a PCbased information system , each training department uses software to evaluate its own
training functions. A training manager can review, for example, an individual employees
training history to identify gaps in development. The system can also accumulate
training cost data which the company uses to determine the ROI of training both
companywide and by business unit.15

Course review triggers in online databaseA company profiled in past Learning and
Development Roundtable research collects course evaluations in a single database
based on a consistent set of metrics related to course relevance and effectiveness; any
time a course scores below a pre-defined quality standard, a trigger is pulled for review
of that course and its instructor. The manager responsible for a particular training
course scoring below the threshold is accountable for presenting to training
management an assessment of what caused the low evaluation scores and
recommendations for improving or eliminating the course.16

Balanced scorecardQwest Communications uses a balanced scorecard that

continually tracks financial and operational performance of the learning function and the
business units with which it has partnered. The company is able to evaluate the
effectiveness of its development interactions by measuring the following categories
within the balanced scorecard:17

Training databases and balanced

scorecards provide companies with
a framework for tracking key
indicators of training effectiveness.

Client impactStudent ratings, content relevance and compression

FinancialUnit costs for assessment, development and delivery; productivity: delivery volume
versus headcount increase
OperationalTotal training hours, classroom yield, enrollment volume
People Percentage of development plans completed, L&D staff satisfaction and engagement

E-Learning MetricsCisco Systems tracks a number of metrics specific to its

extensive e-learning offerings across the organisation. The impact of specific learning
opportunities on employees is measured using the metrics outlined below:18

Pre-test and post-test scores per offering

Time to completion (i.e., time and visit required to complete individual offering)
Time to competency as measured by post-test and on-the-job behaviour against course
objectives
Change in learners ability to perform job responsibilities against time spent learning

2004 Corporate Executive Board

CORPORATE LEADERSHIP COUNCIL

PAGE 5

PRACTICAL EXAMPLES OF M EASURING T RAINING EFFECTIVENESS

KEY FINDINGS

BNA's Human Resources Library on CD, [CD-ROM], "Training Program Assessment and Evaluation.
Learning and Development Roundtable. The Productivity and Efficiency Imperative.
Washington: Corporate Executive Board (December 2001). (Unfortunately, we are unable to provide a copy of
this research since your company is not a member of the Learning and Development Roundtable).
3
Corporate Leadership Council. Benchmarking Training Statistics.
Washington: Corporate Executive Board (April 2002).
4
Learning and Development Roundtable. The Productivity and Efficiency Imperative.
Washington: Corporate Executive Board (December 2001). (Unfortunately, we are unable to provide a copy of
this research since your company is not a member of the Learning and Development Roundtable).
5
Schettler,J., Barbian,J., Galvin,T., Dolezalek,H., and Johnson,G. "The 2003 Training Top 100.
Training (March 2003). (Obtained through Factiva).
6
Author Unknown. Training Strategies. IDS Study 679 (November 1999).
(Due to copyright restrictions we are unable to provide a copy of this research).
7
IRS Employment Review. Delivering an E-learning Package. IRS Employment Review 753 (June 2002).
(Obtained through www.irsemploymentreview.co.uk).
8
Geber,B. Does Your Training Make a Difference? Prove It! Training (March 1995).
(Obtained through LEXIS-NEXIS).
9
Geber,B. Does Your Training Make a Difference? Prove It!
10
Geber,B. Does Your Training Make a Difference? Prove It!
11
Learning and Development Roundtable. Reframing the Measurement Deb ate: Moving Beyond Program Analysis in the
Learning Function. Washington: Corporate Executive Board (2002). (Unfortunately, we are unable to provide a copy of
this research since your company is not a member of the Learning and Development Roundtable).
12
BNA's Human Resources Library on CD, [CD-ROM], "Training Programme Assessment and Evaluation."
Available: The Bureau of National Affairs, Incorporated.
13
BNAs Human Resource Library on CD. [CD-ROM]. Training Programme Assessment and Evaluation.
14
Author Unknown. Training Strategies. IDS Study 679 (November 1999).
(Due to copyright restrictions we are unable to provide a copy of this research).
15
BNAs Human Resource Library on CD. [CD-ROM]. Training Programme Assessment and Evaluation.
16
Learning and Development Roundtable. The Productivity and Efficiency Imperative.
Washington: Corporate Executive Board (December 2001). (Unfortunately, we are unable to provide a copy of
this research since your company is not a member of the Learning and Development Roundtable).
17
Learning and Development Roundtable. Reframing the Measurement Debate: Moving Beyond Program Analysis in the
Learning Function. Washington: Corporate Executive Board (2002). (Unfortunately, we are unable to provide a copy of
this research since your company is not a member of the Learning and Development Roundtable).
18
Author Unknown. Cross-Functional Task Force to Measure Effectiveness and Impact of e-Learning. Cisco (2000).
(Obtained through http://formare.erickson.it/archivio/maggio_giugno/cisco.pdf).
2

2004 Corporate Executive Board