Training on Trial

Mid New Jersey Chapter, ASTD

May 21, 2009

Presented by:
Jim Kirkpatrick , PhD
 The future of training
“Training directors might be well advised to
take the initiative and evaluate their
programs before the day of reckoning
arrives”

- From Techniques for Evaluating Training

Programs, by Don Kirkpatrick
- ASTD Journal, November,
- 1959
What are the charges?
“You, learning
professionals, are
charged with
incurring training
and development
costs that exceed
the value you bring
to the business”
Who exactly is the “we” who are
being charged?

The training industry

Organizational Development
Learning functions
HRD
Training departments
Us
Who is charging us?

Internal business
partners

External clients

The business world

How do we know we are being
charged?
“We would like you to show us your ROI”
“We know you run a lot of programs and
seem quite busy. Why then are so
many good people leaving us?”
“We are thinking we need to put our
efforts in revenue-producing activities.”
“We are going to put that on hold for
awhile.”
“We have decided to make some cuts.”
What evidence is being used
brought against us?
Brinkerhoff Study
Training Application

Did not try new skills: 15%

Tried new skills and failed: 70%
Achieved sustained new behaviors: 15%

- Josh Bersin and Associates, 2008

 Percentage of learning transfer

1975: Percentage of formal learning that is

actually applied to the job:
15%

2005: Percentage of formal learning that is

actually applied to the job
15%
From Dana Robinson, ASTD ICE, 2008
Causes of “training failure”

Preparation and Readiness: 20%

Learning Intervention: 10%
Application Environment: 70%
2006 ASTD Study
Typical Learning Investment

Pre-Work10%
LearningEvent85%
Dr. Brent Peterson, Columbia
Are we guilty?
What do we have to do to be found
not guilty?
We need to extend our role beyond the
traditional definitions of training and
learning
The increase in “informal
learning”
Where Learning Takes Place

Prior to being "trained": 20%

During "training": 10%
On the job: -70%
Josh Bersin and Associates, 2008
 Lets get very practical here
We need to redefine and expand our role
through the entire “learning and
performance” management spectrum:
– Our expertise
– Our involvement
– Our influence
– Our impact
– Our value
How does this relate to you
so far?
Who will take our case?
 Measurement and Evaluation
Gathering data and information to answer specific
questions regarding the value of learning and
performance solutions; focusing on the impact of
individual programs and creating overall measures of
system effectiveness, leveraging findings to provide
recommendations for change and to increase
organizational effectiveness.
- taken from Mapping the Future, ASTD, 2004
 Why Evaluate?
1. Improve the program (or know when to
retire it)
2. Maximize training effectiveness by
reinforcing knowledge, skills, and
attitudes
3. Demonstrate the value of training

p. 3
 Kirkpatrick Four Levels
Level 1: REACTION
To what degree participants react
favorably to the training

p. 4
 Kirkpatrick Four Levels
Level 2: LEARNING
To what degree participants acquire
intended knowledge, skills, and
attitudes based on participation in
learning event

p. 4
 Kirkpatrick Four Levels
Level 3: BEHAVIOR
To what degree participants apply what
they learned during training on the job

p. 4
 Kirkpatrick Four Levels
Level 4: RESULTS
To what degree targeted outcomes occur,
as a result of the learning event(s) and
subsequent reinforcement

p. 4
 Four Level Correlations
Enjoyment (L1) and learning (L2): no correlations
Relevance (L1) and immediate learning (L2): r = .26
Relevance (L1) and transfer to job (L3): r = .18

Immediate knowledge (L2) and retention (L2): r = .34

Immediate knowledge (L2) and skill demo (L2): r = .18
Immediate knowledge (L2) and transfer to job (L3): r = .11
Retained knowledge (L2) and transfer to job (L3): r = .08
Skill demo (L2) and transfer to job (L3): r = .18
 Kirkpatrick Four Level
Evaluation major principles
• Smile sheets, pre and post-tests, and
hope for the best
• Strong connection between Levels 1 and
2
• Strong connection between Levels 3 and
4
• “The Great Divide”
• “The Missing Link”
Let’s take some initial depositions . . .
“What is your job here at the hotel?”
“I am a window washer.”
 “What is your job here at the resort?”
“I am part of a team that creates great experiences
for our guests!”
 Group Activity
What is significant about these two window
washers in relation to:
Group 1: individual employees?
Group 2: training and development
professionals?
Group 3: the training and development
industry?
Crossing over is not easy
What does the word “charged”
mean?

• Accused

• Challenged
 Common Myth
“Evaluation” only occurs at the end of the
instruction design process, or at the end
of a program.
 Reality
An effective evaluation plan is considered at
every step in the program and training
development process, not JUST at the
end.
 Kirkpatrick Four Level
Evaluation major principles
• The end is the beginning
• ROE is the ultimate indicator of value
• Business partnership is key
• Value must be created first
• Demonstrate value through a compelling
chain of evidence
Kirkpatrick Four Level
Evaluation Model

Reaction

Learning

Behavior

Results
 The Kirkpatrick Model
Results Behavior Learning Reaction
Negotiate stakeholder Determine required
Business Identify critical Consider necessary
success indicators, KSAs,
need behaviors and key learning environment
Business outcomes Learning Objectives
identified organizational drivers and conditions

Design and build learning program and evaluation tools

Deliver learning program

Measure L2 Learning Measure L1 Reaction

Initiate ongoing
reinforcement and
monitoring

Measure L4 Results Measure L3 Behavior

,t s uj da, s gni dni f ezyl an A
yr ass ec en s a s pet s t aeper

Gather final four level data / information, prepare for presentation

Present L4 Results Present L3 Behavior Present L2 Learning Present L1 Reaction

ROE
findings findings findings findings

p. 22
 DOT Case Example

Challenge – to increase highway work zone

efficiency, effectiveness, and safety.
 The Kirkpatrick Business
Partnership Model
1. Identify who is on your corporate jury
2. Negotiate jury (stakeholder) expectations
3. Convert expectations to Level 4 outcomes and
determine metrics
4a. Identify the key new participant behaviors that will be
required to bring about the desired outcomes. If L3
behaviors are not relevant, choose L2 ongoing
readiness factors
4b. Identify the few key organizational drivers that will be
necessary to bring about the changes in behavior
5. (Determine and implement prerequisites)
 DOT Example
1. Jury – safety managers, district engineers, traffic
control, risk managers, maintenance supervisors.
2. Expectations – smooth traffic flow; increased safety for
workers and motorists in highway work zones; better
planning.
3. Success Outcomes – less than 15 minute delays;
reduction in injuries and fatalities; increased
compliment to complaint ratio.
4a. Key New Behaviors – written work zone plans; plan
communicated to public; proper work zone set-up;
flaggers following procedures.
4b. Success Drivers – observation, feedback and coaching
by supervisors; ongoing compliance tracking; ongoing
execution of formal and informal recognition programs.
Airborne Warning and Control
System
 Activity #2
1. Break up into small groups as instructed.
2. Discuss and select one high impact
training program from your group
members to work with.
3. Develop your answers to steps 1-4. Use
flip chart pages.
4. Be prepared to share your results.
 Measurement Strategy

Training
Critical Busines
-related Demonstr
Behavio s
Activitie ated SKA
rs Results
s

Level 1 Level 2 Level 3 Level 4

Did the target Did the activities build Did the target audience Did the critical
audience react sufficient capabilities exhibit a sustained behaviors
favorably to the to drive the critical behavior change in the contribute to a
activity? behaviors? work environment? positive impact on
business results?
 Business Partnership
Strategy

Culture Structure

Critical
Business Activitie
Behavior
Results s
s
HR Business
Processe Processe
s s

Talent
Preparing for and presenting
our case
Developing a Plan to Evaluate:
Critical Criteria
 Evaluation methods
Evaluation Levels
1 2 3 4
Methods Reaction Learning Behavior Results

Survey ● ● ● ●
Questionnaire/Interview ● ● ● ●
Focus Group ● ● ● ●
Knowledge Test/Check ●
Work Review ● ●
Skills Observation ● ●
Presentations / Teach Bk ● ●
Action Planning ● ● ●
Action Learning ●
Key Business HR Metrics ●
 Kirkpatrick Four Level Evaluation
Feedback Loop to Eliminate ‘Snags’
Delivery of Learning Event 1
L1 & L2 Formative Evaluation

Level 1
Improve Session Post Session L1 2
4

Was session delivered effectively? 3

NO
YES
5
Level 2
Post Session L2

10
Did targeted learning occur? 6
NO
Improve YES
Reinforcement Post Session Reinfcmt,
and/or Coaching 7 Level 3
Coaching & L3
YES
9
Was failure due Was there successful transfer 8
NO to session? NO of learning to behavior?
YES
Level 4
Expected results will occur.
11
Build your chain of evidence & showcase
 Example: Valdez University
strategic goal scorecard – goal #1
Up from last
month
Down from last
month Metric Actual Target Status
Same as last
month 1. Level 1 satisfaction 93% 90%
scores - aggregate
2. % courses learning 85% 70%
Help to move objectives matched to
our new directive
organization 3. Level 2 skills scores - 92% 90%
from aggregate
transaction-
oriented to 4. % leaders certified as 66% 65%
‘trusted trusted advisor coaches
advisor’ 5. Level 3 scores - 48% 75%
aggregate
6. Gallup scores - aggregate 77% 90%
Preparing for our closing
arguments
 Chain of Evidence

Level 1 Level 2 Level 3 Level 4

Reaction Learning Behavior Results

Gather data at all four levels and show that your

training delivers true value to your organization.
Data is a critical part of our
Chain of Evidence
But do not underestimate the
power of testimonials
What will you do as a result of what you
have learned today . . .
that will lead you to the following tomorrow?
that will hopefully lead to . . .
 References
• Kirkpatrick, D.L. and J.D. Evaluating Training
Programs, 3rd Ed., Berrett-Koehler Publ., Inc. San
Francisco, CA, 2006
• Kirkpatrick, D.L. and J.D. Transferring Learning to
Behavior: Berrett-Koehler Publ., San Francisco, CA,
2005
• Kirkpatrick, D.L. and J.D. Implementing the Four
Levels, Berrett-Koehler Publ., San Francisco, CA,
2007
• Kirkpatrick, J.K., and W.K. Training on Trial,
AMACOM, New York, 2009
• Contact Jim at jim.kirkpatrick@smr-usa.com
• www.smr-usa.com