Beyond ROI - 7 Levels of Training Evaluation

In today''s business world, customers and society expect much more than just return on investment. If you wish to score 10 out of 10 training must demonstrate that it helps a business become more sustainable while also contributing to the customer and society. Over the last four decades, training evaluation has been well served by the ground breaking work of Donald Kirkpatrick and Jack Phillips. Together they have helped shape the nature of training evaluation. Their work can be summarized as including five levels of evaluation. Kirkpatrick setting the foundation for levels 1 to 4 and Phillips supplying the fifth. Level 1 - Reaction - measures individual reaction to a course or experience. This form of evaluation is typically the most common used in business, probably because it is perceived as the easiest to do. This form of review is normally seen as an evaluation sheet at the end of a training course. Here we can discover whether a person had a good time and whether it was meaningful. This lowest form of evaluation can raise more questions rather than answers. For example just because someone had a wonderful time does it necessarily mean that they learnt something valuable. The reaction method can also be used to evaluate on the job coaching and e-learning. Level 2 - Learning - here attention shifts to what knowledge skills and attitudes have actually changed or have been acquired during an experience. Commonly changes in learning are determined with some combination of pre and post assessment as well as direct observation and testing. Sometimes this evaluation will resolve whether new knowledge or learning has been put to use but generally the discussion is quite shallow. Level 3 - Behavior - measures real change as a result of the initiative being undertaken. This normally involves direct observation of peoples'' performance on the job. For this level of evaluation to be successful it requires assessment which is accurate and unbiased. The assessors are normally the managers, coaches or mentors of the people actually performing the work. To do this well, training and good education on assessment is vital. A skilled assessor will not only note behavioral change but also which factors may be inhibiting them doing the job in the first place. For example a lack of resources, unclear policy or some other hindering factors. Level 4 - Business Results - identifies how learning and knowledge is delivered to the business. This would typically involve assessment against measures such as productivity, customer retention, sales and profitability to name a few. Depending on the nature of your business, your interest in knowledge and innovation this measure will vary. In undertaking this evaluation care must be taken to separate those factors which have nothing to do with the initiative. For example, external pressures such as changed market conditions. Level 5 - Return on Investment - shows the cost versus benefits of an initiative. In undertaking such a study you will discover the linkage between learning and improved profitability, efficiency

and effectiveness. For example, did the dollar value of the produced benefits exceed the cost of hosting and setting up an initiative. Here some creativity will be required to quantify the hard to measure competencies and capabilities. In most cases some measurement can be placed on the data you are receiving. The key here is to be clear about your assumptions you are using. There are a number of ratios and metrics that can be used, one simple measure recommended by Jack Phillips is: ROI % = Total Benefit ( in $) x 100 Total Program Cost From the launching pad of return on investment two new levels of training evaluation are required. The sixth is sustainability and the seventh is shared purpose. This extension is consistent with world trends for greater accountability, business governance for social, environmental and economic goals. Or in other words just because training achieves an excellent return on investment it may be failing to deliver in other important areas. For example neglecting broader responsibilities to the customer, society or the community. We need inly to remember the fall out from the recent demise of Enron and World Com, to add extra momentum to the pursuit for better and higher levels of evaluation. So let us explore level 6 and 7. Level 6 - Sustainability - at this level of evaluation we shift our attention to more outward focused questions on ensuring lasting success. In level 6 we are most interested in making sure the capabilities and competencies being learnt are actually helping a business prepare for the future? To do this we need to pay closer attention to the context and environment of change and be receptive to new and more balanced views of what excellence means. Typically, businesses that invest in their people and systems have a good eye on their environment and have a higher chance of measuring success here. The core of sustainability is dependent on competitive intelligence, excellent marketplace scanning and agility. Level 7 - Sharing The Benefit - The highest and most altruistic level of evaluation is asking whether your business know-how is adding value and helping others, whether it is your suppliers, customers, partners or society as a whole. Modern business is increasingly expected to be accountable for its actions and be a good corporate citizen not just for our current generation but for future ones as well. So we must be prepared to have independent assessment as well as contribute to the quality of life of others. . As Dr. Ronald Forbes, from the Leaderskill Group in Sydney reminded me in a timely e-mail. If the benefit of what you and your business do does not add to the wellbeing of the planet, society or the client you should scrap it. In closing, it is important to note that not all measurement and evaluation is good. In fact if we are not careful, what we produce can be inappropriate, meaningless or inaccurate. Just because you are getting great scores, numbers on a balance sheet or you feel good does not necessarily mean you are discovering what you need to know. We have to have a discipline that stretches our boundaries of the known and unknown to reveal the truth.

Excellent evaluation must be backed up with careful planning, transparency and consultation. Training must be prepared to deploy a range of measures without being guilty of producing bad or misleading information. Deploying the right spirit is vitally important. Treat the process of evaluation seriously and you will discover the insights you need to help provide the services and products you desire in a smarter, faster and better way.