You are on page 1of 70

The ADDIE Model

an Instructional Generic Design Model

Strategy and Tactics of Task Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

The ‘Design’ Phase of the ADDIE Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Developing Assessments of Trainee Proficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Strategy and Tactics for Pilot Implementation


in the ADDIE Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Strategy and Tactics of Training Recordkeeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Formative Evaluation in the ADDIE Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Final Implementation of the Training Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55


About the Author
Gordon Welty, Ph.D. has been designing, developing, facilitating, evaluating, and managing technical and regulatory training programs
in the healthcare and pharmaceutical industries for more than 20 years. He is a member of the Journal of GXP Compliance’s Editorial
Advisory Board. Contact Gordon at gwelty@wright.edu.

BONUS ARTICLE:
Suggestions for Better Training Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
About the Author
David Jones is a GMP specialist with 13 years of experience assisting pharmaceutical and medical device manufacturers with training,
audits and quality system improvement. He formerly provided a training module to more than 300 FDA field personnel to introduce
them to the concept of quality system audits. David worked for 20 years for the A. H. Robins Company (since acquired by Wyeth) in
various roles including business unit and division management. He can be reached via email at cgmpman@aol.com or by telephone at
804 350 5410.
Gordon Welty, Ph.D.

Strategy and Tactics of Task Analysis

tion. Should problems be identified with the facility or the


Task analysis is the process of defining the discrete steps, or tasks
that, when executed, will ensure the effective and efficient perfor- organization, the typical remedy would not be training – it
mance of a job. Careful task analysis can add substantial value to would either be a problem-specific (safety, environmental,
an organization’s training courseware. etc.) corrective action2 or it would be an organizational de-
velopment (OD) initiative.3 The organization and its facil-
ity are important for the analysis of training needs because
INTRODUCTION they situate employees and the tasks they are to perform.
This article reviews the place of task analysis in the process The facility and organization can have a profound impact
of developing a standard operating procedure (SOP) and on training needs and training effectiveness.4
the subsequent training module. Task analysis is the process In considering the required attributes of the employee
of defining the discrete steps, or tasks that, when executed, within the organization, we consider the skill-set and the
will ensure the effective and efficient performance of a job. employee’s disposition. The skill-set is encompassed by the
We look initially at strategic issues and then review some employee’s curriculum and training plan, which indicate
tactical concerns. First, we will consider the strategic re- the employee’s completed training as well as competencies
lationship between the analysis of training needs and task and qualifications. The dispositions include the employee’s
analysis; then we will examine how task analysis fits into personality, insofar as it impacts on work-related attitudes,
the ADDIE model of instructional design. ADDIE is the ac- which contribute in turn to motivations and, ultimately,
ronym of a generic instructional design model, comprised behaviors. Two different and important motivations are an
of the following: Analyze, Design, Develop, Implement, and employee’s motivation to participate in the company’s estab-
Evaluate. Turning next to tactical matters, we will review lished training program, and his or her motivation to trans-
how an instructional designer plans and prepares for a task fer the training to the job.5 The conjunction of skills and
analysis, undertakes a task analysis, identifies best practices, dispositions situated in a particular facility and organization
completes the task analysis, and incorporates the data into a creates the employee’s workplace performance.
procedure. Throughout, we will pay special attention to the Finally, our analysis of training needs focuses upon the
importance of standards in any SOP.1 task. The task includes its component steps; the task itself
is included in a larger process within a sequence of tasks.
Analysis of Training Needs Of critical importance in an analysis of the task are the stan-
When we analyze training needs, we typically look at four dards that define the acceptable levels of task performance,
levels of analysis: the organization, its facility, the employee, those levels that will allow the organization to attain its de-
and the task. The organization includes organizational struc- sired outcomes.6
ture, business case, and supervisory factors; it also includes
the employee’s compensation and upward mobility. The fa- Task Analysis and Performance Gaps
cility of the organization includes such specifics as working Careful task analysis can add substantial value to an
conditions, workplace hazards, and utilities. The organiza- organization’s training courseware. The purpose of course-
tion and its facility are situational levels – they impact on ware – learning management system (LMS) software, struc-
training needs, but are not the object of training interven- tured on-the-job training (SOJT) module, etc. – is to close

 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

a performance gap for some group of employees. The term of the courseware. If the evaluation shows that the training
performance gap means that employee performance does module has shortcomings, for example, that the sequence
not meet some standard. of learning tasks is incomplete, those shortcomings are
Standards provide guidance for tasks and processes with- fed back to the author(s) of the courseware to be analyzed
in manufacturing and quality control lab systems, as well as again. Further design and development efforts follow, un-
in clinical operations, Quality Assurance (QA), biometrics, til the courseware meets the organization’s needs. Thus,
and pharmacovigilance. As a result of task analysis, best the ADDIE model has an evident affinity with the DMAIC
practices are identified, and can be mandated as standards. (Define, Measure, Analyze, Improve, Control)8 cycle of Six
These standards may be documented in procedures, pro- Sigma improvement efforts.
tocols, manufacturing orders, packaging orders, etc. For It is important to stress that the iterations within the AD-
example, if a manufacturing order in the pharmaceutical DIE model continue until management decides that organi-
industry stipulates that tablet thickness should be X mm, zational needs have been met. More generally, any relaxation
the pressing cylinder height parameter of an appropriately or deviation from the ADDIE model, such as a “satisficing”
qualified Fette Tablet Press is set to that standard. For brev- level of performance9 or the skipping of a phase, are man-
ity sake, all of these types of documents will be called stan- agement decisions. These decisions can take the form of cri-
dard operating procedures (SOPs).7 teria established before a project is initiated (e.g., a stopping
An SOP stipulates standards – and promotes consistent rule) or while the project is underway (e.g., “That product
practice – for task performance. Once these standards are has been refined enough.”)10
specified, it is possible to identify a performance gap, the The initial phase of the ADDIE model, the Analysis phase,
difference between existing employee performance and the refers to the analysis of four aspects of the process that will be
standard required by the SOP. Courseware can then be de- addressed by the courseware, be it a manufacturing facility,
signed to address the performance gap. Quality Control (QC) lab, clinical operation, or drug safety
Before the courseware has even entered the developmen- process. These are the analysis of the facility or context, the
tal process, the data derived from task analysis can be used analysis of the organization, the analysis of an employee’s
to revise and improve the SOP that is the basis of the train- skill-set and disposition, and the analysis of the task. These
ing. Thoroughgoing task analysis will, of course, add to four aspects – facility, organization, employee skills and
the overall cost of developing the SOP and the subsequent disposition, and task – are to be analyzed to identify the
courseware, but it is a cost that is well worth incurring, be- sequence of tasks, the sum total of the process, which will
cause the cost of quality training will be offset in fewer er- lead to the outcomes that the organization desires.
rors on the manufacturing floor. A process is a systematically organized set of inputs, tasks
or activities, outputs and standards – that pertain to, and
control, the inputs, tasks, and outputs.11 There are several
TASK ANALYSIS AND THE ADDIE MODEL different types of processes.
The ADDIE model is a generic instructional design model. The manufacturing or lab process may be a person-to-
It provides guidance at a fairly high level for instructional equipment (or instrument) process, a person-to-paper
designers, software engineers, etc., as they author and re- process, or a person-to-person process; it may be some
vise courseware. As we have noted above, the elements or combination of the three types. An example of a person-
phases of the ADDIE model are Analyze, Design, Develop, to-equipment process would be a technician calibrating an
Implement, and Evaluate. These elements are sequential instrument; a person-to-paper process would be a techni-
– each element depends upon the successful completion of cian recording the as-found and as-left data from a given
the preceding phase. calibration on the appropriate log; a person-to-person pro-
Moreover, the ADDIE model is an iterative feedback cess would be a line supervisor who is responsible to im-
model, which means that the results of the Evaluation phase mediately inform a technician when an instrument is avail-
are fed back, closing the loop, facilitating further refinement able for calibration. For each of these types of process, the

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 
Gordon Welty, Ph.D.

standards are captured in a document. Develop Project Plan


An SOP documents the standards and tasks that make The first step is to develop the project plan for the task anal-
up a process.12 A well-formatted SOP facilitates correct task ysis. The project plan delineates the process to be analyzed,
performance and consistent practice. It has a column identi- including its boundaries; the participants, including their
fying personnel or positions that are responsible for tasks, a levels of expertise, their responsibilities; and the intended
correlative “activities” column indicating the tasks and their outcomes of the task analysis; i.e., the expectations the par-
sequence, as well as the standards that define the satisfac- ticipants will have, stated in terms of behavioral objectives
tory completion of the tasks. Once standards are specified, or S.M.A.R.T. (Specific, Measurable, Achievable, Relevant,
the performance gap between existing employee perfor- and Time-based) objectives.
mance and that required by the SOP can be addressed. The Suppose, for example, the process to be documented was
courseware can then be designed to close the performance the manufacture of a petrolatum/wax-based topical oint-
gap; this remediation activity begins in the Design phase of ment, with active ingredient X. The boundaries of the pro-
the ADDIE model. cess would be Dispensing, on the input-side, and Distribu-
The question at this point is, How do we winnow through tion Center, on the output-side.
a number of possible standards and tasks to settle upon the For this illustration, the participants would be SMEs who
best practices for that process, and then document those manufacture the ointment; their responsibilities in the task
best practices?13 How do we write good SOPs? analysis include identifying relevant documents, generating
a list of tasks, identifying the responsible parties, etc.
For the illustrative process, the outcomes might include:
TACTICS OF TASK ANALYSIS identify relevant procedures and manufacturing orders,
Clearly, we must analyze the tasks before we can document develop a list of tasks (e.g., obtain raw materials, set up
the tasks. Hence, task analysis comes first; a good SOP de- equipment, clean and sanitize, prepare excipients, load
pends upon a good task analysis. Several questions about excipients, load active ingredient, mix according to pro-
methods must be addressed before a facilitator can under- cess parameters, transfer to holding tank, obtain product
take a task analysis. How does the author structure the task samples, fill tubes, seal tubes, inspect, package and store
analysis – will an informal approach or a more formal ap- product, etc.), develop criteria and protocols for identifying
proach be employed? How will the author collect the data responsible parties, and so forth.
on tasks? There are several options – operator behavior can The finalized project plan has several uses. It will help
be directly observed, critical incidents can be reviewed (a to make the business case for the task analysis, direct the
case-study approach), questionnaires can be developed review of existing documentation, communicate with the
(and, if so, can be distributed to various employees); ex- participants – the SMEs – and help to ensure the continuing
perts can be interviewed, etc. In this article, we will focus focus of the on-going task analysis.
our attention on a less formal approach, involving the in-
terview of experts. This approach is broadly applicable and Identify Subject Matter Experts
relatively inexpensive. The second step is to identify the SMEs. The author of the
SOP is not necessarily the expert, or even an expert, on the
There are a dozen steps to task analysis; we will discuss each in process to be documented. There may be several employees
turn.14 As with any well-planned project, we must first determine
who are SMEs on this process. It is important to identify
the scope of the task analysis. Once the scope is determined,
employees who are experts in the task (called Subject Matter
these SMEs, and to get the agreement of management, busi-
Experts, SMEs) are identified and contacted. Then, existing docu- ness owners, and other experts on the list of SMEs.
mentation is reviewed; this prevents duplication of effort. Finally, Among the criteria for identifying SMEs, the sum of their
the task analysis session is scheduled. These elements are the expertise should be comprehensive – it should cover the
responsibility of the author of the SOP or courseware, the facilita- whole manufacturing or lab process that falls within the
tor of the task analysis session. scope.15 It is important to select several SMEs, to ensure

 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

that a range (and perhaps variability) of expert opinion is From this point, as we undertake the task analysis session, the
included. Moreover, there are several levels of expertise that SMEs must be readied. Next, the tasks included within the process
must be identified; also task responsibilities must be identified.
should be represented – task analysis relates to proficiency
Then the tasks must be chunked and labeled to reflect the busi-
at the task or sub-task level, while domain expertise relates ness process, and the business order must be established within
to more comprehensive proficiency – the “big picture,” if and across chunks. Last, concepts need to be identified for each
you will.16 Also, a number of SMEs will make it more likely chunk.
that the industrial safety, environmental, regulatory, quality
assurance, and cultural aspects of the process are covered
in the task analysis. And they should be covered! Once the Prepare SMEs for Task Analysis
SMEs have been identified, the list of invitees needs man- When getting the SMEs ready for the task analysis session,
agement approval. it is important to remember that they are the key source of
information for documenting the process into an SOP. Treat
Review Existing Documentation them right!17 Make the SMEs comfortable with the agenda,
The third step is the review of existing documentation. The the room, and the building. Locate the restrooms, water
project plan of the task analysis will raise several questions, fountains, etc. Review the Emergency Response Plan and
including: Was there a prior version of the SOP? Are there evacuation routes. Discuss the project plan with them, and
similar SOPs with the same tasks and/or concepts? Are there then finalize it.
pre-existing task analyses that can be utilized?
Sometimes the author of the SOP, or management, will Identify Tasks in Terms of Outcomes
suggest that the SOP can simply be a revision of the prior Next, prepare the SMEs to identify the tasks in terms of
version, with no explicit task analysis. “Clean it up – add a the outcomes. This involves distinguishing between steps,
few steps here, delete a few steps there.” This is the “liter- tasks, and processes. A step is a discrete action (e.g., throw
ary” approach to writing an SOP, which runs the risk that a switch). A task is a set of steps that achieve a specific out-
the successively cloned versions of the SOP will take on a come (e.g., shut down the tablet press). A process is a set of
life of their own, taking a tangent that becomes ever more tasks that achieve a more general outcome (e.g., produce a
distant from the tasks and process they are intended to doc- batch of topical ointment). Once these have been clarified,
ument. This literary approach is sometimes called “writing the SMEs can begin to identify tasks.18
a procedure without leaving one’s desk.” It is not recom- As the group identifies and lists tasks, bear in mind the
mended – it is a major source of bad procedures. role of the facilitator of this task analysis session. Facilita-
Prior versions of a procedure should be reviewed as a pre- tors should focus on getting information from the SMEs
paratory step in a task analysis, not as an alternative to the rather than contributing themselves. The facilitator should
task analysis. The key source of information for document- provide direction as well; the session should focus on tasks,
ing the process is the SMEs who work within the process all not processes. The SMEs should, at a minimum, provide
the time, not a prior version of the SOP. action verbs and a direct object for each task statement.19 As
SMEs identify tasks, begin to map out the process. Encour-
Schedule Task Analysis age the SMEs to review and correct the process map as the
The fourth step is to schedule the task analysis session. This group proceeds; at this stage the process map is a work-in-
includes developing the agenda, sending invitations to the progress.
SMEs, reserving the room, and assembling all materials for
the session. Identify Task Responsibilities
Ask the SMEs to begin to identify task responsibilities.
To this point, we have specified the scope of the task analysis; we
have identified and contacted the SMEs; we have reviewed existing Which employee or position is responsible for each task?
documentation; and we have scheduled the task analysis session. Be alert for ambiguous responsibilities – there can be no
Now we are ready to undertake the task analysis session. task without a unique, responsible party. If it appears that

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 
Gordon Welty, Ph.D.

a task has no responsible employee, or has a number of groups. (This may be another signal that there are multiple
responsible employees, it is unlikely that the SMEs will be SOPs involved.)
able to resolve the issue. Each ambiguity should be noted
for later resolution by management. There may be a need Identify “Chunk” Concepts
to add “swim lanes” to the process map to distinguish the Finally, identify concepts for each chunk. Tasks describe
various responsibilities and collaborations. how to do something; concepts provide the science for task
Be alert for disproportional task responsibilities. If a po- performance. For example, “Completing the documentation
sition has only one responsibility in a lengthy procedure, of the log” might be conceptualized as “Making the calibra-
this may signal that the task should be moved entirely to tion process ‘Audit Proof’,” ensuring that there is a complete
another SOP. For instance, the task responsibilities of end audit trail for every calibration.
users, data managers, and system administrators in a typi-
cal database SOP are quite disproportionate. End users log To this point, we have prepared the SMEs for the task analysis; we
on, maintain the integrity of passwords, navigate the ap- have identified the tasks related to the scope; we have identified
plication, query, print pre-formatted reports, call the help the responsible party for each task; we have chunked and labeled
desk with problems, and log off. In addition to these tasks, the tasks to reflect the business process; we have established the
business order within and across the chunks; and last, we have
data managers enter, edit, and verify data, change fields,
identified concepts for each chunk. Now we are ready to complete
print customized reports, etc. In addition to all these tasks,
the task analysis.
system administrators maintain security, allocate rights, etc.
Rather than including all these tasks in a single lengthy “Da-
tabase SOP,” it is preferable to have several shorter proce- From this point, a draft process map will be prepared that includes
dures, one for each user group. The training implications the tasks, chunks, and concepts. Next, we will highlight best prac-
are less formidable, and the possibility exists of consolidat- tices by means of a peer review of the draft process map by the
SMEs (perhaps a subset) or other experts. Next, we challenge the
ing SOPs, for instance an “End user SOP” for a number of
draft process map on the floor or in the lab, to stress its real-world
applications. applicability. Finally, we revise the process map in light of these
In any case, identifying task responsibilities will prove critical inputs.
useful for a preliminary training audience list of employees
who must be trained on this SOP.
Draft Process Map
“Chunk” and Label Tasks The tenth step is to complete the draft of the process map.
When the task analysis session is underway, begin to chunk Start at a high level and work down, in an iterative fash-
and label the tasks to reflect the business process. Ask ion, to the desired level of detail. Be sure to include the
whether the tasks can be aggregated. For example, “Sign the beginning and the end of the process; check back to the
calibration log,” “Date the calibration log,” and “Have super- scope statement. Do not include anything that is outside
vision review the log” might be aggregated as “Complete the the scope. Keep a relatively uniform level of detail; do not
documentation of the log.” At this stage of the task analysis, have some aspects of the process mapped out in great detail,
it is important to chunk from the employees’ perspective; and other aspects merely sketched. Step back from the map,
use audience-focused words. Labels provide a rationale for as you are drafting it, to review its overall appearance, its
the chunks. Aim for about seven tasks per chunk. Be alert coherence.
for the need to map subtasks, as well as decision points
(and associated decision criteria), strategies, etc. Submit Process Map to Peer Review
Also, establish the business order for the chunks. What When the draft is ready, submit it to a peer review by some
is the flow within the process? If a clear logical order is not or all the SMEs, other experts, or the business owner. On the
available, use a generic order. If a single order is not pos- positive side, the reviewers should look for best practices,
sible, establish a unique order for each of the main user value-adding steps, flexibility in light of changing demands,

 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 1:
Illustrative task analysis of the steps and sub-steps involved in copying a document on a photocopier.20

1. Prepare photocopier
• 1.1. Switch on machine
• 1.2. Wait for warm-up cycle to be completed

2. Select desired number of copies

3. Prepare first page of original for copying


• 3.1. Raise lid
• 3.2. Locate page in appropriate position on the glass
• 3.3. Close lid

4. Activate copying cycle


• 4.1. Press start switch
• 4.2. Ensure that the original does not move

5. Check quality of photocopy


• 5.1. If OK, go to Step 6
• 5.2. If not OK, select appropriate corrective action
• 5.2.1. Put in more copy paper
• 5.2.2. Remove paper jam
• 5.2.3. Readjust position of original
• 5.2.4. Adjust toner setting

6. Remove copied original and replace with next page


• 6.1. Raise lid
• 6.2. Remove copied original
• 6.3. Replace with next page to be copied
• 6.4. Close lid

7. Repeat Steps 4-6 until all pages are copied

8. Remove last page of the original

9. Check that all pages have been copied satisfactorily

10. Switch off photocopier

11. Gather up all materials and depart

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 
Gordon Welty, Ph.D.

and scalability in light of changing output targets, etc. On out of task analysis. At a strategic level, we have examined
the negative side, they should look for bottlenecks in the the relationship between the analysis of training needs and
process, duplication of effort, unnecessary steps or tasks, task analysis. Then, we considered the place of task analy-
non-value-adding steps, role ambiguities (several positions sis in the ADDIE model of instructional design. Turning to
responsible for a task, or no one responsible for a task), etc. tactical issues, we have reviewed how the author of an SOP
Document all the points raised by the peer review. prepares for a task analysis, conducts a task analysis, and
Then, we test the real-world applicability of the process finalizes the task analysis and incorporates the data, includ-
map by challenging it step-by-step on the floor or at the lab ing best practices, into the SOP. The SOP is then used as a
bench. Select a seasoned employee within the scope – not point of comparison with employee task performance, to
an SME – and compare the process as mapped against the identify any performance gap. This points the way to the
employee’s activities. Do they align? Ask questions – look next phase of the ADDIE model, where courseware is de-
for evidence of resistance, repetition, human factors prob- signed to address that performance gap. GXP
lems such as task complexity. Document everything in the
challenge.
DISCLAIMER
Assemble, Evaluate, and Revise Process Map The content, views, and opinions expressed in this article
The last step in the task analysis is to assemble all the data are those of the author, and are not intended to express the
from this stage, evaluate it comprehensively, and revise the views or approach of Schering-Plough.
process map in light of it.

Now we have completed the task analysis. We have drafted the


ENDNOTES
1. We use the following terminology in this article:
process map; we have highlighted best practices; we have chal- Author: An instructional designer, the originator of a procedure, the facilita-
lenged the process map in a real-world setting; and finally, we have tor of the task analysis session, a technical writer, etc.
revised the process map in light of all these inputs. It is time to Courseware: A training module, LMS software, job aid, organizational devel-
opment program, etc.
seek management approval.
Management: Referring to either supervision in a line of business or the
supervision of a given author; which one will be clear in context
SME: A subject m atter expert in a task or process
2. See the excellent case-study in James L. Vesper, Performance: The Goal of
Once the task analysis has been finalized, and approved Training - or Why Training is Not Always the Answer, BioPharm (Eugene),
by management, the facilitator can translate the task analy- (Feb 2001), Vol. 14, Part 2, pp. 44-46.
3. See R. W. Tuli and G. E. Apostolakis, Incorporating Organizational Issues into
sis process map into the documentary form of the SOP, the Root-cause Analysis, Process Safety and Environmental Protection, (1996), 74, 3-16.
actual procedure, protocol, manufacturing order, packag- 4. Irwin Goldstein and Kevin Ford, Training in Organizations, Belmont, CA:
Wadsworth, 4th Ed. (2002).
ing order, etc. Many times, this translation will amount to 5. Jason Colquitt, Jeffrey LePine, and Raymond Noe, Toward an Integrative
the discursive writing-out of the process map captured in Theory of Training Motivation, Journal of Applied Psychology, (2000), Vol. 85,
No.5, 678-707.
Visio®. Any time the documentary form deviates from the 6. Kenneth Nowack, A True Training Needs Analysis, Training and Development
process map, the latter will provide the guidance. Journal, (April 1991), Vol. 45, No. 4, 69-73.
After the SOP has been developed, revised, and approved, 7. The FDA refers to these documents as “procedures” in 21 CFR 211, and as
“standard operating procedures” in 21 CFR 58.35.
the author can turn to the design of courseware that will 8. See Kevin Linderman, Roger Schroeder, Srilata Zaheerr, and Adrian Choo,
close any performance gap(s) that is evident in employee Six-Sigma: A goal-theoretic perspective, Journal of Operations Management,
(2003), 21, 193-203. Indeed, Kaliym Islam has recently suggested that the
task performance. Analysis phase of the ADDIE model be preceded by Define, and Measure,
phases, to ensure that “true business requirements” are specified, and appro-
priate metrics are selected from the outset. See his “Designing an Effective
Certification Program,” LTI Newsline (14 Dec 2005). The importance of
CONCLUSION identifying, and addressing, the business case for a proposed revision of an
We have considered how SOPs - and associated standards SOP, or for the subsequent training intervention, can hardly be overstated.
9. See Herbert Simon, A Behavioral Model of Rational Choice, Quarterly Journal
that allow us to identify performance gaps - are developed of Economics, (1955), Vol. 69, No. 1, 99-118; Richard Day, Profits, Learning,

 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

and the Convergence of Satisficing to Marginalism, Quarterly Journal of Eco- Questions, Training and Development Journal, (2000), Vol. 54, No. 10, 20-22;
nomics, (1967), Vol. 81, No. 2, 302-311; Alex Michalos, Rationality between Joseph Mattoon, Designing and Developing Technical Curricula: Finding the
the Maximizers and the Satisficers, Policy Sciences, (1973), Vol. 4, No. 2, Right Subject Matter Expert, Journal of Industrial Teacher Education (2005),
229-244; and Barry Schwartz, et al. Maximizing versus Satisficing, Journal of Vol. 42, No.2.
Personality and Social Psychology, (2002), Vol. 83, No. 5, 1178-1197. 18. Brenda Sugrue, Performance-Based Instructional Design for E-learning, Per-
10. This consideration obtains within the phases of the ADDIE model as well. For formance Improvement, (2002), Vol. 41, No. 7, p. 52 provides a Task Analysis
example, the “P x C” rule for continuing (or concluding) a task analysis, where Template that is useful for guiding and documenting the interaction with
the probability “P” of performance failure times the cost “C” of performance the SMEs.
failure, is estimated for a given level of granularity of the task analysis. If P x C 19. Darin Hartley, Job Analysis at the Speed of Reality, T + D, (2004), Vol. 58,
is unacceptably large, the task analysis continues to a more fine-grained level; No.9, 20-22.
see Neville Stanton, Hierarchical task analysis: Developments, applications, 20. James Reason, Combating Omission Errors through Task Analysis and Good
and extensions, Applied Ergonomics, (2006), Vol. 37, No. 1, 55-79; also Liu Reminders, Quality and Safety in Health Care, (2002), Vol. 11, No. 1, p. 43.
Xijuan, Wang Yinglin, and Jaing Shouwei, A metrics based task analysis model
for design review planning, Design Studies, (July 2003), Vol. 24, No. 4, 375-
390. Specifying the value of the estimates of P and C, as well as the criterion
of acceptability itself, is ultimately management’s decision, which highlights
the importance of ongoing communication between the facilitator of the task
Article Acronym Listing
analysis, the facilitator’s management, and the business owner of the SOP.
11. Mary Benner and Michael Tushman, Process Management and Technological ADDIE Analyze, Design, Develop, Implement, Evaluate
Innovation, Administrative Science Quarterly, (2002), Vol. 47, No. 4, 676-706. DMAIC Define, Measure, Analyze, Improve, Control
12. John DiLollo, The Use of SOPs in a Pharmaceutical Manufacturing Environ- GMP Good Manufacturing Practice
ment, Journal of cGMP Compliance (2000), Vol. 4, No. 3, 33-35.
LMS Learning Management System
13. Yasar Jarrar and Mohamed Zairi, Best Practice Transfer for Future Competi-
tiveness, Total Quality Management, (2000), Vol. 11, No. 4, 734-741. OD Organizational Development
14. While it restricts itself to e-learning topics, Conrad Gottfredson, Rapid Task QA Quality Assurance
Analysis, Learning Solutions e-Magazine, (June 2002), Issue 160 is very useful. QC Quality Control
15. Robert Gatewood and Hubert Feild, Human Resource Selection, Ft. Worth, TX:
SMART Specific, Measurable, Achievable,Relevant, Time-based
Harcourt, 5th Ed. (2001).
16. Charles Reigeluth, “Elaboration Theory,” in C. M. Reigeluth (Ed) Instructional SME Subject Matter Expert
Design Theories and Models, Mahwah, NJ: Lawrence Erlbaum (1999), p. 435. SOJT Structured On-The-Job Training
17. Peter Zvalo, Writer and Subject-Matter Expert - Establishing a Positive SOP Standard Operating Procedure
Relationship, Writer’s Block (Summer 2000); Dorothy Leeds, The Power of

Originally published in the April 2007 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 
Gordon Welty, Ph.D.

The ‘Design’ Phase of the ADDIE Model

The Design phase of the ADDIE model is where we address any Figure 1
performance gaps identified in the Analysis phase, complete the Addie Model Workflow
Training Outline, and secure management approval.

Analyze
INTRODUCTION
The ADDIE model is a generic instructional design model.
It provides guidance at a fairly high level for instructional
designers, software engineers, etc., as they author and re- Design
vise learning products. The phases of the ADDIE model are
Analyze, Design, Develop, Implement, and Evaluate. These
phases are sequential – each depends upon the successful
completion of the preceding phase. Develop
Moreover, the ADDIE model is an iterative feedback
model, which means that the results of the Evaluation phase
are returned to the origination point (fed back), closing the
loop, facilitating further refinement of the learning product. Implement
If the evaluation shows that the module has shortcomings,
for example, that the objectives of the module do not align
with organizational objectives, those shortcomings are re-
turned to be analyzed again. Further design and develop- Evaluate
ment efforts follow, until the module meets organizational
needs.
In this article, we will examine the three components of
the Design phase (see Sidebar, on facing page) in turn.
• The relationship between learning product and the
The Learning Product In The Larger Curriculum various regulatory requirements (e.g.: FDA, OSHA,
Fitting the proposed learning product into the larger curric- EPA, DEA, etc.)
ulum ensures the articulation of this product with all other
learning products, and the alignment of this product with The Structure of Modules
organizational goals. There are four aspects to this “fit” – The larger curriculum is comprised of a set of modules that
• The structure of modules focus the training effort on accomplishing organizational
• The relationship between the learning product and goals. The Design phase is where the fit between the pro-
the associated SOP posed learning product and the larger curriculum is delin-
• The learning product’s reduction by consolidation of SOPs eated. This means outlining the structure of the training

10 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

module wherein the learning product will fit. Each module

ADDIE Model
includes two types of learning product, an Overview Train-
ing element and one or more associated Skills Training ele-
ments.1 A module is configured as shown in Figure 2:
In the Design phase, the precise location of the learning ANALYSIS:
product – as an Overview Training element or a Skills Train- The Analysis phase of the ADDIE model identifies a
ing element – is determined. To briefly review the difference performance gap, a discrepancy between a standard
stipulated in a standard operating procedure (SOP)
and some employee performance. A performance
Figure 2 gap can be addressed by a learning product, i.e.: a
Structure of Training Module
set of training and assessment materials.

Overview Training DESIGN:


CBT or classroom: The Design phase follows Analysis, where a
presents the planned approach to addressing the performance
“science” gap is outlined and approved. This planned ap-
proach has three components:
• Fitting the proposed learning product into the
larger curriculum
• Outlining the proposed learning product
Skills or task • Securing management approval of the outlined
Skills or task
training: training: learning product
SOJT SOJT
DEVELOP:
If management approves the design, a Development
phase comes next, where the learning product - the
training materials and the assessment materials - is
between these two types of elements: the Overview Training developed to address the performance gap.
is more conceptually focused, while Skills Training is more
task or performance oriented. Concepts tell what a thing is, IMPLEMENT:
why it is important; tasks describe how to do something. The Implementation phase follows, wherein the
Concepts provide the “science” for task performance. For training materials and associated assessment materi-
example, the tasks involved in sanitizing equipment might als are rolled out, on a provisional basis, to ascer-
be conceptualized as “Reducing the levels of microorganisms tain their real-world impact.
and particulates to acceptable limits” thereby minimizing the
risk of product contamination from the equipment.
EVALUATE:
Finally, an Evaluation phase of the ADDIE model
The Overview Training element will typically be delivered either:
by an instructor in a classroom; if a full-featured Learning (a) documents aspects of the learning product
Management System (LMS) is available, it may be delivered that require further development, where-
electronically. There will be an SOP for this Overview Train- upon that documentation is fed back to be
ing event. The Skills Training elements will usually be de- analyzed again, or
livered, one-on-one, on the shop floor, by a subject matter (b) documents that the product meets the
organization’s needs, whereupon it is
expert (SME) who is also a trainer, as a Structured On-the-
finalized and rolled out.
Job Training (SOJT) event.2 There will be an SOP for each
of the SOJTs in the module.
The Overview Training element includes an assessment

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 11
Gordon Welty, Ph.D.

Figure 3
Example Overview and SOJT Elements for Training Modules

Module Overview element SOJT element


• Central Weigh Module • Material Management • Storage of Raw Materials
• Dispensing Raw Materials
• 1st Cleaning Module • Cleaning and Sanitizing I • Facility Cleaning
• Preparation of Solutions • Media and Buffer Preparation • pH Measurement
and Buffers • Preparing Media
• Preparing Buffers
• 2nd Cleaning Module • Cleaning and Sanitizing II • CIP (Clean-in-Place)
• SIP (Sterilize-in-Place)

of training effectiveness – a Knowledge Transfer Assessment training.” The second is indirectly, where the learning prod-
(KTA), for example. The training event is documented in a uct is mandated in the procedure, but the product does not
Training Record where the trainer and trainee concur that train to the procedure; this is called “non-document based
the trainee has, or has not, successfully concluded the event. training.” An example of the latter is training in current
In the case of classroom instruction, this training record is Good Manufacturing Practices (GMPs), a Food and Drug
entered into the training tracking system and the entry is
verified. In the case of a validated LMS, the training record
will be an integral part of the learning product and will be Figure 4
Document Based Training
electronically entered into the trainee’s training history.
The precise fit of each of these modules into the larger
curriculum is determined in the Design phase. Training on
Once that Overview Training event is successfully con- Management Management
Notification Notification
cluded, the trainee goes on to the SOJT events. The sev-
SOP SOP
eral SOJTs are documented in Skill Demonstration Assess-
ments (SDAs), where the trainee’s ability to independently
perform the task is documented. The results of the SDA
are then entered into the training tracking system, and the
entry is verified. After all the relevant SDAs are successfully
completed, the trainee is qualified, meaning the trainee is Train-the- TTT Management
Trainer Course
ready to perform that module’s tasks independently.
Let us consider several examples (see Figure 3). SOP

TTT Courseware
The Relationship between the Learning Product
and the Associated SOP
A second aspect of the fit between learning products and
the larger curriculum is the relationship between the learn-
ing product and the associated procedure. That, too, will be TTT
delineated in the Design phase. Qualification
There are two ways that a learning product can be related
to a procedure. The first is directly, where the product trains
to the procedure; this is sometimes called “document based

1 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Administration (FDA) requirement. The FDA requires that is the courseware for the subsequent TTT qualification ses-
this training be both “current” and “conducted on a continu- sion. These learning products have different training audi-
ing basis.”3 These requirements are typically met by training ences; the first product – the program management product
on courseware that is repeatedly offered, say on a quarterly – has the organization’s training unit as its audience; the
basis, and is also frequently revised to ensure current tech- second and third products have the prospective qualified
nological methods are included in training. The SOP that trainers as their audience.
provides guidance for the GMP regulatory training is, by Document based training and non-document based train-
contrast, relatively fixed. ing must carefully be distinguished in the Design phase; if
In the diagram seen in Figure 4, the procedure is on the not, there is the possibility that all the learning products in
left and the learning product is on the right. In the case of a non-document based training will be named after the same
procedure such as Management Notification, which identi- procedure. The author is aware of instances where a several-
fies the routine as well as exceptional situations where em- hour classroom session of mandated training had the same
ployees must notify their management, the product trains name and course number as a one-hour program manage-
directly to the procedure. ment course, causing confusion in the training tracking sys-
In the case of a procedure such as Train-The-Trainer tem and among the several training audiences.
(TTT), by contrast, there are several learning products; one It is better to delineate clearly the more complex relation-
trains to the management of the TTT program; another is ship between learning products and the associated proce-
the courseware for the TTT classroom sessions; and a third dure. The next diagram, shown in Figure 5, more clearly

Figure 5
Non-Document Based Training

Training on Quarterly
Regulatory Regulatory GMP GMP
GMP Training SOP Refresher
Training Courseware
SOP

TTT
Train the TTT Management Courseware
Trainer Course
Program
SOP

TTT
Qualification

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 1
Gordon Welty, Ph.D.

displays the non-document based training structure; it clave, and another SOP for operating a capping
is now viewed as similar to a GMP regulatory procedure, machine. There is no redundancy here; it would
where there is training to the procedure and also (a differ- be counterproductive to consolidate the two pro-
ent thing) training on courseware that is mandated by the cedures, since they deal with two distinctly differ-
procedure. ent types of machines.
Now the one-hour training on the management of 4) Department F has three SOPs and three packag-
the TTT program will have its own name in the training ing lines, one procedure for the operation of each
tracking system, and the several-hour long TTT class- line; each includes a brief section on equipment
room course will have a different name, as will the subse- maintenance. There is redundancy here, but un-
quent TTT qualification session. The two different train- like that in examples one and two. The redun-
ing audiences can clearly recognize the relevant learning dancy here is in the sections on maintenance.
products. Consolidation of the procedures would remove
The clear statement of the relation between the learning the sections on maintenance and put them in a
product(s) and the associated procedure should take place maintenance procedure of their own. [We will
during the Design phase of the ADDIE model, and will be return to this issue in the next section.]
an important contribution to the ultimate success of the Consolidation of SOPs is essentially an issue of the cor-
learning product. rect writing of procedures. Very briefly, procedure writing
has six steps, all but the last involving the collaboration of
The Learning Product’s Reduction by a procedure author (usually a technical writer) and one or
Consolidation of SOPs more SMEs.
Several learning products can be associated, directly or indi- • First, the SME(s) and the author identify the pro-
rectly, with a single procedure. This suggests that a straight- cess to be captured in this SOP.
forward means of reducing training time within a training • Second, they identify the audience for this SOP.
system might be to consolidate SOPs, thereby reducing the • Third, they develop the process map for this pro-
number of learning products. However, consolidation (or cess. The process map breaks down the process
“streamlining”) of SOPs should be logical and be done to into its elements and displays the logical inter-
eliminate redundancies, not simply to reduce the number connections between the elements.
of SOPs. We will clarify this point. • Fourth, the SME(s) and the author “chunk” the
Consider four examples that illustrate the issue: process. The chunks are developed from the pro-
1) The FDA requires gowning procedures.4 Depart- cess map, putting like elements together, and
ment A has a gowning procedure. Department pulling unlike elements apart.
B has a different gowning procedure. Consoli- • Fifth, the text of the SOP will be written from
dation of SOPs would remove the redundancies the chunks. The author writes this up and the
here; Departments A and B would work together SME(s) reviews the text in light of the intended
toward a single gowning procedure. audience.
2) Department C has a protocol on the use of Equip- • Finally, the text will be revised by the author of
ment Specific Instructions (ESIs), say involving the procedure into the standard format for SOPs.
equipment maintenance manuals. Department Consequently, if procedures are correctly written, they
D has a different protocol on the same kind of will need little streamlining in the future, and will facili-
ESIs. Again, streamlining procedures would re- tate consolidation whenever new processes come online
move the redundancies; Departments C and D and procedures need to be created. Of course, if proce-
would work together toward a single protocol on dures have been poorly written, poorly chunked, or if
the use of ESIs. there is a great deal of redundancy, then they must be
3) Department E has an SOP for operating an auto- revised along the lines sketched out above.

14 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

The Relationship between Learning Product and the the whole GMP procedure. Third, it confounds the “op-
Various Regulatory Requirements erational” level of detail in the routine situations with the
A fourth aspect of the fit between learning products and more fine-grained level of detail in the abnormal situations,
the larger curriculum is the relationship between the learn- a level of detail that can only be addressed by reference to
ing product and the various regulatory requirements. This technical manuals. Fourth, it blurs regulatory requirements
aspect is also delineated in the Design phase. There are a that differ between normal situations and exceptional situa-
number of regulatory regimes that impact on the training tions, e.g.: OSHA safety regulations.
environment. These regimes include such agencies as the For these reasons, among others, it seems more appropri-
FDA, OSHA, EPA, DOT, DEA, and others, each with its own ate to create a procedure dealing with Abends by having
set of regulations.5 separate procedures; an illustration will clarify this.
On the one hand, the number of regimes means that there If we represent the manufacturing cycle in a vertical pro-
are simply more regulations to be taken into account. On the cess map, consisting of the Setup Period, followed by the
other hand, the various regimes can present the problem of Startup Period, then the Operate Period, and finally the
regulatory overlap, where different agencies have differing Shutdown Period, abnormal events can be represented in
regulations covering the same situation.6 We will consider a horizontal process map that intersects the manufacturing
how this impacts the design of the learning product. cycle at the point of the disruption. This horizontal map
If we conceptualize the manufacturing cycle where each lays out the process of trouble-shooting, reviewing service
process has a Setup Period, a Startup Period, followed by or maintenance instructions that are specific to the equip-
an Operate Period, and then a Shutdown Period, operat- ment, implementing a set of corrective and preventive ac-
ing under GMP procedures, this cycle can be disrupted at tions, conducting follow-up monitoring, as well as address-
any time by a malfunction or abnormal end event (Abend). ing safety or other relevant regulatory concerns.
The process is interrupted by a breakdown, jam, or some At the point of an abnormal event, the GMP requirements
other malfunction. An initial question is, How do we deal of the manufacturing cycle are suspended, temporarily, by an
with these Abends consistently, as a procedure? There seem OSHA-mandated Lockout/Tagout (LOTO).7 That is where
to be two options: they can be captured in the same GMP the mechanics or engineers (in OSHA terms, the “LOTO au-
procedures that cover the process, or they can be captured thorized employees”) intervene with their Troubleshooting
in procedures of their own. SOP and associated Equipment-Specific Instructions (ESI)
In either case, when we create a procedure to deal with to troubleshoot and maintain or repair the equipment.
an Abend, we must include a trouble-shooting process, the These troubleshooting procedures and ESI protocols
original equipment manufacturer (OEM) technical manuals make up the horizontal process map. Its training module
or other instructions that are specific to the equipment, a set will include a Troubleshooting SOP that would be delivered
of corrective and preventive actions, follow-up monitoring, by an instructor in a classroom, or electronically; the ESI
and any safety or other relevant concerns. protocols would be SOJTs. These would appear on the cur-
The author has witnessed instances where the process of riculum of the mechanics (see Figure 6).
dealing with Abends is captured in the same GMP proce- After the abnormal event is under control, the LOTO is
dures that cover the routines of the manufacturing cycle. removed and the LOTO affected employees (the operators)
There are several problems with this approach. First, it over- resume the manufacturing process, under the guidance of
looks the very abnormality of the Abend. After all, this is the GMP procedures.
called an Abend because, in important ways, it is abnormal. Back in the GMP realm, and depending on the specifics of
Second, it overlooks the distinct division of labor between the abnormal event – for instance, the impact on the prod-
operators who enact the routine steps of the manufacturing uct – a Management Notification is prepared (a Notification
cycle and the mechanics who address the abnormal events. of Event, NOE) that could lead to an investigation, correc-
This has substantial training implications; the procedures tive and preventive action, and follow-up monitoring.8
tend to be much longer, and both groups must train on By keeping these processes separated (vertically and hori-

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 15
Gordon Welty, Ph.D.

Figure 6 Outlining The Proposed Learning Product


Troubleshooting Procedure Structure Outlining the proposed learning product will usually con-
sist of completing a Training Outline template (see Figure
7). We will first display an illustrative template, with twelve
fields and instructions for completing each field. This will
be followed on several of the fields.
Troubleshooting Once the template is completed, it is ready for manage-
Procedure
ment signoff, which concludes the Design phase of the AD-
DIE model.

Securing Management Approval Of The Outlined


Learning Product
Equipment Equipment The final component of the Design phase is management
Specific Specific approval of the proposed learning product. This approval is
Instructions Instructions important for at least three reasons. First, this ensures that
resources allocated to the subsequent phases, Development,
Implementation, and Evaluation, have approval at the appro-
priate organizational level. The investment of resources - par-
ticularly in the Development phase - will be substantial, and
knowledge workers, be they instructional designers, software
engineers, or whomever, are in no position to make the man-
zontally), the operators would have training on the GMP agement decision about resource allocation. Second, Qual-
procedures on their curricula, and would qualify on these ity Unit approval ensures that the proposed learning product
modules. The mechanics would have training on the trou- meets the organization’s quality criteria. Finally, there are a
bleshooting SOPs on their curricula, and would qualify number of points where training implications of the pro-
on those modules. Thus the operators would not need to posed learning product - the training audience, the course
train on the troubleshooting modules and the mechanics length, etc. – can have a profound impact on business lines,
would not need to train on the operational modules. Of and again, this impact must have managerial approval. The
course, this would require that the SOPs would be writ- signatures on the Training Outline satisfy these needs.
ten in a focused fashion.
We have seen how the proposed learning product is
fit into the larger curriculum in the Design phase of the CONCLUSION
ADDIE model. The learning product thereby aligns with The Design phase of the ADDIE model is the occasion for a
other learning products, and with organizational goals. planned approach to addressing a performance gap identi-
We reviewed four aspects to this “fit” – fied in the Analysis phase. This planned approach includes
• The structure of training modules fitting the proposed learning product into the larger curric-
• The relationship between learning product and ulum; it involves outlining the learning product in terms of
SOP a systematic template, the Training Outline; and, it includes
• The reduction of training time by consolidating the need for securing management approval of the outlined
SOPs product.
• The relationship between learning products and When the proposed learning product has moved through
various regulatory requirements the Design phase, it is ready for the Development phase
Now we will consider how the proposed learning prod- where training materials and assessment materials are de-
uct is outlined in the Design phase. veloped to address the performance gap. GXP

1 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 7
Training Outline Template

FIELDS INSTRUCTIONS
COURSE TITLE: Enter the title of the document or course.

COURSE NUMBER and VERSION: Enter the number and version of the procedure, protocol, or
document.

TRAINING AUDIENCE: Those required job positions included in the scope of the learn-
ing product. Identify the areas, departments, and positions. For
example, a training audience may consist of:
• All managers in a department
• All managers and supervisors in an area
• All employees in a department
• All employees who operate the bottler on Packaging Line 84.

CURRICULUM FIT: Identify the training module; other associated courses

PREREQUISITE COURSES/ REQUIRED SKILLS: List any prerequisite courses; any required skills

TRAINERS: All qualified trainers who have been identified as SMEs on the
course, including the Originator and Business Owner, if they are
qualified trainers.

BEHAVIORAL OBJECTIVES: Specify the observable competencies that trainees will demon-
strate upon completing the training. For example, “At the end
of this training session, the trainee will be able to demonstrate
the following skills or perform the following tasks…”

TRAINING DELIVERY METHOD: Check as appropriate:


• Classroom
• Structured On-the-Job Training
• Computer Based Training, etc.

COURSE LENGTH: Enter the approximate time required to deliver the training ses-
sion. This information is for planning purposes only.

SPECIAL INSTRUCTIONS: Instructions to facilitate the preparation and execution of the


event (e.g.: safety issues, logistical requirements, pre-work,
handouts, etc.)

MEASURES of EFFECTIVENESS: The KTA (and Answer Sheet) or SDA should be attached. The
content of the KTA or SDA is derived from the Behavioral Objec-
tives.

APPROVAL: Includes signatures from:


• Originator
• Department Management and/or Business Owner
• Quality Unit

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 17
Gordon Welty, Ph.D.

Training Audience: The personnel included in the learning product’s training audi-
ence must be negotiated. Many times a learning product will
impact not only the business unit of the business owner of the
SOP, but will impact other units as well. Personnel in those
impacted units will be listed on the Scope Statement of the
SOP, and also in the list of Task Responsibilities within the SOP
itself. Unfortunately, these two lists of personnel do not always
coincide.

Precisely defining the training audience becomes critical be-


cause those are the personnel who must be trained on the learn-
ing product associated with the new or revised SOP. After a new
or revised SOP has been approved, there is a “training window”
before the procedure goes into effect within which the impacted
personnel can be trained on the SOP. This window is typically a
week or two in length. It is critical that the training audience be
defined before that window opens – before the SOP is approved
– so that all the training will be completed before the effective
date. Thus, the risk of untrained personnel “touching” the regu-
lated product will be minimized.

When the learning product is in the Design phase, the author of


the product can provisionally prepare a Training Audience List
based on a review of the SOP Scope Statement as well as the
Task Responsibilities. When the Training Outline is circulated
for approval, the Training Audience List can be circulated as
well. Management of each impacted unit reviews the list and
recommends limiting it or expanding it, based on their direct re-
sponsibility for the task assignments of the impacted personnel.
The author of the learning product can then take those recom-
mendations into account as the product is finalized. Moreover,
management in the impacted areas are alerted for the approval
and implementation dates of the SOP, and can accordingly
schedule personnel for necessary training.

As an additional comment, it is important to recognize the dif-


ferent kinds of personnel that may be included in the training
audience for a given SOP: (1) employees (in the strict sense),
(2) independent contractors, (3) contract company (third-party)
workers, and (4) temporary agency workers.9 These four types
of employees are cross-cut by several levels or ranks: (a) sub-
ordinates, (b) supervisors (i.e., managers, directors, etc.), and
(c) executives. The finalized Training Audience List must iden-
tify impacted (and non-impacted) personnel from each of these
groups.

Behavioral Objectives: There is a strong case to be made for behavioral objectives,


sometimes called S.M.A.R.T. (Specific, Measurable, Achiev-
able, Relevant, and Time-based) objectives, in training.10 On
the one hand, behavioral objectives permit the alignment of the
intended training outcomes with organizational objectives. Any-
one who advocates cognitive (i.e., non-behavioral) objectives for
training must be prepared to explain how these objectives are to
be aligned with those of the organization. On the other hand,

18 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Behavioral Objectives continued: behavioral objectives permit the trainee to have clear expecta-
tions of the trainer’s (and the organization’s) intended training
outcomes.11 These clear expectations play a critical role in ef-
fective adult learning.
Many academics reject the role of behavioral objectives in the
university classroom; this highlights the difference between
training in industry, on the one hand, and higher education on
the other. In higher education, accredited institutions award
diplomas to students on the basis of a series of learning ex-
periences over an extended period of time. The organizational
objectives include (a) awarding the diplomas and (b) maintain-
ing the accreditation. This has very little to do with training in
industry, where the organizational objectives include (a) improv-
ing employees’ task performance on-the-job, and (b) addressing
the requirements of various regulatory regimes.12

Training Effectiveness: Assessment of training effectiveness must be distinguished


from evaluation of training programs. There is a difference in
kind – trainees are human individuals; training programs are
organizational entities. Of course trainees participate in training
programs, but the difference in kind means that the measures
are different. For instance, trainee reactions (Donald Kirkpat-
rick’s Level One,13) are perhaps useful in evaluating training
programs – favorable trainee reactions may factor in decisions
about program continuity. Trainee reactions are much less use-
ful in assessing training effectiveness, which involves assess-
ing performance improvement that will impact on-the-job – a
supervisor’s reactions are much more relevant.14

In the present context, training effectiveness is assessed by one


of two types of measures – a KTA or an SDA. The KTA in par-
ticular need not be validated in terms of the task(s) at hand. If
the KTA is validated, then performance improvement on-the-job
can be predicted from trainee performance on the KTA. If the
KTA has not been validated, the measure can still be included
in the learning product, as an interactive element of the course-
ware, and as a promissory note of future validation, if you will.
The training event will be concluded in this case by the trainee
(and trainer) concurrence that the trainee was trained on this
courseware, and thereby on the SOP.

An SDA, by contrast, directly and validly documents the train-


ee’s ability to independently perform the task(s). Furthermore,
once the relevant SDAs for a process are completed, the trainee
is qualified, able to independently perform the tasks in that
process.

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 19
Gordon Welty, Ph.D.

10 D. D. Ely “Training by Objectives, a Systems Approach to Instruction,”


REFERENCES Training & Development Journal (June 1975), Vol. 29, No. 6, pp. 23-24.
1 See D. Zielinski’s discussion of “blended learning models” in “A Closer 11 As Craig Cochran has stated, “People have trouble contributing to fuzzy,
Look,” Training (Nov 2005), Vol. 42, No. 11, p. 21; also Harvey Singh undefined objectives;” “Creating and Meeting Objectives,” Quality Digest
“Building Effective Blended Learning Programs,” Educational Technology (September 2004), Vol. 24, No. 9, p. 56.
(Nov. 2003), Vol. 43, No. 6, pp. 51-54. As B. Hall and J. LeCavalier put it 12 As Harry Moser, “The ROI for Manufacturing Training,” Modern Machine
in their “E-Learning across the Enterprise: The Benchmarking Study of Best Shop, (May, 2005), Vol. 77, No. 12, p. 98 ff. points out, higher education
Practices,” Sunnyvale, CA: BrandonHall.com (2000), p. 6: across a range of is not incompatible with training in industry - just different. See also E.J.
industries, the emerging best practices model “is a highly compatible ‘ménage Rice-Munro and R. A. Munro “Continual Improvement of Training,” Quality
à trois’ uniting online learning for information transfer and procedural skill acqui- Digest (August 2004), Vol. 24, No. 8, pp. 43-53.
sition (often this constitutes pre-work for the next tier of the model), classroom or 13 Donald Kirkpatrick, Evaluating Training Programs, San Francisco: Berrett-
other site-based learning for higher order competencies, and structured on-the-job Koehler (1994); also James Kirkpatrick, “Transferring Learning to Behavior,”
learning, integrated with knowledge management and competency evaluation.” Training & Development Journal (April 2005), pp. 19-20.
2 On SOJT, see William Rothwell and H C Kazanas, Improving On-The-Job 14 R. K. Mahapatra and V. Lai, “Evaluating End-User Training Programs,” Com-
Training: How to Establish and Operate A Comprehensive OJT Program, San munications of the ACM (Jan 2005), Vol. 48, No. 1, pp. 67-70.
Francisco: Pfeiffer (2004); also Ronald L Jacobs, Structured On-The-Job Train-
ing, San Francisco: Berrett-Koehler Publishers (2003).
3 See 21 CFR Part 211.25, “Personnel Qualifications.”
4 See 21 CFR Part 211.28, “Personnel Responsibilities.” Also Jan Eudy, “Clean SUGGESTED READING
Manufacturing,” Controlled Environments (March 2004), Vol. 7, No. 3. • Bender, A., N. Shannon, and J. Braun-Davis, “Orchestrating Compliance,”
5 See for example Lawrence Bierlein, “Who Has Responsibility for Employee Pharmaceutical Executive (Oct 2005), Vol. IV.
Hazmat Training?” Transportation and Distribution (Nov. 1998), Vol. 39, No. • Bierlein, Lawrence, “Who Has Responsibility for Employee Hazmat Training?”
11, page 123; also L. Bierlein “Conduct a Hazmat Review before the DOT Transportation and Distribution (Nov. 1998), Vol. 39, No. 11.
Comes Calling,” Logistics Today (July 2005), Vol. 46, No. 7, page 16; and • Bierlein, Lawrence, “Conduct a Hazmat Review before the DOT Comes Call-
Shadid Jamil, H. L. Floyd, and D. Pace, “Implementing Electrical Safety ing,” Logistics Today (July 2005), Vol. 46, No. 7.
Regulations and Standards,” IEEE Industry Applications Magazine (Jan 1999), • Cochran, Craig, “Creating and Meeting Objectives,” Quality Digest (September
Vol. 5, No. 1, pp. 16-21, esp. pp. 20-21. There are also state laws and 2004), Vol. 24, No. 9.
regulations that must be taken to account; see A. Bender, N. Shannon, and • Ely, D. D., “Training by Objectives, a Systems Approach To Instruction,” Train-
J. Braun-Davis, “Orchestrating Compliance,” Pharmaceutical Executive (Oct ing & Development Journal (June 1975), Vol. 29, No. 6, pp. 23-24.
2005), Vol. IV, pp. 17-18. • Eudy, Jan, “Clean Manufacturing,” Controlled Environments (March 2004), Vol.
6 For instance, the chain of custody required by DEA’s 21 CFR Part 1301.73, 7, No. 3.
“Physical Security Controls…” and the evacuation requirements of OSHA’s • Garen, John, “Use of Employees and Alternative Work Arrangements in the
29 CFR §1910.38(c), “Emergency Action Plans.” See also National Academy United States,” Labour Economics (February 2006), Vol. 13, No. 1, pp. 107 ff.
of Sciences/ Institute of Medicine Ensuring Safe Food: From Production to • Hall, B. and J. LeCavalier, “E-Learning across the Enterprise: The Benchmark-
Consumption Washington, DC: National Academy Press (1998); Society of ing Study of Best Practices,” Sunnyvale, CA: BrandonHall.com (2000).
the Plastics Industry, Food, Drug, and Cosmetic Packaging Materials Commit- • Jacobs, Ronald L., Structured On-The-Job Training, San Francisco: Berrett-Koe-
tee Newsletter (08 June 1998): “over the years, the food packaging industry hler Publishers (2003).
has been subjected to an undue burden as a result of the regulatory overlap • Jamil, Shadid, H. L. Floyd, and D. Pace, “Implementing Electrical Safety Regu-
among FDA, USDA, and the Bureau of Alcohol, Tobacco, and Firearms lations and Standards,” IEEE Industry Applications Magazine (Jan 1999), Vol. 5,
(BATF);” and “FDA Cancels Part 11 Meeting,” Part 11 Compliance Report (09 No. 1, pp. 16-21.
June 2004), Vol. 4, No. 12, p. 2, for a discussion of the regulatory overlap • Kirkpatrick, Donald, Evaluating Training Programs, San Francisco: Berrett-Koe-
between 21 CFR Part 11 and other regulations affecting life science com- hler (1994).
panies, such as HIPAA and Sarbanes-Oxley. For an overview, see Robert W. • Kirkpatrick, James, “Transferring Learning to Behavior,” T + D (April 2005),
Hahn, “Government Analysis of the Benefits and Costs of Regulation,” Journal pp. 19-20.
of Economic Perspectives (Fall 1998), Volume 12, Number 4, pp. 201-210. • Liggett, Danny P., “Training and Qualifying your Employees,” Petroleum and
7 See 29 CFR §1910.147, “Control of Hazardous Energy.” This standard man- Chemical Industry Conference, 2005. Industry Applications Society 52nd Annual
dates that each workplace, with few exceptions, must develop a program Conference (Sept. 2005), pp. 327-332.
to “disable machinery or equipment and prevent the release of potentially • Mahapatra, R. K., and V. Lai, “Evaluating End-User Training Programs,” Com-
hazardous energy while maintenance and servicing are being performed.” munications of the ACM (Jan 2005), Vol. 48, No. 1, pp. 67-70.
Hazardous energy includes electrical, mechanical, hydraulic, pneumatic, • Moser, Harry, “The ROI for Manufacturing Training,” Modern Machine Shop,
and other energy sources. The mandated LOTO program will have three (May, 2005), Vol. 77, No. 12, p. 98 ff.
components: (a) a set of written procedures for the control of hazardous • National Academy of Sciences/ Institute of Medicine, Ensuring Safe Food: From
energy, (b) an employee training program to ensure that the procedures are Production to Consumption, Washington, DC: National Academy Press (1998);.
implemented, and (c) an annual inspection to ensure that the procedures • Rice-Munro, E.J., and R. A. Munro “Continual Improvement of Training,”
continue to be followed. See also Federal Register (Nov. 06, 1989), Vol. 54, Quality Digest (August 2004), Vol. 24, No. 8, pp. 43-53.
No. 213, p. 46610; and Danny P. Liggett, “Training and Qualifying your • Rothwell, William, and H C Kazanas, Improving On-The-Job Training: How To
Employees,” Petroleum and Chemical Industry Conference, 2005. Industry Establish And Operate A Comprehensive OJT Program, San Francisco: Pfeiffer
Applications Society 52nd Annual Conference (Sept. 2005), pp. 327-332. (2004).
8 The corrective and preventive actions (CAPA) may, in the event, be the same • Singh, Harvey, “Building Effective Blended Learning Programs,” Educational
for the troubleshooting process and the manufacturing process. Technology (Nov. 2003), Vol. 43, No. 6, pp. 51-54.
9 See John Garen “Use of Employees and Alternative Work Arrangements in • Society of the Plastics Industry, Food, Drug, and Cosmetic Packaging Materials
the United States,” Labour Economics (February 2006), Vol. 13, No. 1, pp. Committee Newsletter (08 June 1998).
107 ff. • Zielinski, D., “A Closer Look,” Training (Nov 2005), Vol. 42, No. 11.

20 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Article Acronym Listing


ADDIE Analyze, Design, Develop, Implement, Evaluate LOTO Lockout/Tagout
CIP Clean-In-Place NOE Notification of Event
DEA Drug Enforcement Agency OEM Original Equipment Manufacturer
DOT Department of Transportation OSHA Occupational Safety and Health Administration
EPA Environmental Protection Agency SDA Skill Demonstration Assessment SIP Sterilize-In-Place
ESI Equipment Specific Instruction SMART Specific, Measurable, Achievable, Relevant, Time-based
FDA Food and Drug Administration SME Subject Matter Expert
GMP Good Manufacturing Practice SOJT Structured On-the-Job Training
KTA Knowledge Transfer Assessment SOP Standard Operating Procedure
LMS Learning Management System TTT Train-The-Trainer

Originally published in the July 2007 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 21
Gordon Welty, Ph.D.

Developing Assessments of Trainee Proficiency

The company can either recruit employees who have spe-


In the world of training, the development of assessments is as
essential to training administration as keeping score is to any
cific work experience or can retain employees until they
professional sporting event. The author identifies the four key acquire the desired level of on-the-job experience.
components of the processs for developing assessments and Training, by contrast, is typically documented in terms of
how to incorporate them into the training module. specific training events in which the employee has partici-
pated. Management develops an Individual Training Plan
(ITP) for each job position, listing the various training mod-
INTRODUCTION ules that incumbents must complete.
Employees in regulated industries must be trained before The ITP and the associated training can be contrasted
they “touch” the product. According to the Food and Drug to the employees’ educational attainment and work expe-
Administration (FDA), the “person” must have the “educa- rience in several ways. First, the ITP tends to foreground
tion, training, and experience, or any combination thereof, short-term objectives rather than long term. Second, it
to enable that person to perform the assigned functions.1” tends to focus on enhancing employees’ task performance
This requirement has substantial implications for both man- on-the-job, rather than their more general knowledge and
agers and regulators. It also has substantial implications for experience. Third, because of the longer time frames asso-
the assessment of trainee proficiency. ciated with educational attainment and work experience,
Education and experience are dealt with differently than these factors are taken as givens by management. Train-
training. Education is typically documented in terms of ing, however, is widely seen as a corrective action for many
grade level completed by - or diplomas awarded to - stu- problems. To the extent employees manifest performance
dents in accredited institutions on the basis of a series of gaps, those gaps will typically be remediated by training.
learning experiences over an extended period of time.2 Ex- Jack Gordon has pointed out that:
perience typically refers to work experience, and is docu-
mented by the employee’s length of time in a specific job “Managers send people to [training] courses precisely
position; many times that is an extended period of time.3 because they want to see observable behavior changes
None of these measures is entirely adequate; in both work that will produce observable business results.4”
experience and education, the qualitative richness of the
process is obscured by the measure of duration. But they So management’s concern is with behavior change,
provide guidance to corporate policy nonetheless. which necessitates assessment of the effect of training on
The pharmaceutical or other regulated company seeks the trainee’s task performance. Should the training not cor-
to ensure that employees have requisite educational attain- rect the performance gap, the management will then turn
ment by effectively implementing recruitment policies or to the more extreme measures of progressive discipline or
policies in support of continuing education. The company discharge.
can either recruit employees who meet educational levels or This concern is shared by regulators, who want to ascer-
can subsidize the employees’ further education until they tain how well the organization has its processes in control.
meet the desired level. Likewise, the company strives to If there are problems in pharmaceutical manufacturing, let
ensure that employees have the requisite work experience. us say, the regulator wants to know how the organization

22 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

responded to the problem. How was management notified? Behavioral objectives have a number of dimensions.8
Was an investigation conducted, and if so, how well? How Each must specify:
was the root cause identified? How was the corrective ac- • The training audience
tion formulated, and how was it executed? Many times the • The trainee performance
“cause” of the problem is identified as the residual category • Conditions for the performance
“human error,” hardly a root cause at all. And then “re-train- • The measure of the performance
ing” is proposed as the “corrective action,” as though the • Criteria for successful performance
initial training could unexceptionably have been ineffective.
Regarding auditors, as James Vesper has put it, Let us consider each of these dimensions in turn.

“GMP auditors and regulatory inspectors are Training Audience


becoming more savvy about training and performance. The training audience is the set of job positions whose in-
They are asking to see evidence that the training was cumbents must receive training before they “touch” the
effective.5” product, or, in the case of supervisors, before they super-
vise employees who will be touching the product.9 All job
Following the lead of management and regulators, we will positions that have responsibilities in a given Standard Op-
focus in this article on the development of training assess- erating Procedure (SOP) are part of the training audience
ments. There are four components to the process of devel- for that procedure. For example, by regulation, sanitization
oping assessments: (1) identifying the behavioral objectives SOPs apply to contractors and temporary employees as well
of the training module, based on the relevant operational as to full-time employees.10 As Roelof Kuipers has pointed
procedure and the Training Outline, (2) determining the out, it is important to define who needs what kind of train-
kind of assessment to be used, (3) preparing the assessment ing in a given organization.
materials, and (4) incorporating these assessment materials
into the training module. “With a large pool of electrical, mechanical and
We will now examine the four components of the process maintenance engineers, electricians, machine opera-
of developing training assessments. tors, contractors, and many more, not everyone needs
the same skill set.11”

BEHAVIORAL OBJECTIVES IN THE ADDIE MODEL The Boones have summarized this nicely: “Your behavior-
The ADDIE model is a generic instructional design mod- al objective should identify the specific audience you plan
el. It provides guidance at a fairly high level for instruc- to target.12”
tional designers, software engineers, etc., as they author
and revise learning products. The phases of the ADDIE Trainee Performance
model are Analyze, Design, Develop, Implement, and Trainee performance is the set of behaviors that the trainee
Evaluate. These phases are sequential - each depends will demonstrate upon completing the training. In saniti-
upon the successful completion of the preceding phase. zation processes, regulations stipulate that employees must
Behavioral objectives have been specified in the De- follow written SOPs.13 As an example of a behavioral objec-
sign phase of the ADDIE model.6 These objectives have tive that highlights observable performances, “At the end of
several important functions. First, they permit intended this training session, the trainee will be able to demonstrate
training outcomes to be aligned with organizational ob- the correct mopping technique for sanitizing controlled ar-
jectives. Second, they permit trainees to have clear ex- eas.” This demonstration - this performance - will be ob-
pectations of intended training outcomes. Third, they servable to the trainer who will certify the performance took
provide a sound basis for the design and development of place; in principle, the performance would be observable to
training materials.7 the trainee’s manager or an auditor as well.

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 23
Gordon Welty, Ph.D.

Conditions of Performance lights that the behavioral objectives are built into the assess-
The conditions of the performance is a list of conditions ment measures. As Jack Gordon (ibid) has put it,
under which the trainee is expected to perform the behav-
ior. For example, a behavioral objective might state “At “When you know what targets you’re shooting at,
the end of this training session, the trainee will be able to keeping score becomes much easier.”
demonstrate the correct mopping technique for sanitizing
controlled areas, using the double-bucket process.” The Section Summary
bolded text in the objective would be (part of) the condi- In this section we have considered the five dimensions of
tions of the performance. behavioral objectives - training audience, trainee perfor-
mance, conditions for the performance, the measure of the
Measure of Performance performance, and criteria for successful performance - and
The measure of the performance provides the categories the role they play in providing focus to the training module.
or scale that represents the performance in qualitative or Given the behavioral objectives and their associated mea-
quantitative terms.14 A measure of performance on a paper- sures and criteria, the particular kind of assessment can be
and-pencil quiz could be the number of correct answers stipulated.
(perhaps compared to the total number of questions). The
measure of sanitization of an area could be provided by en-
vironmental monitoring data. In James Popham’s terms, it is WHICH KIND OF ASSESSMENT
a major shortcoming when At a very general level, training involves two components
- a Training Event, followed by a Performance that dem-
“Behavioral objectives were not accompanied by onstrates whether the training had (or did not have) the
assessment instruments specifically linked to those desired impact on the job, in the workplace.17
objectives.” The Training Event might be a structured on-the-job
training (SoJT) event; it might be a classroom event; it
He continues that learning objectives might be an e-learning event. The Performance might be
the SoJT trainee’s independent execution of the relevant
“Will have little effect on day-to-day instruction if not tasks; it might be the trainee’s accurate logbook entry fol-
accompanied by relevant assessment devices.15” lowing a classroom session; it might be the trainee’s comple-
tion of a quiz at the end of an on-line session with 80% of
FDA regulations stipulate that the responses correct. Of course, the performance might be
unsuccessful - the trainee might fail to execute the relevant
“A protocol is required to contain …A description of tasks, might make an inaccurate logbook entry, might score
the observations and measurements to be made to less than 80% on the quiz.
fulfill the objectives of the study.16” The Training Event is a set of independent variables (the
predictors); the associated Performance is a set of depen-
dent variables (the criteria). Both components - the Training
Performance Criteria Event and the Performance - are multi-dimensional.18
Finally, criteria for performance specify the limits of success- The Training Event includes trainer(s), trainee(s) with
ful performance. For instance, many times, the performance skill-set(s) and disposition(s), training organization (super-
on a paper-and-pencil quiz is considered successful when visory factors, business case), training facilities (allocated
80% of the responses are correct. Again, the sanitization of space, allotted time, utilities), and training materials (in-
an area has been successful when environmental monitor- struments and equipment, raw and in-process materials).19
ing data for each room after sanitizing is within acceptable Training materials also include the training script - for a
limits. The inclusion of criteria is important because it high- typical SoJT event, for instance, the script would spell out

24 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

in some detail the steps in the Prepare, Tell, Show, Do, and Turning next to the core concept “task completion,” there
Follow-up cycle to be followed in this event.20 are situations where the process cannot be broken into dis-
The Performance component (continuing with the SoJT crete tasks, or is for some reason inaccessible. Consider, for
illustration) includes the trainee’s independent and satisfac- example, equipment that has a biennial preventive mainte-
tory performance of the relevant tasks in a real work setting, nance schedule. That equipment may not be available for
as judged by a supervisor or as indicated on some business- the training of mechanics for more than a year. In such a
process metric, and usually has both individual level and case, another kind of proxy must suffice. That is a Knowl-
group level (work team) elements. There is a feedback loop edge Transfer Assessment (KTA). A KTA is a paper-and-pen-
between the performance and the training event. As we ob- cil test that predicts performance on-the-job. If task comple-
served before, it is possible that the task performance by the tion or non-completion can be correlated with a test score,
trainee was unsuccessful. In that case, the adequacy of the so that high scores correlate with task completion and low
trainer’s ability or preparation, the suitability of the training scores correlate with non-completion, then the KTA is vali-
materials, the capability or motivation of the trainee, as well dated, and performance on-the-job can be predicted from
as the timing or situation of the training event (or a combi- trainee performance on the KTA. 23
nation of these) can be called to account. If the KTA has not been validated, it can still prove use-
ful as an interactive element within the courseware itself. It
The core concepts of Performance are as follows: can take the form of “study questions,” providing guidance
• a real work setting wherein to trainers as they interact with trainees in facilitating the
• a task is completed21 course. Perhaps needless to say, in this form the questions
are not part of any assessment.
This concept of performance is not always logistically We have not included Donald Kirkpatrick’s Level 1, the
feasible. Which tasks in a specific process must be com- “trainee reaction” measure,24 in our list of assessments for
pleted? How can a real work setting - with all the demands several reasons. First, there is no evidence that a trainee’s
of production and output - be accessed for training purpos- appreciation of - or affective response to - a training event
es? These are difficult questions to answer, difficult enough correlates with the trainee’s task performance.25 Thus the
that it is frequently necessary to use proxies for purposes of trainee reaction is not a surrogate for performance. Second,
training assessment. if an assessment of the utility of the training content or ma-
Whether core concepts or their proxies are utilized in as- terials is needed, a review of the module during the pilot
sessment of training, they must be documented in proce- implementation, by the training and development peers,
dures, protocols, SOPs. An SOP stipulates the standards for will likely provide a more focused and accurate assessment
the core concepts of Performance or for the proxies. than the reactions of the trainees. Third, the use of trainee
Turning first to the core concept “real work setting,” if reactions raises the possibility of documented negative judg-
that setting is unavailable for task samples, a surrogate mea- ments. For instance, suppose the trainee reaction form uses
sure must suffice. Brinkerhoff gives the example of training a question such as “What can be done to improve this train-
on cardiopulmonary resuscitation (CPR) techniques: ing module (or training event).” What shall be the correc-
tive action of the trainer to negative judgments? A regulator
“Barring a workplace heart attack, we would find no may come across these documents during an audit, and can
on-the-job application of the skill learned.22” rightly ask about the remediation that followed from them.
Better these judgments were not documented in the first
The surrogate in such a case is a Skill Demonstration As- place, if there were no intention to take corrective action.
sessment (SDA), where the trainee independently performs
the task(s) on relevant equipment outside of the real work
setting - off-hours, on placebo batches, during production SECTION SUMMARY
shutdowns, etc. In this section we reviewed several kinds of assessments that

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 25
Gordon Welty, Ph.D.

can be considered for incorporating in a particular training independently perform the task on relevant equipment out-
module. These range from assessments that approximate the side of the real work setting. As Bringslimark (ibid) has put
core concept of a Performance, through a series of increas- it, the process of how the assessment sheets are approved,
ingly distant proxies, to an assessment based on the trainee’s distributed, and evaluated also should be defined in that
knowledge of the job, as reflected in a validated KTA. [training] SOP as part of the overall training system. We will
briefly describe that process.
The originator uses the number and version of the rel-
PREPARING THE ASSESSMENT MATERIALS evant operational SOP as the course number and version for
Once the kind of assessment has been selected, the assess- the SDA form. The form includes a number of yes/no state-
ment materials can be prepared. The first step in preparing ments. These describe the identified critical or representa-
assessment materials is to complete a task analysis. Once tive tasks to be assessed on the SDA. These are the items
the task analysis has been completed, the specific tasks and assessing the trainee’s performance. (See Figure 1)
sub-tasks will be listed, groups of tasks will be aggregated The trainee performs, and the trainer or some other sub-
or “chunked,” the flow of the process will be indicated, and ject matter expert (SME) monitors the performance and
concepts providing the “science” for task performance will checks each item in turn: “yes” if the performance was
have been associated with each chunk.26 successful, “no” if not. When the performance is complete
This completed task analysis will include an extensive set (whether successful or not), the trainee and the trainer sign
of tasks. The second step is to winnow through the particu- and date the SDA. Area management may sign as well. The
lar tasks whereby the trainee’s performance will be assessed. completed form is submitted to the data entry personnel of
One way would be to take a representative or random sam- the validated training tracking system, or, in case of manual
ple of the set of tasks. Another would be to take a purposive data processing, to the staff of the document repository.
sample of those tasks that are judged critical to the whole If SDAs are not available, situational judgment testing can
process. be a proxy. In a typical situational judgment test, trainees
Once the list of tasks is a manageable length, this becomes are presented with a variety of situations they might en-
the work sample for assessment. The third step is to prepare counter in the workplace. Most situational judgment tests
a protocol for the assessment, indicating that the trainee is take a paper-and-pencil form, although they could take an
expected to perform the listed tasks under specified con- on-line form. These situations (or scenarios) are usually es-
ditions, and meeting certain criteria for success. As Vivian tablished on the basis of a job or task analysis. The trainee
Bringslimark has expressed it, selects the best way to handle each situation. The trainee’s
choice is compared to a response called “correct.” The “cor-
“Using an approved [operational] SOP, a qualified rect” response is established either empirically or by the col-
observer or trainer should observe the employee per- lective judgment of a panel of SMEs.28
forming the [operational] procedure, compare the Should situational judgment testing not be a feasible al-
performance to the [operational] SOP, and record ternative, a job knowledge test can be a surrogate. A KTA
the results on a qualification or competency assess- is a paper-and-pencil test that predicts performance on-
ment sheet. The results should be communicated to the-job. The items in the KTA can be constructed either (a)
the employee, his or her supervisor, and to the train- out of material contained in training courses, user manu-
er responsible for the original training, indicating als, technical handbooks, etc. or (b) from material provided
whether the prescribed level of competency has been by a panel of SMEs; in either case the material reflects the
attained.27” content of the job. The items that should be selected are
As we have noted above, there are circumstances where the best discriminators between employees who are judged
task sampling is not practicable, and a surrogate is neces- more proficient and less proficient performing the task.
sary for assessment. That surrogate is the SDA. A training Thus high scores correlate with proficiency and low scores
procedure stipulates how, when, and where the trainee can correlate with less proficiency; the KTA is validated, and

26 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 1
Illustrative SDA items from GMP Train-the-Trainer Program
Yes No
1. Use of appropriate GMP requirements that relate to the defined process. h h
2. Use of Adult Learning practices and principles during the training. h h
3. Demonstration of effective delivery skills for the training session. h h
4. Use of behavioral objectives h h
5. Demonstration of appropriate interaction with audience h h

performance on-the-job can be predicted from trainee per- training as well as the effects of the training. Tennant, et al,
formance on the KTA. (ibid) suggest three kinds of post-training assessments: an
“immediate test,” to be carried out when the training has
Section Summary been completed, an “intermediate test” when the trainee has
In this section we have sketched out the preparation of sev- returned to the job, and an “ultimate test” to be employed
eral forms for assessing training, ranging across the con- “after an appropriate time has elapsed in order to measure
tinuum from work sampling, through SDAs, situational the improvement of the skills, and behavioral changes.”
judgment tests, and finally to KTAs. Once the assessment Post-test scores can also be compared to pre-test scores.
forms have been prepared, they can be incorporated into Given equivalent forms, differences in scores can be taken
the training module. as some evidence of training effects.
Finally, depending on how the work process has been
INCORPORATING ASSESSMENTS INTO THE chunked and conceptualized, assessments can be incorpo-
TRAINING MODULE rated throughout the training material, in addition to any
Assessments can be incorporated into a training module in other assessments that are used as pre- or post-tests. Assess-
several ways: as a pre-training assessment, as a post-train- ments throughout the material serve to reinforce training at
ing assessment, and interspersed throughout the training a more fine-grained level, to break up training material into
material.29 lengths closer to adult attention span, etc.
Pre-training assessments (pre-tests) are administered be- Not only is the timing of assessments critical, but the se-
fore the training begins. These assessments can take the curity of the assessment process is critical as well.
form of task samples, SDAs, or KTAs. If they have been
administered before the trainees congregate at the training Test Security
site, the trainer can compile the scores, which may allow Assessment of training can place trainees under consider-
the trainer to adapt the training materials to the specific able personal and organizational pressure to succeed.30 In
levels of trainee preparedness. addition, valid assessment forms can be quite costly to de-
Post-training assessments (post-tests) are administered velop. Therefore, attention must be paid to ensuring test
after the training has been completed. Again, they can take security - that is ensuring that the training event and associ-
many forms. They can be administered before the train- ated performance comply in terms of the five dimensions
ees leave the training site, or they can be administered at a of the behavioral objectives listed above. The performance
later date, or both. If the post-tests are administered while must be identifiably that of the individual trainee, under
the trainees are still on-site, and then at one or more later the stipulated conditions, and demonstrably successful (or
times, they can serve as measures of the sustainability of the not). These compliance issues have been highlighted by the

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 27
Gordon Welty, Ph.D.

increasingly widespread use of on-line testing and assess- Returning then to the ADDIE model, the Analysis phase
ment.31 The compliance issues have a much longer history, identifies a performance gap, a discrepancy between a stan-
of course, as a review of test security problems involving dard stipulated in a procedure (SOP) and some employee
home-schooled children makes clear.32 performance.35 A performance gap can be addressed by a
There are several approaches to test security for assess- training module, that is, a set of training and assessment
ment of training. These include verifying the identity of the materials. This is followed by the Design phase, where a
trainees, and monitoring the assessments. These approaches planned approach - documented in a Training Outline - is
are quite familiar to those of us who work in regulated in- prepared to address the performance gap. Also, behavioral
dustry, and should be stipulated in an appropriate training objectives are specified in the Design phase.
SOP.33 Verifying the identities of users (trainees) means be- If management approves the design, a Development
ing Part 11 compliant.34 Monitoring the assessment of train- phase comes next, where the training module is created and
ing means having the task defined in the Action column, approved. This phase is where training materials - trainee
and the responsibility for that task listed in the Responsi- activities, instructional content, delivery methods, and in-
bilities column of the relevant operational SOP. structional technologies - and assessment materials are cre-
ated to close the performance gap. We focused our attention
Section Summary in this article on the development of training assessments.
In this section we have commented on several issues that After the training materials and assessment materials have
arise as we incorporate assessments into the training mod- been developed, the training module can be rolled out in
ule. These include the timing of assessments - whether to the Implementation phase, and finally studied in the Evalu-
conduct assessments before, during, or after the training ation phase. GXP
event - as well as how to ensure the integrity of the assess-
ment process.
ENDNOTES and REFERENCES
1. Thus for pharmaceuticals - see 21 CFR 211.25; for non-clinical lab person-
nel, 21 CFR 58.29; for biopharm personnel, 21 CFR 600.10; for medical
CONCLUSION device personnel, 21 CFR 820.25; for human tissue recovery personnel, 21
In this paper, we considered four components to the process CFR 1271.170.
2. Jaeger, David, “Reconciling the Old and New Census Bureau Education
of developing assessments. First, we reviewed five dimen- Questions: Recommendations for Researchers,” Journal of Business and Eco-
sions of behavioral objectives - training audience, trainee nomic Statistics, Vol. 15, No. 3 (1997), pp. 300-309.
3. Tesluk, Paul and R. Jacobs, “Toward an Integrated Model of Work Experi-
performance, conditions for the performance, the measure ence,” Personnel Psychology, Vol. 51, No. 2 (1998), pp. 321-355; esp. page
of the performance, and criteria for successful performance 324: “work experience has been used almost interchangeably with tenure
- and the role they play in providing focus to the training and seniority.”
4. Gordon, Jack, “Eye on ROI?” Training, Vol. 44, No. 5 (2007), p. 45.
module. Next, we examined the kinds of assessments that 5. Vesper, James, “Performance: The Goal of Training - or Why Training is Not
can be incorporated in a particular training module, rang- Always the Answer,” BioPharm (Eugene), Vol. 14, No. 2, (Feb 2001), p. 46.
6. Welty, Gordon “The ‘Design’ Phase of the ADDIE Model,” Journal of GXP
ing from assessments that approximate the core concept of Compliance, Vol. 11, No. 4 (July 2007), pp. 37-38.
a performance, through a series of increasingly distant sur- 7. Campbell, Clifton, “A Cost-effective Procedure for Adapting Instructional Ma-
terials,” Education + Training, Vol. 42, No. 2 (2000), pp. 101-102; also Clifton
rogates, to an assessment based on the trainee’s knowledge Campbell, “Instructional Materials: Their Preparation and Evaluation,” Journal
of the job, a KTA. Third, we outlined the preparation of sev- of European Industrial Training, Vol. 23, No. 2 (1999), pp. 57-107.
eral kinds of forms for assessing training, ranging across the 8. Boone, Harry and Deborah Boone “ABC’s of Behavioral Objectives,” Journal
of Extension, Vol. 43, No. 5 (Oct 2005).
continuum from work sampling, through SDAs, situational 9. See 21CFR 211.25 (b), “Personnel Qualifications.”
judgment tests, and finally to KTAs. Fourth, we commented 10. See 21 CFR 211.56 (d), “Sanitation.”
11. Kuipers, Roelof, “Trained to Improve Performance,” Process & Control Engi-
on several issues that arise as we incorporate assessments neering (PACE), (Aug 2004), p. 36.
into the training module, including the timing of assess- 12. Boone and Boone, ibid.
13. See 21 CFR 211.56 (b), “Sanitation” for Facilities; 21 CFR 211.67 (b),
ments as well as how to ensure the integrity of the assess- “Equipment Cleaning and Maintenance.”
ment process. 14. Merrill, M. David, “Hypothesized Performance on Complex Tasks as a

28 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Function of Scaled Instructional Strategies,” in Jan Elen and R.E.Clark (Eds). 26. Welty, Gordon, “Strategy and Tactics of Task Analysis,” Journal of GXP Com-
Dealing with Complexity in Learning Environments, Boston: Elsevier (2006), pliance, Vol. 11, No. 3 (April 2007), pp. 30-31.
pp. 265-282. 27. Vivian Bringslimark, “If Training Is So Easy, Why Isn’t Everyone in Compli-
15. Popham, W. James, “Whittling Wish-list Standards Down to Reality,” The ance?” Biopharm International, Vol. 17, No. 1 (Jan 2004), pp. 49.
Education Digest, Vol. 66, Issue 7 (March 2001), p. 10. 28. See Michael McDaniel and N. T. Nguyen, “Situational Judgment Tests,”
16. See 21 CFR 312.23 (a)(6)(iii)(f) This is true of the study of training no less International Journal of Selection and Assessment, Vol. 9, No. 1 (March 2001),
than clinical trials. pp. 103-113; also Michael McDaniel, F.P. Morgeson, E. B. Finnegan, M.A.
17. It may be too strong to state that a Training Event “causes” a Performance; Campion, and E.P. Braverman, “Use of Situational Judgment Tests to Predict
perhaps we should say Training Events “influence” Performances. An Job Performance,” Journal of Applied Psychology, Vol. 86, No. 4 (August
important part of evaluating training programs consists of determining how 2001). pp. 730-740.
substantial that “influence” is. But evaluating programs comes later; for now 29. Charles Tennant, M. Boonkrong, and P. Roberts have stressed the signifi-
it is crucial to keep in mind that the assessment of training involves perfor- cance of the timing of assessments in training; see their “The Design of a
mances by an individual or a group (work team). On the topic of evaluation, Training Programme Measurement Model,” Journal of European Industrial
see also Kaye Alvarez, E. Salas, and C. M. Garofano, “An Integrated Model of Training, Vol. 26, No. 5 (2002), pp. 230-240, especially page 237.
Training Evaluation and Effectiveness,” Human Resource Development Review, 30. This is a broader social issue; see Carolyn Kleiner and M. Lord, “The Cheat-
Vol. 3, No. 4 (Dec 2004), pp. 385-416. ing Game,” US News & World Report, (22 Nov 1999).
18. The Training Event/Performance model is overly general. Between the set of 31. Rowe, Neil C., “Cheating in Online Student Assessment: Beyond Plagia-
independent variables and the set of dependent variables is a set of interven- rism,” Online Journal of Distance Learning Administration, Vol. 7, No. 2 (Sum-
ing variables. These intervening variables include cognitive, conative, and mer 2004).
affective factors that impact the transfer of training to the workplace. See also 32. Mueller, Richard, and D. Brunetti, “Testing Problems of Home-Schooled
David Chan “Current Direction in Personnel Selection Research,” Current Children,” Clearing House, Vol. 62, No. 7 (Mar 1989), pp. 325-326.
Directions in Psychological Science, Vol. 14, No. 4 (2005), pp. 220-223. 33. Vesper, James, “Defining Your GMP Training Program with a Training Proce-
19. The organization and its environment - within which the Training Event, dure,” BioPharm (Eugene) Vol. 13, No. 11 (Nov. 2000), pp. 28-32.
training organization and training facilities are located - are also important 34. See 21 CFR 11.10, “Controls for Closed Systems,” and 21 CFR 11.30, “Con-
for situating employees and their tasks. These two categories can have a trols for Open Systems.”
profound impact on the conduct and effectiveness of training. 35. See also Paul B. Roberts, “Analysis: The Defining Phase of Systematic Train-
20. See also Paul Lyons, “Scripts Creation Training,” Industrial and Commercial ing,” Advances in Developing Human Resources, Vol. 8, No. 4 (Nov 2006), pp.
Training, Vol. 37, No. 6 (2005), pp. 309-313. 476-491, and Dinah Gould, D. Kelly, I. White, and J. Chidgey, “Training
21. Callinan, Militza and Ivan Robertson “Work Sample Testing,” International Needs Analysis: A Literature Review and Reappraisal,” International Journal
Journal of Selection and Assessment, Vol. 8, No. 4 (2000), pp. 248-260. See of Nursing Studies, Vol. 41, No. 5 (2004), pp. 471-486.
also C. L. Brisley, “How You Can Put Work Sampling to Work,” Factory
Management and Maintenance, Vol. 110, No. 7 (July 1952), pp. 84-89.
22. Brinkerhoff, Robert, “An Integrated Evaluation Model for HRD,” Training and Article Acronym Listing
Development Journal, Vol. 42, No. 2 (Feb 1988), p. 66.
23. There are substantial legal implications to the use of non-validated tests in
ADDIE Analyze, Design, Develop, Implement, Evaluate
employment-related situations; see U.S. Department of Labor, “Testing and
Assessment: An Employer’s Guide to Good Practices,” Washington, DC: CPR CardioPulmonary Resuscitation
Employment and Training Administration, (2000), Chap. 2, p. 5. See also FDA Food and Drug Administration
Elizabeth Shoenfelt and L. Pedigo, “A Review of Court Decisions on Cogni- ITP Individual Training Plan
tive Ability Testing, 1992 - 2004,” Review of Public Personnel Administration,
KTA Knowledge Transfer Assessment
Vol. 25, No. 3, (Sept 2005), pp. 271-287.
24. Kirkpatrick, Donald, Evaluating Training Programs, San Francisco: Berrett- SDA Skill Demonstration Assessment
Koehler (1994). SME Subject Matter Expert
25. Alliger, George, S. Tannenbaum, W. Bennett Jr, H. Traver, and A. Shotland, SoJT Structured on-the-Job-Training
“A Meta-Analysis of the Relations Among Training Criteria,” Personnel Psy-
SOP Standard Operating Procedure
chology, Vol. 50, No. 2 (1997), pp. 341-358.

Originally published in the October 2007 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 29
Gordon Welty, Ph.D.

Strategy and Tactics for Pilot Implementation in


the ADDIE Model
The ADDIE model is a generic design model. Our focus formance gap, a discrepancy between a standard stipulated
here will be the role of ADDIE for instructional design proj- in a Standard Operating Procedure (SOP) and some em-
ects in Food and Drug Administration (FDA) regulated in- ployee performance. A performance gap can be addressed
dustry. The model provides guidance at a fairly high level by a learning product, that is, a set of training and assess-
for instructional designers, software engineers, and others ment materials.
as they author and revise training modules or courseware - This is followed by the Design phase, where a carefully
the learning product. There are several application values of planned approach to addressing the performance gap is
the ADDIE model. First, the model clarifies and standardiz- outlined and approved. This planned approach has three
es the process of addressing performance gaps in an organi- components: (1) fitting the proposed learning product into
zation, allowing best practices in instructional design to be the larger curriculum, (2) outlining the proposed learning
identified and implemented. Second, this model is widely product, and (3) securing management approval of the out-
utilized in the industry, which facilitates benchmarking of lined learning product.3
instructional design between organizations. The phases of If management approves the design, the Development
the ADDIE model are Analyze, Design, Develop, Imple- phase comes next, where the learning product - the training
ment, and Evaluate.1 These phases are sequential - each materials and the assessment materials - is developed to ad-
depends upon the successful completion of the preceding dress the performance gap.4
phase. The Implementation phase follows, where the training
The ADDIE model is scalable to all size pharmaceutical, materials and associated assessment materials are rolled
biopharm, and medical device companies. The model can out, on a provisional basis, to ascertain their real-world im-
be scaled to various size organizations, and fitted to the pact. This is the first of two kinds of implementation - Pilot
particular needs of a specific organization on a case-by- Implementation or pilot testing - followed by a Formative
case basis, or by an overall decision. As an example of a Evaluation phase; the second kind of implementation is Fi-
particular case, the decision may be made in the Analysis nal Implementation, followed by a Summative Evaluation
phase to forego the needs analysis of the employees’ skills of the relative cost and benefit of the finalized program to
and dispositions - these attributes may be well-known and the organization.
documented, requiring no further analysis. Thus manage- Pilot testing of a learning product can add considerable
ment makes the decision to limit the analysis phase to a value for an organization. While the learning product - e.g.
task analysis. training module, organizational development program,
As another example, management may make the overall LMS courseware - is still in the developmental process, not
decision to forego Pilot Implementation - and the associated yet approved for final rollout, a pilot can provide significant
Formative Evaluation - and roll out every learning product data about the real-world impact of the product, going well
directly. In this instance, the Implementation phase is fol- beyond the data that can be inferred from the material that
lowed by Summative Evaluation. In both examples, it is a appears on the story-board. The data derived from the pilot
management decision to save costs by limiting the ADDIE can be used to revise and improve the learning product be-
model.2 fore it is rolled out to the department, site, or entire work-
The Analysis phase of the ADDIE model identifies a per- force. This will of course add to the overall cost of module

30 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 1 which means that the results of the Evaluation phase are
The Development Process fed back, closing the loop, facilitating further refinement
of the learning product. If the evaluation shows that the
module has shortcomings, such as lacking clarity, those
Analyze shortcomings are fed back to the author(s) to be analyzed
again. Further design and development efforts follow until
Design the module meets the organization’s needs and standards,
but that feature of the model, iterative feedback, strongly
Develop
suggests that the “Implementation” phase cannot simply be
the finalized rollout of the learning product.
Pilot implementation Final implementation Indeed, the “Implementation” phase of the ADDIE model
includes pilot implementation as well as final implemen-
Formative evaluation Summative evaluation tation.6 As Gillis and Beauchemin have put it, “The term
‘pilot’ warns everyone to expect some adjustments. […] Re-
visions and modifications make even the best training pro-
grams more effective, and evaluating the pilot reveals po-
tential program improvements.”7 The notion that the phase
development, but it is a cost that is well worth incurring. is a “pilot” of the learning product, rather than a finalized
We review the role of a pilot implementation in the pro- rollout, highlights the iterative feature of the model.
cess of developing a learning product, looking initially at Thus the ADDIE model should be conceptualized as
strategic issues and then reviewing some tactical issues. having two paths out of the Development phase. One path
First, we consider the relationship between a pilot and the leads to pilot implementation, followed by formative evalu-
ADDIE design and development model. Next, we compare ation, from which a feedback loop allows further analysis,
pilot implementation to other pilot projects in the pharma- design, and development. At some point, determined by
ceutical industry. Then, we consider a number of conditions management, the learning product is judged to be ready for
that will facilitate or inhibit the implementation of a learn- the other path. It then moves to final implementation, fol-
ing product. Turning to tactical issues, we review how an lowed by summative evaluation. (See Figure 1).
instructional designer prepares for a pilot implementation In this article we will focus on the place of pilot imple-
of a learning product, conducts a pilot implementation, and mentations in the development process.
finally, evaluates a pilot implementation.
Pilot Projects in the
Pharmaceutical Industry
Pilot Implementation and ADDIE In the pharmaceutical industry we have a well-known ex-
There appears to be some confusion about the meaning of ample of a pilot activity that illuminates the relationship be-
the term “Implement.” We hear folks saying that the “Imple- tween the (Pilot) Implementation phase and the rest of the
mentation” phase means that the training module is devel- ADDIE model. That is the transition between laboratory re-
oped, finalized, and ready to be rolled out. However, this search and development, and commercial manufacturing.
viewpoint gives rise to two questions. First, what then are When a pharmaceutical company has discovered a prom-
we to make of the “Evaluation” phase that comes after the ising product in the R&D laboratory, it goes into a develop-
Implement phase? Is this to be only a summative evalua- ment phase. The company subjects the product to clinical
tion? Does this mean that there is no place in the ADDIE trials to determine its safety and efficacy. If it is deemed
model for formative evaluation?5 That would be an unduly safe and efficacious, it is a candidate for commercial manu-
restrictive view of this generic model. facture and marketing. The question is: how does the com-
Second, the ADDIE model is an iterative feedback model, pany move from the scale of laboratory production, perhaps

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 31
Gordon Welty, Ph.D.

several ounces of product in total, to the commercial scale There are two basic possibilities for the pilot implemen-
of thousands or millions of units of product? This is where tation of a learning product, depending upon two kinds of
the pilot project fits in.8 participants in the pilot. These participants involve end-us-
The company pilots the manufacture of the product, as a ers on the one hand, and training and development peers
transition from the laboratory scale to the commercial scale. on the other. End-user testing intends to assess how rep-
The pilot has a number of outcomes, four of which are par- resentatives of the target audience interface with the learn-
ticularly important: ing product that has been developed for them. The peer
• It demonstrates the feasibility of the scale-up in inspection subjects the learning product to a review for
general. consistency with design standards and program logic;9 it
• It demonstrates the validity and reliability of the par- also can identify problems such as repetition, overtaxing
ticular process selected for the pilot. of memory, etc.
• It generates parametric product and process data for These two possibilities may disclose different kinds of
commercial manufacturing. problems with the learning product. End-user testing can
• It provides data for budgeting, planning and scheduling find problems that are overlooked by peer inspection; like-
of subsequent manufacturing. wise, peer inspection methods can find problems that are
Each of these outcomes may prove positive or negative overlooked by user testing. In many cases, the best results
for the future of the product. As examples of negative out- can often be achieved by combining the two approaches.10
comes: the scale-up may not prove feasible, the particular
process may be unreliable, there may be off-spec findings
during scale-up, and the process may not be economically Conditions Facilitating Implementation
feasible. There are specific conditions that facilitate the pilot imple-
The relationship between the (Pilot) Implementation mentation, and eventual rollout, of a learning product. The
phase and the rest of the ADDIE model is similar. When a absence of these conditions can inhibit the implementation
pharmaceutical company has discovered a promising solu- and rollout. We should ensure that these conditions are
tion to a training gap, it goes into a development phase. present for our pilot implementation.
The company assigns an instructional design team to take Donald Ely has discussed eight conditions.11 There must
the promising solution and develop it into a draft training be the following:
module. If the training module seems to be efficacious, in • A dissatisfaction with the status quo - things could be
terms of face validity and peer-review, for example, it be- better
comes a candidate for department-wide, site-wide, or even • Sufficient knowledge and skills on the part of those
corporate-wide rollout. The question is: how will the com- who would implement the learning product
pany move from the instructional designer’s desktop and • Adequate resources
storyboard to the whole workforce? This is where the pilot • Time - as Ely puts it: “Good time; Company time; Paid
implementation fits in. time.”
The company pilots the training module, as a transition • Rewards or incentives for participants
to the entire workforce. The pilot has several outcomes. It • The expectation and encouragement of participation in
shows whether or not the promising solution can be scaled decision-making about the implementation
up in general. The pilot shows the validity and reliability of • Commitment by those who are involved
the specific interpersonal and institutional process selected • Evident leadership
for the pilot (or perhaps it shows unreliability). It gener- Ely points out that this list of conditions has been vali-
ates process and outcome data that may be important for dated, and can be used to develop a checklist for the im-
the finalized learning product. And it provides data on cost plementation project. But, he cautions, they must not be
and scheduling considerations that should be taken into viewed as formulas or rules; they should be subject to local
account in the wider rollout. conditions.

32 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Moreover, there can be a profound political aspect - either ric. In the case of team-led training events, it will include
pro or con - to an implementation effort. As Carol Weiss has both individual level and group level (training team) ele-
expressed it, “This is because policies and programs are pro- ments. In the absence of such standards and criteria, as
posed, defined, debated, enacted, and funded through po- Gamse, et al., have pointed out, “if no impact were to be
litical processes, and in implementation they remain subject found, it would be impossible to know if it was because
to pressures both supportive and hostile.”12 of a failure of implementation, a failure of [the training
design], or both.”16

Obstacles to Implementation Shifting training audience


There is also a series of obstacles to implementation. Abt There are obstacles to implementation on the trainee side
Associates has identified a number of these;13 of particular as well. Employees are transferred or reassigned and are
interest to us are the following three obstacles: no longer part of the training audience. Curriculums and
• Disappearing training intervention individual training plans (ITPs) change and the training
• Variable implementation is no longer relevant to the employee’s work assignments.
• Shifting training audience This attrition and change has an obvious effect on imple-
mentation of training modules and the assessment of sus-
Disappearing training intervention tainability of training. Gamse, et al., have commented that
The training intervention is the trainer’s execution of it is not so much “that such changes create a bias between
a script.14 This script is executed (or performed) by the groups;” they go on that what is especially “problematic is
trainer(s) in specified training facilities, within allocated that the changes create noise or unknown variation against
space and allotted time, and employing requisite training which it is difficult to detect [program] impact.”17
materials. It is performed for a specific group of trainees. These three obstacles are listed in order of increasing
The training intervention disappears when the trainer fails seriousness. The disappearance of the training interven-
- for any number of reasons - to perform the script within tion can be addressed and perhaps controlled by a suitable
that space and time, for those trainees. The trainer might train-the-trainer program, a remediation that is within the
not be proficient in performing the script, resulting in a scope of the Training Department. Likewise the variability
clumsy performance; the trainer might not have physi- of implementation can be remedied by well-known quality
cal access to the script, resulting in an impromptu per- control measures, which are within the scope of the Qual-
formance; the trainees might be inattentive or asleep, etc. ity Assurance Department. The problem of shifting train-
In any case, should the training intervention disappear, ing audiences is less tractable, since it is directly caused by
there is no predictor of interest for the subsequent trainee the business needs of the organization.
performance.15 With those strategic considerations in mind, let us turn
to some tactical issues. Based on our own experience with
Variable implementation pilot projects, we will review how to prepare for, conduct,
The trainer performance of the script must be relatively and evaluate a pilot implementation.
standardized across trainers, facilities, times, and trainees.
The word “standardized” is critical here: standardization
implies standards, or criteria for the performance. The Preparing for a Pilot Implementation
training intervention becomes (unacceptably) variable Preparing for a pilot implementation has seven steps. The
when the performance deviates from those standards. On first step is to review all relevant material that has been de-
the one hand, the criteria will be set by management; on veloped so far, including any pertinent SOPs, the training
the other hand, the trainer’s preparation must include an materials, the trainee assessment materials, and the evalua-
assessment of the relevant scripted tasks, as judged by a tion materials. It is important to distinguish between trainee
supervisor or as indicated on some business-process met- assessments that measure the trainee skill acquisition,18 and

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 33
Gordon Welty, Ph.D.

the evaluation measures of the training module’s adequacy • Job aids


in terms of some set of institutional standards. Just as the • Placards
organizational or institutional purpose of a training mod- • PowerPoint® slides
ule should not be confused with its behavioral objectives, • Script for trainers
so evaluation should not be confused with assessment. The • Transparencies
two are, of course, related - trainee success will contribute to • White board and markers
program adequacy - but the two are, nonetheless, distinct.
Next, we should prepare a plan for the pilot, sometimes Safety materials include:
called an Execution Plan. We can turn to the Training Out- • Job Safety Analysis (JSA)
line for a brief overview of the module.19 This is the brief, • Material Safety Data Sheet (MSDS)
one- or two-page outline that lists the name and course • Personal Protective Equipment (PPE)
number of the module, identifies the training audience, in-
dicates how the module fits in the larger curriculum, lists Process materials include:
the behavioral objectives, indicates the delivery modality, • Equipment
the anticipated duration of the training session, identifies • Controls (make sure the switches work, etc.)
the assessment materials, etc. In addition to the information • Instruments
derived from the Training Outline, the plan should sketch • Utilities (make sure the water runs when you open the
the various roles and responsibilities for the preparation, tap, etc.)
execution, and evaluation of the pilot implementation.
This plan will indicate the extent to which end-user test- The sixth preparatory step is to review the set of end-us-
ing, and peer inspection, will be involved. Once the plan ers, the target audience for the learning product. Who is in
is ready, it is important to get management approval of the the scope of this training? Ensure coverage of all significant
plan for the pilot. groups within the scope.20 This means including differing
Our third step is to prepare a checklist for the pilot imple- technical skill levels; different cultural, language, and ethnic
mentation. As with any well-planned training event, it is groups; different sites and facilities; differing tenures - some
hard to imagine too much detail in the checklist. Better the new hires, some old timers, etc. It is important to estimate
checklist should be overly detailed than to realize at the last the percentage of invitees that will actually be attendees;
minute, with the participants coming in the door, that we that estimate will ensure you have enough participants at-
have neglected some critical factor. Once we have developed tending the pilot to provide reliable and credible data on
a comprehensive checklist, this can provide a template for outcomes and process. The estimate of invitees who will
subsequent pilots. actually attend will depend upon your experience, or the
Next, we must schedule room(s) in the case of classroom experience of your training and development peers. Then
training, or work station(s) and equipment in the case of you can assemble the list of invitees, and again be sure to
Structured On-the-Job Training (SOJT) sessions. When get management approval. Each attendee’s manager will
scheduling, try to get the same room, work station, or need to approve participation.
equipment that would be used in any other training event. The final preparatory step is to send invitations to the
Fifth, we must prepare all required materials for the pi- pilot session. Invitations should be sent to each participant
lot session, including training materials, safety materials, (trainee), as well as to your training and development peers.
and process materials. These materials can be listed on the Inviting your peers is a courteous collegial gesture, and
comprehensive checklist, and can be ignored (N/A’d) if not these attendees can provide peer evaluations of the session
needed. that the participants may not be prepared to do. The invita-
Training materials include: tion should include a brief overview of the module indicat-
• Flip charts and markers ing that this is a pilot; be sure to mention that training credit
• Handouts for trainees in the employee training history will depend on extent of

34 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

revisions that are required. If minor revisions are called for, by section, task by task. For each section and task, discuss
training credit can be given for the session. If major revi- the purpose of the task; the importance of the task; when
sions are needed, attendance can be noted but credit cannot and where to perform the activity; and the expected results
be given, since the revised module that will ultimately be of correct performance and the potential results of incor-
rolled out will not be the same as the pilot module. rect performance. Highlight critical safety points for each
task (as needed); also highlight key Good Manufacturing
Practice (GMP) points for each task (as needed). Then invite
Conducting a Pilot Implementation questions or concerns. It perhaps goes without saying that
Conducting a pilot implementation has eight steps. When training and development peers should hold their questions
the day and time of the pilot session arrive, use your check- and concerns for a post-session debriefing. The author has
list to make sure that everything is in place and ready to seen training sessions where the peers raise questions while
go. Welcome the end-user trainees and your training and the trainees are present, and it can be quite disruptive. On
development peers. Indicate again that this is a pilot imple- the one hand, the trainees can be confused by the differ-
mentation; repeat that credit to the participants’ ITPs will ent “spins” on the training material. On the other hand, the
depend upon the extent of revisions that are needed. Even exchange between training and development peers can sug-
if credit cannot be given because major revisions are called gest that there is dissention within the training unit.
for, the trainees’ participation in the development of this The fifth step is to demonstrate each task (as needed).
module will be noted and appreciated. Discuss the logistics This will be particularly important in SOJT sessions. Also
of this facility, where the water fountains, coffee machines, in SOJT sessions, allow the trainee to practice; record the
and restrooms are located, etc. Address relevant Emergency trainee’s progress through the sequence of tasks. It is im-
Response Plans, fire escape routes, etc.21 portant to track trainee progress on an explicitly non-GMP
The second step in conducting the pilot is to distribute the progress form. Since trainee progress will only be on part of
training materials, and indicate criteria for success - 80%, the module - representing part of a SOP - that progress can-
100% or whatever. The preliminary knowledge check, if not be recorded on a controlled (GMP) form. The non-GMP
applicable, should then be administered. progress form can be disposed of after the module is com-
The third step is to explain the content of the pilot mod- pleted, after the session is duly recorded on a controlled
ule. This is an opportunity to present the “science” of the training tracking form.
process; it is more than a sequence of tasks. Present the While the trainees are progressing through the sequence
behavioral objectives for the module. It is worth repeating of tasks, provide assistance as needed - while the trainee
that adults learn best when they have crystal clear expecta- prepares for independent performance (for SOJT), and
tions about their projects; hence we always use behavioral while the trainee prepares for an assessment (for a class-
objectives. Invite questions or concerns from the partici- room module).
pants (trainees), and specify the feedback process. Stress In the case of SOJT, after the session is completed, al-
that you welcome feedback; that the main purpose of a pilot low independent performance by the trainee. Observe the
implementation is to elicit feedback for program improve- trainee perform each task safely, correctly, and without any
ment. Specify how the participants should make their feed- coaching from the trainer.
back - whether they should interact immediately with the When the independent performance is completed, or
trainer(s) when they have an issue, or they should note the when the classroom session is completed, assess each train-
issue for later discussion. ee’s performance. Utilize the appropriate GMP assessment
In either case, every issue should be recorded for later form, and assess independent performance (for SOJT); as-
attention. Also, mention that they will be called upon to sess knowledge transfer (for a classroom module).
evaluate the pilot session before they leave - we will return The final step in conducting the pilot session is to record
to this point in the next section. the completion of the module. Use the training tracking
The fourth step is to move through the module, section form, which as we have noted is a GMP form.

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 35
Gordon Welty, Ph.D.

Once the pilot session is completed, it is time to evaluate ing significant data about the real-world impact of the prod-
the adequacy of the training module, propose revisions as uct. This data can go well beyond what can be inferred from
needed, and prepare a report to management. the material that appears on the story-board. The data from
the pilot implementation can be used to revise and improve
the learning product - as part of a formative evaluation - be-
Evaluating a Pilot Implementation fore it is finalized and rolled out. GXP
Evaluating a pilot implementation has six steps. The first
step is to invite the end-user trainees to evaluate the pi-
lot module. Explain the evaluation process, and how the Endnotes
1. There are some slight variations in the terms applied to the phases of the
evaluations will be used in feedback for program improve-
ADDIE model. For instance, with reference to medical devices, the FDA has
ment. Use explicitly non-GMP evaluation forms. Since at used the term “establish” to encompass three of the phases, “define,” “docu-
this point we are evaluating a work in progress, the training ment,” and “implement;” see 21 CFR 820.3(k). Similarly, critics of the AD-
DIE model occasionally refer to the implementation phase as the “Instruct”
module that is under development, not yet approved - there phase of the model. See Jack Gordon and Ron Zemke, “The Attack on ISD:
should be no record of that progress on a controlled (GMP) Have we got Instructional Design all wrong?” Training, Vol. 37, No. 4 (April
2000), p. 42.
form. Sometimes “sticky notes” - clearly not controlled 2. See Gordon Welty “Strategy and Tactics of Task Analysis,” Journal of GXP
documents, can be used to record the trainees’ evaluations. Compliance, Vol. 11, No. 3 (April 2007), pp. 26-34.
The non-GMP evaluation forms can be disposed of after the 3. See Gordon Welty, “The ‘Design’ Phase of the ADDIE Model,” Journal of
GXP Compliance, Vol. 11, No. 4, (July 2007), p. 40-52.
module is completed and approved. Collect the evaluations 4. See Gordon Welty, “Developing Assessments of Trainee Proficiency,” Journal
from the trainees as they depart the room. of GxP Compliance, Vol. 12, No. 1, (Oct 2007), pp. 64-73.
5. See Michael Scriven, “Types of Evaluation and Types of Evaluator,” Evalua-
The second step is to collect evaluations of the session tion Practice, Vol. 17, Issue 2, (Spring/Summer 1996), p. 151 ff.
and the module from your training and development peers. 6. On the complexity of the Implementation phase, see James Mosley and
Nancy Hastings, “Implementation: the Forgotten Link on the Intervention
This can be done by a face-to-face debriefing or, again, by Chain,” Performance Improvement, Vol. 37, No. 4 (April 2000), pp. 8-14.
the use of an explicitly non-GMP evaluation form. Irene Visscher-Voerman and Kent Gustafson distinguish “implementation
The third step is to review all the evaluations of the mod- anticipation” (what is usually called Pilot Implementation) and “implemen-
tation” proper; see their “Paradigms in the Theory and Practice of Education
ule and the pilot session. and Training Design,” Educational Technology Research & Development,
Then, prepare an evaluation report summarizing the eval- Vol. 52, No. 2 (2004), pp. 74-75.
7. See Marge Gillis and Katherine Beauchemin, “The Ideal Rep,” Pharmaceuti-
uations; consider making revisions to the learning product. cal Executive, Vol. 20, No. 12, (Dec 2000), p. 60; see also David Gallup, K.
Propose needed revisions to the module, and get manage- Beauchemin and M. Gillis, “Competency-based Training Program Design,”
PDA Journal of Pharmaceutical Science and Technology, Vol. 53, No. 5,
ment approval of these revisions. As Gillis and Beauchemin (Sept 1999), p. 244 on “Implementation.” See J. Lynne Brown and N.
have put it, “Revisions may include incorporating new ma- Kiernan, “Assessing the Subsequent Effect of a Formative Evaluation on a
terial to help the program meet its objectives or changing Program,” Evaluation and Program Planning, Vol. 24, No. 2, (May 2001) pp.
129-143, and J. Lynne Brown and N. Kiernan, “Using Formative Evalua-
the objectives themselves based on trainees’ or managers’ tion to Improve an Osteoporosis Prevention Program,” Journal of Family &
input. Changes must support specific, measurable objec- Consumer Sciences, Vol. 90, Issue 2, (Summer 1998), pp. 23-29.
8. Douglas R. Watson, S. M. Clark and G. S. Joglekar, “Use of Simulation in
tives.”22 In light of the seriousness of the needed revisions, the Design of a Pharmaceutical Process,” Prepared for presentation at the
determine the appropriate training credit for participants. Annual AIChE Meeting, (01 Nov 2001), also P. Lendren, M. Owen and
S. Godbert, “Design of Experiments in Development Chemistry,” Organic
The fifth step is to dispose of all non-GMP evaluation Process Research and Development, Vol. 5, (2001), p. 326.
forms. 9. On program logic and the Logical Framework Approach (Logframe) to
The last step is to submit the training tracking form for training program design, see Des Gasper “Problems in the Logical Frame-
work Approach and Challenges for `Project Cycle Management’,” The ACP-
appropriate credit to each participant’s ITP. EU Courier, No. 173, (Jan 1999), pp. 75-77; also H. Eggers “Project Cycle
Management Revisited,” The ACP-EU Courier, No. 169, (May 1998), pp.
69-72.
10. See Jakob Nielsen, “Usability Inspection Methods,” CHI’94 Companion
Conclusion (Boston, MA, 24 April 1994), Association for Computing Machinery, p.
413; also J. Nielsen, Designing Web Usability, Indianapolis, IN: New Riders
The well-executed pilot implementation of a learning prod- (1999).
uct can add considerable value for an organization, provid- 11. Donald P. Ely, “Conditions that Facilitate the Implementation of Educational

36 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Technology Innovations,” Journal of Research on Computing in Education, 21. John Erich “Get ‘Em Out Alive: Evacuation Considerations for EMS,” EMS
Vol. 23, No. 2, (Winter 1990), pp. 298-305; see also D. P. Ely, “New Per- Magazine, Vol. 36, Issue 2 (01 Feb 2007), p. 41
spectives on the Implementation of Educational Technology Innovations,” 22. Gillis and Beauchemin, “The Ideal Rep” (op. cit.), p. 60; also David Gal-
ERIC Number: ED427775 (1999); and Barry Porter, “Time and Implement- lup, K. Beauchemin and M. Gillis, “Competency-based Training Program
ing Change,” British Journal of Educational Technology, Vol. 36, No. 6, (Nov Design,” (op. cit.), p. 244 on “Refining.”
2005), pp. 1063-1065.
12. Carol Weiss, “Where Politics and Evaluation Research Meet,” Evaluation
Practice, Vol. 14, (1993), pp. 94.
13. Beth Gamse, M.A. Millsap and B. Goodson “When Implementation Threat-
ens Impact: Challenging Lessons from Evaluating Educational Programs,” Article Acronym Listing
Peabody Journal of Education, Vol. 77, No. 4 (2002), pp. 152-157.
14. Robert Godfrey, “Session Plan: Road Map to Successful Training,” Training & ADDIE Analyze, Design, Develop, Implement, Evaluate
Development Journal, Vol. 27, Issue 9 (Sept 1973), pp. 6-10.
15. Gamse et al. (op cit) give the example of an educational intervention which FDA Food and Drug Administration
intended to compare two methods of training school personnel: some were GMP Good Manufacturing Practice
trained by university staff, while others were trained by utilizing a videotape ITP Individual Training Plan
series developed by the same university. The evaluators soon discovered that JSA Job Safety Analysis
few schools had ordered the videotapes, and those that did, weren’t using
them appropriately. Hence that arm of the intervention had “disappeared.” LMS Learning Management Software
16. Gamse et al. (op cit), p. 156. MSDS Material Safety Data Sheet
17. Gamse et al. (op cit), p. 157. PPE Personal Protective Equipment
18. See Welty, “Developing Assessments of Trainee Proficiency” (op cit.), pp. R&D Research and Development
64-73.
19. See Welty, “The ‘Design’ Phase of the ADDIE Model” (op cit.), p. 48. SOJT Structured On-The-Job-Training
20. As Annie Koh has put it, “you need to segment the people in the company SOP Standard Operating Procedure
by functions and give them what they need to get the job done;” see “Rolling
Out Major IT Projects,” Business Times Singapore (02 Aug 2007)

Originally published in the January 2008 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 37
Gordon Welty, Ph.D.

Strategy and Tactics of Training Recordkeeping

Recordkeeping is necessary for any training system that is terested in the trainees’ currency in their ITPs, for purposes
subject to audit. This necessity may come to be recognized of work assignments. At the beginning of each shift, the
as early as the point in program development when training supervisor wants to know if the employees on this shift are
assessments are created1. It certainly comes to be recognized trained to the current versions of each and every standard
when the program is being implemented (or even piloted) operating procedure (SOP) that is listed in the ITP, that will
and actual training records are generated. This documenta- be executed during that shift. The supervisor reviews the
tion could include training records (or attendance sheets), employees’ training histories (i.e., the summary of the train-
training assessments, and curricula or individual training ing records). Then the supervisor makes work assignments
plans (ITPs). These documents could be in electronic form accordingly. Thus, the training records are used proactively
or hard copy. to control and direct the organization.
This article examines training recordkeeping in the phar- Auditors include internal and external auditors (e.g., reg-
maceutical, biopharmaceutical, medical device, blood prod- ulatory investigators, etc.) who are interested in whether
uct, and other regulated industries. First to be considered, the signer of the particular operational document (i.e., a
in this paper, are some strategic aspects of recordkeeping batch record) was trained to the appropriate SOP before
before turning to tactical issues. signing. The auditor reviews the signer’s training history in
In the mid 1990s, the University of Pittsburgh conducted light of a set of documents being inspected. In these cases,
a major study of functional requirements for recordkeep- the training records provide evidence of the organization’s
ing, called the Electronic Records Project2. Reporting on past fulfillment of its regulatory obligations.
this project, and specifically describing records, Wendy
Duff stated: “[Records] are created in the first instance to
control or direct an organization and to help orient staff to RECORDKEEPING REQUIREMENTS
a common goal or purpose.” That is, records serve the pur- As the Electronic Records Project at the University of Pitts-
pose of controlling and directing the organization. She con- burgh has indicated, recordkeeping requirements can be
tinues, “They have residual value because they document considered at several levels—that of the organization, that
the outcomes of the directing and controlling activities and of the recordkeeping system, and that of the record itself
because they provide evidence of an organization’s rights as (see Figure 1).
well as its obligations to its staff and society. For records to The organization level (I), the highest level of require-
fulfill these roles, they must be readable, understandable, ments, must be compliant with all relevant legislation, regu-
and trustworthy3.” lations, and best practices concerning training records4.
There are two main audiences for recordkeeping: opera- The training recordkeeping system (II)—whether elec-
tional staff and various quality auditors. The operational tronic, paper based, or hybrid—must be implemented, re-
perspective is typically proactive, while the auditor’s per- sponsible, consistent, and appropriately backed up accord-
spective is typically retroactive. There are also other audi- ing to the following definitions:
ences including the training unit itself. • Implemented means that training events can be duly
Operations staff includes employees (the trainees) and recorded in the system
their supervisors. Both employees and supervisors are in- • A responsible system’s controlled documents (i.e., SOPs

38 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 1 • Authorized training records have been created by an


Levels of Recordkeeping Requirements authorized person, for example a qualified trainer
• Comprehensive means that a training record has been
created for every training event. For an instance of not
Organizational Level (I) meeting this functional requirement, see FDA’s Warning
Letter to Rhytec, Inc. (24 April 2007): “Documentation
of training is not consistently maintained.”9
System Level (II)
• Identifiable means only one training record has been
created for a given training event, and it is linked to that
Record Level (III) particular training event.
• Complete training records include all information about
the training event, for instance, which employee was
trained, on which SOP, by whom (the trainer), and at
for training recordkeeping) are written and followed, what time and date. As an instance of the failure to meet
plus the procedure clearly identifies the responsible this functional requirement, consider FDA’s Warning
party for each task5. As an example of the failure to Letter to Omnicare, Inc. dated 11 January 2007: “all of
meet this functional requirement, and its GXP impli- the employee records lacked the ‘Supervisor Signature’
cations, consider the US Food and Drug Administra- to show that the training was given.”10
tion’s Warning Letter to Arrow International dated 10 Maintained records must be inviolate, auditable, and ap-
October 2007: “According to procedure #CHR-001. propriately retained according to the following definitions:
‘Training Administration, Documentation, and Record- • Inviolate is defined as any alteration or modification of
keeping Procedure,’ your firm has 30 days to complete the record is traceable, and further, that repudiation of
the training. One training requirement was over six (6) the record is not possible. As an illustration of not meet-
months late.”6 ing this functional requirement, consider FDA’s Warn-
• Consistent systems have been validated, so identical ing Letter to Concord Laboratories dated 11 July 2006:
processes generate identical outcomes. Vendors some- “Appropriate controls are not exercised over computers
times suggest that their software has been validated or or related systems to assure that changes in analytical
audited. However, FDA has specifically stated that the methods or other control records are instituted only by
organization using the software must validate it for that authorized personnel.”11
situation7. • Auditable means that every use of the record leaves an
• Appropriately backed-up systems protect documents audit trail. As an example of the failure to meet this re-
from loss or corruption by being subject to a regularly quirement, see FDA’s Warning Letter to Concord Labo-
scheduled backup. As an example of the failure to meet ratories dated 11 July 2006: “…review of audit trails is
this functional requirement, see FDA’s Warning Letter not required.”12
to the Cordis Corporation in Warren, NJ dated 01 April • Appropriately retained training records must be subject
2004: “…validation did not include testing and verifi- to a retention schedule and then disposed according to
cation of backup and restoration of the electronic data procedure13.
files.”8 Usable training records must be exportable, retrievable,
The documentation of the training itself can be viewed and accessible to authorized parties according to the follow-
as a captured record, a maintained record, and a usable re- ing definitions:
cord. • Exportable records must be portable from one system
The required characteristics of a captured training record to another without loss of information
are authorized, comprehensive, identifiable, and complete • Retrievable training records are in a form that can be
according to the following definitions: searched and retrieved within a reasonable period of

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 39
Gordon Welty, Ph.D.

time and expenditure of resources complaint be conducted; or the record shall include the rea-
• Documents accessible to authorized parties must be son and the name of the person responsible for a decision
available to those who are authorized to access them not to conduct an investigation. That is a third predicate
and unavailable to those who are not authorized14. rule; since it also deals with maintaining records of investi-
After identifying two main audiences for the documen- gations, it also implicates Part 11.15
tation of training—operational staff and auditors —the Equipment cleaning and maintenance under GLP, GMP,
training recordkeeping must possess characteristics of good and medical device regulations have broader scope (see
documentation management. If at each level—organiza- Table I). The cleaning and maintenance requirement (first
tion, training recordkeeping system, and documentation predicate rule) also stipulates that the cleaning and mainte-
of training—characteristics are present that are appropriate nance must be recorded (second predicate rule).16
for that level and proceduralized, that level will be “audit The form of these typical regulations involves two aspects:
proof,” which is to say it can survive an internal or external a requirement (one predicate rule) that a task or activity be
GXP audit, and will moreover have business value to opera- proceduralized and the SOP be followed, and a requirement
tional staff. (a second predicate rule) that an associated record be kept of
the activity or task. The second predicate rule, dealing with
recordkeeping, implicates Part 11 if the organization had
PART 11 COMPLIANCE decided to manage the record electronically. Insofar as Part
When document management is discussed with reference 11 is implicated, procedures and controls must ensure the
to training and assessment, the topic of Part 11 compli- authenticity and integrity of electronic records. Moreover,
ance frequently comes up (Part 11 refers to “Electronic Re- procedures and controls must hold individuals accountable
cords; Electronic Signatures,” which is Part 11 of 21 CFR). and responsible for actions initiated under their signatures.
In keeping with the emergence of electronic technologies,
FDA issued regulations in 1997 for e-records and e-signa-
tures that sought to permit wide use of electronic technol- TRAINING RECORDS
ogy, compatible with the protection of public health. Soon By contrast, the documentation of training, including train-
after they became effective, FDA announced a reconsidera- ing records and training assessments, is not covered by such
tion of these regulations. In 2003, FDA withdrew the guid- predicate rules. FDA regulations for areas such as phar-
ances that had accompanied the regulations. While the maceutical and biopharmaceutical operations, clinical tri-
reconsideration of the regulations was under way, FDA in- als, medical device operations, or human tissue processors
dicated they would narrowly interpret the scope of Part 11 require that personnel be trained.17 These are examples of
and promised to exercise enforcement discretion. During the first predicate rule noted in Table I.
this period, records and recordkeeping need still comply The only requirement for documentation of training is
with the underlying regulations. found in FDA’s GLPs, where it is stipulated that “Each test-
A typical example of FDA regulations and associated re- ing facility shall maintain a current summary of training
cordkeeping is quality complaints about drug products. 21 and experience and job description for each individual en-
CFR 211.204 requires written procedures for the handling gaged in or supervising the conduct of a nonclinical labora-
of all product quality complaints. This requirement (“pred- tory study”.18 That implicates a “current summary” of the
icate rule”) further stipulates that “a written record of each individual’s training records, which might take the form of
complaint shall be maintained in a file designated for drug the individual’s training history, not the training records, or
product complaints.” That is a second predicate rule; since training assessments themselves.
it deals with recordkeeping, it implicates Part 11, if the or- Regarding clinical trials, FDA stipulates that “A protocol
ganization has chosen to manage that record electronically. is required to contain the following […] The name and ad-
Moreover, the initial regulation also stipulates that a record dress and a statement of the qualifications (curriculum vitae
shall be maintained, should an investigation of the product or other statement of qualifications) of each investigator,

40 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

TABLE I: Equipment Cleaning and Maintenance Under GLP, GMP, and Medical Device Regulations.
Regulation First Predicate Rule Second Predicate Rule

(a) Equipment shall be adequately inspected, cleaned, (c) Written records shall be maintained of all inspection, mainte-
21 CFR 58.63
and maintained. nance, testing, calibrating and/or standardizing operations.

(a) Equipment and utensils shall be cleaned, main-


tained, and sanitized at appropriate intervals to prevent
b) Written procedures shall be established and followed for clean-
malfunctions or contamination that would alter the
21 CFR 211.67 ing and maintenance of equipment, including utensils, used in the
safety, integrity, strength, purity, and quality (SISPQ) of
manufacture, processing, packing, or holding of a drug product.
the drug product beyond the official or other established
requirements.

Equipment and utensils shall be cleaned, maintained,


and sanitized at appropriate intervals to prevent A written record of major equipment cleaning, maintenance (except
malfunctions or contamination that would alter the routine maintenance such as lubrication and adjustments), and
21 CFR 211.182
safety, integrity, strength, purity, and quality of the use shall be included in individual equipment logs that show the
drug product beyond the official or other established date, time, product, and lot number of each batch processed.
requirements.

(g) Each manufacturer shall ensure that all equipment


used in the manufacturing process meets specified
(1) Maintenance activities, including the date and individual(s)
21 CFR 820.70 requirements and is appropriately designed, constructed,
performing the maintenance activities, shall be documented.
placed, and installed to facilitate maintenance, adjust-
ment, cleaning, and use.

and the name of each sub-investigator (e.g., research fellow, the education, training, and experience to perform their as-
resident) working under the supervision of the investiga- signed tasks,” which is another example of the first predi-
tor” on a given clinical study.19 Notice that this predicate cate rule noted in Table I.22
rule about “curriculum vitae or other statement of qualifica- In light of this, the documentation of training does not
tions” of the clinical trials investigator was not extended to fall within the scope of Part 11. What are the implications
the subordinates, the research fellows, residents, etc. of this limited scope? We assume throughout that this doc-
Finally, the GMPs require “records shall be maintained umentation is controlled, as well as duly signed by the party
stating the name, address, and qualifications of any consul- responsible for the action described. The documentation
tants and the type of service they provide”.20 It is instructive of training can be considered as instances of what FDA has
that the rulemaking that applied this predicate rule to the called a hybrid situation. In such a situation, paper record
qualifications of consultants did not apply it to the major- and signature components can co-exist with electronic re-
ity of pharmaceutical operations employees and supervisors cord and signature components, “so long as […] the con-
covered in 21 CFR 211.25. tent and meaning of those records are preserved.”23
FDA regulations are silent about training records for oth- Consider the following scenario: A GXP training event—
er areas such as pharmaceutical and biopharmaceutical op- either technical training or regulatory training—has just oc-
erations, medical device operations, blood products proces- curred. All the trainees have been assessed as “successful” in
sors, or human tissue processors.21 Indeed, 21 CFR Part 11 the training. There is a training record—a controlled docu-
does not include such a requirement for itself. It requires ment—and its use is proceduralized, including entry into
only a “determination that persons who develop, maintain, a validated Learning Management System (LMS). Trainees
or use electronic record/electronic signature systems have then sign and date the paper-training record, and the trainer

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 41
Gordon Welty, Ph.D.

TABLE II: Necessary Fields in Training Records.

This name must correspond to the employee’s name as entered in the annually renewed
1 Employee (trainee) Name
signature card.

Temporary employees, etc. who do not have an authenticated network login must be provided with a
2 Employee ID Number
unique ID number.

3 Course/SOP Name What is the organization’s naming convention?

4 Course/SOP Number What is the organization’s numbering convention?

5 Type of Training This includes one of a number of delivery modalities—classroom, e-learning, coaching, etc

6 Date of Training Event What is the organization’s date/time convention?

This name must correspond to the trainer’s employee name as entered in the annually renewed
7 Trainer’s Name
signature card.

Consultants, etc. who do not have an authenticated network login must be provided with a unique
8 Trainer’s Employee ID Number
ID number.

9 Data Clerk Name [See comment to #1 above.]

10 Data Clerk’s Employee ID Number [See comment to #2 above.]

11 Date Entered into LMS [See comment to #6 above.]

12 Data Steward Name [See comment to #1 above.]

13 Data Steward’s Employee ID Number [See comment to #2 above.]

14 Date Verified [See comment to #6 above.]

countersigns and dates the record. At this point, the event the scope of Part 11. Instead of the scenario outlined in
is fully documented; the trainees are fully trained to perform the previous paragraph, the documentation of the train-
the GXP tasks. They can “touch” the product or supervise ing event is treated like a regulatory submission, say, where
those touching the product. Then, according to procedure, each of the parties involved must provide an electronic sig-
duly authorized data entry clerks enter the data from the nature to the electronic record. So the “draft” training re-
training record into the LMS within 72 hours, and sign off cord is routed to each trainee for their review and electronic
on the entries. A duly authorized data steward verifies the signature, then is routed back to the trainer for review and
data entries and signs off. At this point, by procedure, the electronic signature, and finally routed to QA for “release.”
electronic record becomes the controlled document, and The number of “transactions” increases dramatically. When
the paper copy can be archived or disposed. finally released, the training record ceases to be a draft, and
Sounds straightforward; however, there have been situa- becomes “effective.” Before the training record became
tions where it is assumed that all training records fall within “effective,” employees who had just been trained were in-

42 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 2
Continuum of Training Impact.

Measure of Inputs Measure of Outputs

Knowledge Skill
Staffing of Training seats Return on
transfer demonstration
training unit occupied investment
assessments assessments

structed not to “touch” the drug product. Training records, for instance, have a number of neces-
In a study of a random sample of training records (n = sary fields. These fields are listed in Table II, insofar as the
11) processed this way, involving employees—not consul- documentation corresponds to the first scenario.
tants, not non-clinical lab staff, not clinical trials investiga- The training procedure must not only list each of these
tors—the average time between the conclusion of the train- fields, plus any optional “nice-to-have” fields, but must
ing event and the “release date” was 10 days. indicate the roles and responsible party for each field. If
If training records had been recognized as outside the the training SOP describes a fully electronic scenario where
scope of Part 11 and the first scenario had been followed, trainees record their participation in the training event on-
the time between conclusion of the training event and the line, and trainers also record their facilitation of the event
full documentation of the training would have been about online, the SOP must additionally describe the fall-back
10 minutes—the time it takes the trainer to check all the process in the event there are connectivity problems, train-
trainees’ signatures and dates and countersign the training ing in remote locations, etc. Thus the roles and responsi-
record. bilities of the data clerk and data steward are still critical.
FDA regulations require that personnel touching the prod- If hard copies are archived following data entry into the
uct be trained. The regulations, with few exceptions, do not validated LMS, they should be placed in the document re-
address training records, including training assessments. It pository with, at a minimum, the following indexing in-
is important to review carefully the cases where the regula- formation: record type, file names, date ranges, and posi-
tions do apply and ensure compliance. It is equally impor- tions (titles of personnel) who are authorized to access the
tant to ensure that the organization is not wasting time and archived records. Or, by procedure, the hard copies should
resources in overbuilding (i.e., hyper-scoping the training be appropriately disposed.
recordkeeping process).24

THE TRAINING UNIT AS AUDIENCE


TACTICS OF TRAINING RECORDKEEPING The training unit itself is another audience for the documen-
Training recordkeeping includes training records (or atten- tation of training. First of all, the training unit can use train-
dance sheets), training assessments, and curricula or ITPs. ing records to document the level of effort of the unit. This
In each case, this is a controlled document; each has neces- documentation can be used in annual budget sessions to jus-
sary fields and optional (“nice-to-have”) fields. An example tify requests for more training staff or other resources.
of a “nice-to-have” feature of a training record would be Second, the training unit can use training records and
a field where a supervisor vouches for the record creator’s training assessments to test the accuracy of statements
qualification. about training.

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 43
Gordon Welty, Ph.D.

Documentation of training impact can be represented as Figure 3


a continuum ranging from an endpoint reflecting training Post-training Performance.
inputs to an endpoint reflecting training outputs (see Figure
Anticipated
2). At the first endpoint on the continuum, training impact results
is documented by staffing levels of the training unit. The
Desired
supposition is, “The larger the training staff, the greater the performance
training impact.” This is clearly only a measure of inputs, a
very rough proxy for training impact. Actual
At the other end of the continuum, training impact is results

documented by return on investment (ROI) for training. Current


This can be calculated in a number of ways. For instance, performance

the marginal benefit/cost ratio is the change in trainee pro-


ficiency divided by the change in the training unit’s expen-
diture on this training module.25 ROI is clearly a direct and Before training During training Post-training
salient measure of training outputs. Effective training leads
to improved performance, hence performance measures
should be used to document training impact. These mea- of performance—throughout the training event for the first
sures might include numbers of documents with defects, question, and during the immediate post-training period for
amount of rework, and other business metrics. the second. Such tracking, taking the form of performance
The number of seats occupied in training sessions falls assessments (most likely in the form of SDAs), will require
on that continuum, somewhere between staffing levels and a substantial recordkeeping effort.
ROI, nearer to the rudimentary endpoint of staffing levels.26 In the typical case, however, assessments are conducted
Other points on this continuum of the documentation of at only two points in time—one a pre-training assessment
training impact include training assessments such as knowl- just before the training event, and a second, post-training
edge transfer assessments (KTAs) and skill demonstration assessment, say an “intermediate test” conducted after the
assessments (SDAs).27 trainees have returned to the job. Given only two data
How can training documentation be used to test the accu- points, the average performance level would appear to be a
racy of theoretical statements about training? Consider the straight line, perhaps upward-sloping to the right, perhaps
following statements and accompanying graphic by Harold flat; in any case we would not be able to address the ques-
Stolovitch: “Because training is costly and removes work- tions about Stolovitch’s figure.29
ers from productive tasks, we expect significantly improved These uses of the documentation of training do not have
performance after training. Reality, however, is usually very the enterprise-wide significance of the other two audienc-
different, as illustrated by the post-training solid line” (see es—the operational use and the auditor’s use. The opera-
Figure 3).28 These statements and the graphic are quite in- tional staff represents the line of business. This audience
triguing to trainers. and its proactive use of training records for work assign-
Two methodological questions about Stolovitch’s figure ments directly relate to the bottom line. The auditors rep-
arise: What kind of evidence would be required to de- resent the regulatory requirements within which the orga-
termine whether there is a dip in performance during the nization will be profitable (or not).
training event (the second panel in the triptych)? What The training unit is usually viewed as an overhead depart-
kind of evidence would be required to determine if there is ment, engaging in capacity building and thereby (only indi-
a dip in performance following training (the third panel in rectly) contributing to the bottom line. Donald Kirkpatrick
the triptych)? and others have held that “trainers must justify their exis-
This article stresses the methodological point that the tence.”30 An effective training department should address
answer to each question depends on continuous tracking business performance issues, and “justify their existence”

44 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

by pointing to the improved performance of the workforce. ENDNOTES


1. See Gordon Welty, “Developing Assessments of Trainee Proficiency,” Journal
Otherwise the training department will be viewed as over-
of GXP Compliance, Vol. 12, No. 1 (Oct 2007), pp. 64-73. More generally,
head and not active contributors. In this sense, trainers are training materials are types of programmatic interventions, while assessment
indeed an audience for training recordkeeping. However, materials are types of measures of program effect.
2. See Margaret Hedstrom, ed. Electronic Records Management Program Strate-
we see that this is a distinctly secondary audience, behind gies,” Pittsburgh, PA: Archives and Museum Informatics (1993), and David
the two major audiences. Bearman, Electronic Evidence: Strategies for Managing Records in Contemporary
Organizations, Pittsburgh, PA : Archives and Museum Informatics (1994).
Good training recordkeeping may well contribute to a 3. Wendy Duff, “Ensuring the preservation of reliable evidence,” Archivaria,
corporate climate that supports widespread and disciplined Vol. 42 (1995), pp. 28-45, esp. p. 29.
use of organizational metrics. Such metrics allow bench- 4. Duff, op. cit., pp. 33-35. These are functional requirements of recordkeep-
ing, not necessarily GXP regulatory requirements. As we shall see, the func-
marking and trending to enhance organizational control. tional requirements do have GXP implications. For the current situation
on organizational compliance, see Roger Matus, “It Is No Longer Business
As Usual When It Comes to Handling Electronic Documents,” Journal of
Health Care Compliance, Vol. 9, Issue 2 (March 2007), pp 11-14, p. 73, and
CONCLUSION Darwin Stephenson, “Accelerating Compliance,” E-DOC Magazine, Vol. 21,
Issue 4 (July 2007), pp. 27-29.
There are two main audiences for training records, opera- 5. See Gordon Welty, “Strategy and Tactics of Task Analysis,” Journal of GXP
tional staff and auditors. In addition, there are other audi- Compliance, Vol. 11, No. 3 (April 2007), p. 28.
ences such as the training unit itself. To serve these audi- 6. Available at www.fda.gov/foi/warning_letters/s6537c.htm. See also “Arrow
warned about quality systems,” Reading Eagle (16 October 2007).
ences, training recordkeeping must possess characteristics 7. Federal Register, Vol. 62, No. 54 (20 March 1997), “Rules and Regulations,”
of good document management. At each level of the or- p. 13445. See also Gwendolyn M. Wise-Blackman, “Out-of-Specification
Results and the Quality System Approach to GMPs,” Pharmaceutical Technol-
ganization, document management must be appropriate so ogy, Vol. 30, Supplement (2006), p. S-6: “validation must be achievable with
that training recordkeeping will be “audit proof,” and will, staff at the work site.”
8. Available at www.fda.gov/foi/warning_letters/archive/g4601d.htm See
moreover, have business value to operational staff. also “Federal Regulators Find Fault with Stent-Making Practices at Florida’s
Regulatory compliance, as it relates to training record- Cordis,” Miami Herald (06 April 2004).
keeping, requires that personnel touching the product be 9. Available at www.fda.gov/foi/warning_letters/s6341c.htm.
10. Available at www.fda.gov/foi/warning_letters/g6208d.htm.
trained. FDA regulations, with few exceptions, do not ad- 11. Available at www.fda.gov/foi/warning_letters/archive/g5973d.htm
dress training records. It is important to review carefully 12. Available at www.fda.gov/foi/warning_letters/archive/g5973d.htm
13. Laurie Fischer, “Condition Critical: Developing Records Retention Sched-
the several cases where the regulations do apply and ensure ules,” Information Management Journal, Vol. 40, Issue 1 (Jan 2006), pp. 26-
compliance. It is equally important to ensure that the orga- 34; Ellie Myler, “The ABC’s of Records Retention Schedule Development,”
E-DOC Magazine, Vol. 20, Issue 3 (May 2006), pp. 52-56; Tina Torres,
nization is not wasting time and resources overbuilding the “Creating a Process-Focused Retention Schedule,” Information Management
training recordkeeping process. Journal, Vol. 40, Issue 5 (Sept 2006), pp. 62-69.
The fields that are necessary for training records, as well 14. Authorization involves authenticated users; usually this is two-factor
authentication involving two of the three factors: (a) What you know
as necessary roles and responsibilities, need to describe the (e.g., a password), (b) What you have (e.g., a security swipe card), and
fall-back process in case there are access or other system (c) What you are (e.g., a biometric characteristic). See Joan Engebretson,
“Federal Agencies Must Begin to Support New Standard for Logical Access
problems in a fully electronic approach to training record- this Month,” SDM: Security Distributing & Marketing, Vol. 36, Issue 8 (Aug
keeping. Validated electronic training tracking systems 2006), page 61; however, see Bruce Schneier, “Two-Factor Authentication,”
Communications of the ACM, Vol. 48, No. 4 (April 2005), p. 136.
should be employed to manage training records and train- 15. On predicate rules, see Tammala Woodrum, “21 CFR Part 11: The role of
ing assessments in an effective manner. predicate regulations and associated internal policies,” Drug Information
Training records can provide data to justify budget re- Journal, Vol. 37, No 2, (2003), pp. 159-164. See 21 CFR 211.198 (a) on
the requirement of written SOPs, 211.198 (b) on the requirement of written
quests. They can provide data to test the accuracy of state- records for each complaint, and 211.198 (b) (2) and (3) on the requirement
ments about training. The training unit’s use of these re- of written records for each investigation or the decision not to investigate.
16 For GLPs, see 21 CFR 58.63, “Maintenance and calibration of equipment;”
cords will not have enterprise-wide significance; yet, such for GMPs, see 21 CFR 211.67, “Equipment cleaning and maintenance;”
use can contribute to the overall impact of organizational also 21 CFR 211.182, “Equipment cleaning and use log;” and for medical
devices, see 21 CFR 820.70, “Production and process controls.”
metrics. GXP
17. For pharmaceutical employees, see 21 CFR 211.25; for biopharm personnel,
21 CFR 600.10; for non-clinical lab personnel, 21 CFR 58.29; for medical
device personnel, 21 CFR 820.25; for human tissue recovery personnel, 21

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 45
Gordon Welty, Ph.D.

CFR 1271.170. dicated that Part 11 was never intended to “significantly increase the costs of
18. See 21 CFR 58.29 (b), “Personnel.” As Robert McDowell has put it “It ap- compliance to an extent that was not contemplated at the time the rule was
pears that Part 11 would not apply to computerized systems holding GMP drafted,” nor was it intended to “discourage innovation and technological
training records, in contrast to GLP systems holding similar records where advances without providing a significant public health benefit.” See Office
the rules would apply,” in his “Quo Vadis: 21 CFR 11,” LC • GC Europe, Vol. of Compliance, CDER, op. cit. A hybrid recordkeeping system might best
17, Part 2, (Feb 2004), pp. 80-81. address the situation where the vast majority of training documents would
19. See 21 CFR 312.23(a)(6)(iii)(b). A curriculum vitae is more a record of be maintained in electronic form, and the few exceptions would be managed
educational attainment than a training history. Since the protocol is part of in paper form.
the IND regulatory submission, it will implicate Part 11 on that ground. 25. See Jack Phillips and P. Phillips, “Show me the Money: The Use of ROI in
20. See 21 CFR 211.34, “Consultants.” Performance improvement, Part 1,” Performance Improvement, Vol. 46, No. 9
21. But see Wise-Blackman, op. cit., p. S-10, who repeatedly refers to “software (Oct. 2007) and Jack Phillips “Measuring the ROI of a Coaching Interven-
that is 21 CFR Part 11 compliant and houses a database of training records.” tion, Part 2,” Performance Improvement, Vol. 46, No. 10 (Nov. 2007), pp.
The software that she advocates is not justified in terms of predicate rules— 10-23, esp. p. 18.
as we’ve seen, there aren’t any—but as follows: “One benefit of compliant 26. Chris Moore refers to the number of seats occupied in training sessions as
training software is the ease of routine scheduling of required training” and “fill rates;” see his “Learning from Key Learning Indicators,” Chief Learning
“Routine retraining can be accomplished efficiently through the use of group Officer, Vol. 6, No. 8 ( Aug 2007), p. 39.
sessions or individual Web-based compliant software.” Of course, “routine 27. On KTAs and SDAs, see Welty, “Developing Assessments of Trainee Profi-
scheduling” and “routine retraining” can be more easily and efficiently ciency,” op. cit., pp. 67-70. It is not the case, contrary to Wise-Blackman,
accomplished with any validated training tracking system, regardless of its op. cit., p. S-10, that “documenting the transfer of knowledge about the
Part 11 compliance. SOP is best accomplished through a Web-based system that incorporates
22. See 21 CFR 11.10 (i); see however Office of the Commissioner, “Guidance short quizzes as a prerequisite to receiving approval for training,” because
for Industry: Computerized Systems Used in Clinical Investigations,” Wash- there is a substantial legal exposure to the use of unvalidated KTAs (short
ington, DC: FDA (May 2007), p. 7: “Those who use computerized systems quizzes), and there are serious costs to validating KTAs; see Welty, op. cit. p.
must determine that individuals (e.g., employees, contractors) who develop, 68, esp. note 23. The trainee’s completion of prerequisites would best be as-
maintain, or use computerized systems have the education, training, and certained through a SDA, as would the documentation of trainee proficiency.
experience necessary to perform their assigned tasks…We recommend that 28. The quotation and figure are from Harold D. Stolovitch, “The Story of Train-
computer education, training, and experience be documented.” Neither a ing Transfer,” Talent Management Magazine, Vol. 3, No. 9 (Sept 2007), p. 12.
guidance nor a recommendation constitutes a predicate rule. 29. See Welty, “Developing Assessments of Trainee Proficiency,” op. cit., p. 70.
23. Office of Compliance, CDER, “Guidance for Industry; Part 11, Electronic Charles Tennant, M. Boonkrong, and P. Roberts have suggested three kinds
Records; Electronic Signatures—Scope and Application,” Washington, DC: of post-training assessments—an “immediate test” when the training has
FDA (Aug 2003), note 8. On the choice between electronic, paper-based, been completed, an “intermediate test” when the trainee has returned to the
and hybrid recordkeeping systems, see Rakesh Shukla, “The Case for job, and an “ultimate test” to measure behavioral changes; see their “The
Electronic Records Management,” Financial Executive, Vol. 20, Issue 7 (Oct design of a training programme measurement model,” Journal of European
2004), pp 50-52. See also Dan Riordan, “How Enterprise Software Facili- Industrial Training, Vol. 26, No. 5 (2002), pp. 230-240.
tates FDA Compliance,” Quality Digest, Vol. 27, No. 12 (Dec 2007), who 30. Donald L. Kirkpatrick, Evaluation of Training Programs, San Francisco, CA:
echoes the first predicate rules on page 33: “The FDA requires that medical Berrett-Koehler (1994), p. 18. The best justification for trainers’ existence is
device manufacturers and pharmaceutical companies give their employees probably recordkeeping that satisfies the needs of operational staff and GXP
adequate training in their job responsibilities, and in their roles in ensuring auditors
the quality of a company’s goods and services.” Riordan continues with a
list of software functions: “In compliance software systems, users can create
documentation for training requirements, job descriptions, qualification
definitions, courses, work instructions, and many other criteria. System
Article Acronym Listing
administrators can configure multiple levels of secure access, so that all
employee records remain confidential. Many systems have capabilities for FDA US Food and Drug Administration
managing and tracking employee certifications, scheduling courses and ITPs Individual Training Plans
training sessions, and monitoring employee performance. Approved changes SOP Standard Operating Procedure
to documentation can automatically update employee training records and
SISPQ Safety, Integrity, Strength, Purity, and Quality
simultaneously notify appropriate employees and managers of the need for
retraining on these documents. Automating these processes makes it much LMS Learning Management System
easier to keep employee records updated, and is an important element in ROI Return on Investment
FDA compliance.” KTAs Knowledge Transfer Assessments
24. See Tammala Woodrum, op.cit., esp. pp. 162-163 on the problem of organi-
SDAs Skill Demonstration Assessments
zational “policies that potentially expand the scope of Part 11.” FDA has in-

Originally published in the Spring 2008 issue of Journal of GXP Compliance

46 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Formative Evaluation in the ADDIE Model

INTRODUCTION of a pilot implementation, the results of the program evalu-


The ADDIE model provides high-level guidance for the de- ation can be fed back, closing the loop, facilitating further
velopment and revision of programs of all sorts, including refinement of the training program. This is called a “forma-
GXP training programs. The phases of the ADDIE model are tive evaluation.” As Robert Gagné and Leslie Briggs have
analyze, design, develop, implement, and evaluate. These stated5 “Formative evaluations provide data on the basis of
phases are sequential; each depends upon the successful which to revise and improve the materials, the lesson plans,
completion of the preceding phase. the performance tests, and indeed the operation of the en-
The analysis phase of the ADDIE model identifies a per- tire instructional system.” If the evaluation shows that the
formance gap that can be addressed by a training program training module has shortcomings, those shortcomings are
(i.e., a set of training and assessment materials, a qualified fed back to be analyzed again. Further design and develop-
trainer, and a training audience). A performance gap is a dis- ment efforts follow, until the module meets organizational
crepancy between a standard stipulated in a standard operat- needs. Thereupon there is a final implementation, and an
ing procedure (SOP) and some employee performance 1. evaluation that documents the extent to which the training
This is followed by the design phase, where a carefully program meets the organization’s needs. This is called a
planned approach, documented in a training outline, is “summative evaluation.” “Summative evaluation is usually
prepared to address the performance gap. Also, behavioral undertaken when development of an instructional entity is
objectives are specified in the design phase2. in some sense completed, rather than on-going. Its purpose
If management approves the design, the development is to permit conclusions to be drawn about how well the
phase comes next, where the training program—the train- instruction has worked,” state Gagne and Briggs.6
ing materials and the assessment materials—is developed The ADDIE model can be conceptualized as having two
to address the performance gap. There are four compo- paths out of the development phase. One path leads to
nents of the development phase: identifying the objectives pilot implementation, followed by formative evaluation,
of the training module, based on the relevant operational from which a feedback loop allows further analysis, design,
SOP and the training outline; preparing the training materi- and development. At some point, determined by manage-
als, with special attention to structured on the job training ment, the training program is judged to be ready for the
(SOJT) materials and e-learning materials; preparing the as- other path. As Gagne and Briggs7 have pointed out, “There
sessment materials; and assembling these materials into the is no standard number of formative evaluations that small
training program3. components or segments or the entire system undergo. The
The implementation phase of the ADDIE model follows, number depends on the budgets and time available, the de-
where the training materials and assessment materials are gree of excellence set as the system design objective, and the
rolled out, either provisionally in a pilot implementation total circumstances surrounding the project.” The program
(e.g., a proof of concept study) or in a final implementation. then moves to final implementation, followed by summa-
The notion that this phase may be a “pilot” of the training tive evaluation (see Figure 1).
module, rather than just a finalized rollout, highlights the There are several ways to conceptualize the ADDIE model
iterative feature of the model.4 at this point. One is to include pilot implementation and
The evaluation phase follows implementation. In the case formative evaluation within the development phase. When

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 47
Gordon Welty, Ph.D.

Figure 1 dictates that intervention (e.g., training materials, training


The two paths of the ADDIE model. “script,” etc.) must remain invariant during the program
cycle. Friedman, Furberg, and DeMets8 state that a clini-
cal trial study protocol “should be developed before the
beginning of subject enrollment and should remain essen-
Analyze
tially unchanged except perhaps for minor updates. Care-
ful thought and justification should go into any changes.
Major revisions which alter the direction of the trial should
Design
be rare.” If preliminary findings were fed back, this would
allow a modification of the training materials and other
Develop aspects of the program, thus invalidating design and the
evaluative research findings.
Pilot implementation Final implementation This position has until recently been held regarding clini-
cal trials in the pharmaceutical industry. As Derek Lowe9
has expressed it, in a clinical trial, you “establish your ‘null
Formative evaluation Summative evaluation
hypothesis’ (typically that your drug is no better than a pla-
cebo or the current standard of care) and you start collect-
ing data, in the hopes that you’ll fail to prove it. Everything
stays carefully blinded. The investigators have no idea what
the pilot and the formative evaluation are completed, the they’re administering and the patients have no idea what
program moves into the (final) implementation phase, fol- they’re taking until a predetermined endpoint—to do oth-
lowed by the (summative) evaluation phase. Another con- erwise would destroy the statistics.” Notice the significance
ceptualization is to include two types of implementation, of “blinding” in clinical trials, particularly “double blind-
pilot and final, within the implementation phase, and two ing,” where both subjects and investigators are unaware
types of evaluation, formative and summative, within the of the assigned intervention—whereby findings for pro-
evaluation phase. These different conceptualizations bear gram improvement cannot straightforwardly be fed back10.
on the logic of the ADDIE model, but not on the process of In contrast to the logic of blinding, the actual conduct of
program development. blinding in randomized clinical trials (RCTs) has been as-
As a final introductory point, it is clear that management sessed in several recent studies, including Boutron et al.11
has one very significant role in a formative evaluation; that and Fergusson et al.12.
is specifying the overall goal, and level of effort, for the eval- This position has been held by researchers outside the
uation. What might be management’s response to evaluative field of clinical trials as well: Michael Brooks13 states that
findings gathered during the formative evaluation of a pro- continuous feedback of evaluative findings “. . . has the
gram? The response of the program designers and develop- unfortunate effect of tossing a monkey-wrench into the re-
ers to evaluative findings is clear; they will consider making search design constructed at the program’s outset.” Daniel
program improvements based on the evaluative findings. Stufflebeam, a leading figure in the evaluation community,
What about the response of management? describes14 the development of his own position, “I had to
reject basically everything I had thought necessary for evalu-
ating educational projects, including behavioral objectives,
FEEDBACK VERSUS RESEARCH DESIGN experimental designs, and standardized tests. Instead, I ad-
The question often arises in the formative evaluation of a vised educators to key evaluations to provide information
training program or other program: what is the effect of the for decision making.”
dissemination of evaluative findings during the life of the The argument against the “experimental method” is a
program? It is often assumed that the “experimental design” methodological, not a practical argument15. The critics of

48 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

experimental design are speaking of characteristics inherent design. Of course, desirable modification practice does
to evaluation theory that account for a sharply limited util- not (or should not) mean a hodgepodge array of “random”
ity. The critics are not suggesting that formative evaluation interventions resulting from poor program definition; this
would be more successful if the experimental designs were should have been preempted in the design phase of the AD-
more precisely constructed, if randomization of subjects DIE model. Nor should “random” interventions result from
were more diligently pursued, or if experimental methods the capacity of those who implement training programs to
were more carefully practiced. According to the critics, ex- understand adequate definitions; that should have been
perimental method in program evaluation, especially RCT, addressed in the implementation phase18. It makes no dif-
is defective. ference, for the experimental method, whether an evalu-
The importance of this matter can hardly be overstressed. ative judgment of program ineffectiveness is available for
As indicated above, feedback of evaluative findings is of vi- program adaptation or not. It makes no difference, for the
tal importance for improving the process in training and de- experimental method, whether changes in training inter-
velopment. If there is an incompatibility between feedback vention are implemented or not. Evaluators can fill their
and the “experimental method,” one obviously must be mandate for dissemination of timely data and concomitant
abandoned. But to abandon the former, evaluators forego programmatic change.
their mandate to provide timely and relevant information The evaluator can realize, based on an on-going program
for program adaptation. To abandon the latter, they seri- evaluation, that “training intervention G will not produce
ously limit the research techniques they have available for the desired results.” The intervention can be revised in the
evaluating the program; instead of Donald Campbell and “middle of the stream,” so to speak, and evaluators can still
Julian Stanley’s famous “Experimental and Quasi-experi- complete their formative evaluation. The dissemination of
mental Designs,” the formative evaluator is restricted to just evaluative findings through an appropriate study monitor-
the quasi-experimental designs and even more inferior ap- ing committee, and a managerial reaction to such findings,
proaches, such as “pre-experimental designs”.16 As Green enhancing the likelihood of program success, will not inval-
states, “RCTs are the gold standard of treatment trial meth- idate the evaluation effort, even though an initial judgment
odology, and to deprive complex (often psychosocial) in- predicted program failure. Only a misunderstanding of the
terventions of their imprimatur is potentially to undervalue nature of evaluative research could foster the view that the
these areas in an evidence-based climate17.” Moreover, this training intervention is fixed.
limitation sacrifices what rigor the evaluators’ discipline has. Assume that an evaluation of a training program is un-
We find, however, that the critics of “experimental design” derway. The program, in essence, takes a set of inputs and
have misplaced their criticism. given conditions, Z, and by means of some process G, trans-
The next section will give an existence proof that forma- forms the inputs into an output described by the dependent
tive evaluation can be conducted within the framework of variable x. We assume x to be a behavior. The dependent
experimental design, and evaluative findings can at the same variable may be a business measure, such as number of re-
time be provided for improvement of the training program. worked batches, or an index, such as OSHA recordables.
This means that the full range of evaluative approaches is The evaluator is randomly assigning employees to control
available to the formative evaluator, including experimental or treatment groups, manipulating variables, recording and
designs (i.e., RCT) as well as quasi-experimental designs. communicating results, etc.
Thus behavior x is a function G of a complex state of af-
fairs z, given by
THE GOOD NEWS, PART 1
It is not the case that training intervention must remain in- x = G(z). [Equation 1]
variant. Program enhancement, in light of feedback from
a formative evaluation, can take place concurrently with This says G and an index z of the set of independent vari-
an evaluation in the framework of the RCT experimental ables Z are sufficient for the prediction of the dependent

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 49
Gordon Welty, Ph.D.

Figure 2 Figure 3
Performance levels of hypothetical program. Reaction function for hypothetical program.

G(z)
F
R(x)
X
X(1)
X*(1)
X(0)

X*(0)

0 z(0) z(i) z 0 X(0) X(1)

variable x, in the absence of dissemination of G or z. This might react to the report and implement a change from G
can be represented by a two dimensional diagram (see Fig- to G*. An example of such a change would be mandat-
ure 2). ing all training programs be offered early in a shift. Then
We note that for a given interval [z(o), z(i)], x will have a the output, rather than being x would be x*.19
range of [x(o), x(i)]. Thus z might be an index of prior train- Let a reaction function R be introduced, indicating the
ing history, on-the-job experience, etc. and x, a measure of dependence of the actual outcome x* on the program man-
productivity such as unit output, impounded batches, or ager’s knowledge of the disseminated judgment (or predic-
quantity reworked. The set Z would include such items as tion) of x. This is given by
appear in the employee’s training history, etc. We assume
throughout that the interval of x is continuous and closed. x* = R(x) [Equation 2]

Given the relevant range [x(o), x(i)], we can represent


THE COEXISTENCE OF RCT AND the reaction function by a two-dimensional diagram (see
Figure 3).
DISSEMINATION OF RESULTS
With the variance of x through the range x(o), to x(i) will
Consider the following scenario. G is the training and
be associated a variance of x* between x(o)* and x(i)*. If
qualifying process, exemplified in a particular train-
ing program and including training materials, training R(x) is continuous over [x(o), x(i)], and if R(x) is bounded
“script,” etc. Through an appropriate channel such as the (i.e., 0 < R(x) < F) then by the generalized Brouwer Fixed-
study monitoring committee, the program manager has point Theorem there exists at least one x and one x* such
discovered a credible RCT evaluation report indicating that, x = x*. Also, for x = x*, the system described by equa-
that some aspect of G was tending to increase the quan- tions 1 and 2 is in equilibrium20 (i.e., the manager will cease
tity of rework. Say this aspect was the hour in the shift to react to x). Thus, for x = x*, that value of x is the correct
(i.e., whether the training event occurs early in the shift public prediction, as well as the correct formative evaluative
or late). Say further that the manager would be held ac- judgment.
countable for the increase of rework. Then the manager In this section, we have shown that formative evaluation

50 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

can be conducted within the framework of experimental cal and Scientific Affairs, has expressed it, FDA is interested
design, and evaluative findings can at the same time be pro- in “adaptive sampling designs, including response-adaptive
vided to the manager of the training program that is being designs for statistical experiments, where the accruing data
evaluated, via the study monitoring committee or another from experiments—the observations—are used to adjust
appropriate channel. This means that the full range of eval- the experiment as it is being run23.” He goes on to say “the
uative approaches is available to the formative evaluator, advantages of these approaches, rigorously designed, are
including not only quasi-experimental designs but experi- becoming more evident, including among the ranks of our
mental designs (i.e., RCT) as well. experts at FDA. It’s essential that we at the FDA do all we
can to facilitate their appropriate use in modern drug devel-
opment.” Gottlieb discusses several adaptive approaches to
THE GOOD NEWS, PART 2 the design of experiments including the following:
The preceding material incorporated an existence proof, • In an adaptive clinical trial, patient outcomes can be
showing that under specified conditions, training program used as they become available to adjust the allocation
modification could take place in response to evaluative of future patients or some other aspect of the study de-
findings developed within an experimental design. The fol- sign
lowing questions can still be raised: What are the implica- • A second type of adaptive trial design involves ongoing
tions of this for evaluation practice? Does anyone care? The assessment of the sample size, to avoid under- or over-
answer to these questions is yes. allotment of patients.24
Let us look at a methodologically analogous situation, • [Another includes] seamless designs that allow learn-
that of clinical trials of investigational new drugs. There ing to be more iterative and less method-limited. That
is a long history of interest in adaptive designs in clinical allow continuous discovery that isn’t defined by phases
trials, dating from Abraham Wald’s pioneering researches in but rather by what we learn as we go.25
the 1940s.21 The US Food and Drug Administration has ex- Gottlieb acknowledges that “adaptive approaches are not
pressed interest in adaptive clinical trials and the associated a panacea to all of our challenges, and enabling them is not
research designs. The dilemma of clinical trials has been a sure thing. Adaptive procedures are more complicated to
described well by Derek Lowe: design and to analyze, and in some settings are more dif-
ficult to implement.” Moreover, he is well aware of “trepi-
“In too many cases, the chief result of a trial is to dation about the use of adaptive features and reluctance to
show that the trial itself was set up wrong, in ways consider a variety of enrichment and adaptive designs. In
that only became clear after the data were unblinded. many cases, researchers are still unaware of the option to
Did the numbers show that your dosage was subop- use adaptive designs because standard statistical courses
timal partway into a two-year trial? Too bad—you and packages do not include them.” (op. cit.)
probably weren’t allowed to know that. Were several There are political and ethical issues here as well. “Pur-
arms of your study obviously pointless from the start? ists will argue that changing a trial midway through a study
Even if you knew, what could you do about it without somehow benefits pharmaceutical companies by potentially
harming the validity of the whole effort?” (22) allowing them to manipulate results. Some worry that bias
is more likely when results are known during the trial, com-
The problem is to conduct clinical trials so that both the pared with keeping trials blind,” notes Steve Zisson.26 Con-
rigor of the experimental design (RCT) will be maintained crete proposals are under consideration to mitigate such
throughout, and program revision can occur, based on the worries.27
timeliest data. And, methodologically speaking, that is pre- Since FDA is interested in adaptive designs for the study
cisely the problem cited by Lowe, with reference to the of investigational new drugs, it is unlikely they would object
evaluation of training programs. to the use of adaptive designs in the formative evaluation of
As Scott Gottlieb, FDA Deputy Commissioner for Medi- training programs. What works for clinical trials can just as

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 51
Gordon Welty, Ph.D.

well work for the evaluation of training initiatives. MANAGEMENT’S PREROGATIVE


Several uses of such adaptive designs include the Management has an overarching role in a formative evalua-
following: tion. That role is to specify the overall goal, and level of ef-
• The formative evaluator can communicate interim fort, for the evaluation. What does management want from
training outcomes through a channel such as the study this evaluation? There is a range of possibilities here. Does
monitoring committee to the program manager, allow- management want the most credible evaluative report pos-
ing timely revision of the training intervention, includ- sible? Or does management want the most blatant problems
ing revision based on comparison of programmatic al- in the pilot project to be corrected? The evaluator must ne-
ternatives gotiate with management to determine the actual goal.
• The evaluator can use interim training outcomes to al- Once management’s goal is set, the evaluator can recom-
low more effective assignment of trainees to particular mend approaches to aspects of the formative evaluative de-
training sessions, for example by sequential sampling. sign, such as the following:
“The manner of conducting formative evaluations varies • Parallel group or cross-over design
widely,” state Gagné and Briggs.28 We are suggesting that • Recruitment of trainees
one approach to formative evaluation of training programs • Random assignment
is utilizing an adaptive RCT design. “Quantitative data are • Blinding
definitely necessary for formative evaluation,” state Gagne • Sample sizes
and Briggs.29 • Statistical analyses.
The steps in conducting a formative evaluation can be With recommendations of costs and benefits of each ap-
summarized as follows: proach, management can decide between the approaches.
• The first step is to develop a formative evaluation plan A memorandum of understanding between management
for the training module, including an evaluation design, and evaluator can then be prepared, including a statement
and any evaluative instruments of the level of effort that will be required to attain the goal
• The second step is to collect evaluative data as you be- that management has set.
gin to pilot the training module, including data from At that point the evaluator can begin to plan the logistics
both the pilot trainees and from your training and de- of the formative evaluation. The primary audience for this
velopment peers evaluation will be the instructional designer who will make
• The third step is to review all the evaluative data you program revisions as warranted.
have gathered, in light of the statistical portion of the
formative evaluation plan
• Then, prepare an evaluation report summarizing the CONCLUSION
evaluations; propose revisions to the training module This article has reviewed the possibilities for formative eval-
• Get the study monitoring committee as well as manage- uation of training programs as well as any other kind of pro-
ment approval of these revisions gram, within the framework of the ADDIE model. It can
• The sixth step is to utilize the feedback for program be concluded that the formative evaluation of training pro-
improvement grams can utilize the full range of experimental and quasi-
• Then, continue the pilot (with further adaptations as experimental designs, as well as any other approaches. In
required), until management is satisfied that the train- this paper, the possibilities of employing adaptive designs
ing module meets the organization’s needs. have been considered. Thereby, the data gathered in that
The essence of this process is the negotiation between the evaluative effort can at the same time be made available to
evaluator and program manager. This negotiation works management, during the course of the training process, to
towards a settlement that takes into account both method- allow decisions to be made about program improvement.
ological rigor on the one hand, and program goals and val- FDA has recently expressed interest in the use of adaptive
ues on the other. designs in clinical trials.

52 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Reported in a Random Sample of Randomized, Placebo Controlled Trials,”


This does not mean that the evaluator must use any par- British Medical Journal, Vol. 328, February 2004, p. 432.
ticular design or approach. While the full range of meth- 13. Michael P. Brooks, “The Community Action Program as a Setting for Applied
ods and techniques is available, the decision about which of Research,” Journal of Social Issues, Vol. 21, No. 1, 1965, p. 38; see also Susan
Jacobson, Julie Morris, J. Scott Sanders, Eugene Wiley, Michael Brooks, et
those techniques and methods will depend upon two fac- al., “Understanding Barriers to Implementation of an Adaptive Land Man-
tors. One is management’s goal for the evaluation; the other agement Program,” Conservation Biology, Vol. 20, No. 5, 2006, 1516–1527,
esp. p. 1518: “Adherence to an experimental design […] may preclude some
is the needs of the main audience of the formative evalua- of the flexibility managers have experienced previously.”
tion, namely the needs of the instructional designer who 14. Daniel S. Stufflebeam, “The CIPP Model for Evaluation: An Update,”
will revise the training program. The formative evaluation Presented at the 2003 Annual Conference of the Oregon Program Evalua-
tors Network (OPEN), Portland, OR, October 3, 2003. See also his early
and re-piloting of the training module can continue until “The Use and Abuse of Evaluation in Title III,” Theory into Practice, Vol.
management has decided that the needs of the organization VI: 3, 1967, p. 128: “the application of experimental design to evaluation
problems conflicts with the principle that evaluation should facilitate the
have been met. Once the formative evaluation has been continual improvement of a program. Experimental design prevents rather
completed and all necessary changes to the module have than promotes changes in the treatment because treatments cannot be
been made, it is time to move to final implementation of the altered in process if the data about differences between treatments are to be
unequivocal.” Subsequently, in “MetaEvaluation,” Occasional Paper Series,
training program. GXP No. 3. Kalamazoo, MI: Evaluation Center, Western Michigan University,
1975, p. 54, he maintained that “experimental design often would not
provide timely feedback for decision making.” See also his “Evaluation
Checklists,” American Journal of Evaluation, Vol. 22, Issue 1, Winter 2001, p.
ENDNOTES 72: “almost everything I had learned about experimental design, measure-
ment, and statistics was largely irrelevant to evaluating new, heavily funded,
1. Gordon Welty, “Strategy and Tactics of Task Analysis,” Journal of GXP Com-
but ill-defined projects […] Gradually, I began to evolve an approach to
pliance, Vol. 11, No. 3, April 2007, pp. 26-34.
evaluation that seemed to work […] The approach was directed to design-
2. G. Welty, “The ‘Design’ Phase of the ADDIE Model,” Journal of GXP Compli- ing evaluations that would address stakeholders’ evaluative questions and
ance, Vol. 11, No. 4, July 2007, pp. 40-52. See also John W, Hansen, provide them a flow of timely, relevant information.”
“Training Design,” Advances in Developing Human Resources, Vol. 8, No. 4,
15. On the often misused term methodology, cf. Fritz Machlup, “Papers and
November 2006, pp. 492-499.
Proceedings,” American Economic Review, Vol. 53, 1963, p. 204.
3. G. Welty, “Developing Assessments of Trainee Proficiency,” Journal of GXP
16. Donald T. Campbell and Julian C. Stanley, Experimental and Quasi-experi-
Compliance, Vol. 12, No. 1, October 2007, pp. 64-73.
mental Designs for Research, Chicago: Rand McNally, 1963.
4. G. Welty, “Strategy and Tactics for Pilot Implementation in the ADDIE
17. Jonathan Green, “The Evolving Randomized Controlled Trial in Mental
Model,” Journal of GXP Compliance, Vol. 12, No. 2, January 2008, pp. 12-19.
Health,” Advances in Psychiatric Treatment, Vol. 12, 2006, p. 270.
5. Robert Gagné and Leslie Briggs, Principles of Instructional Design (2nd ed)
18. As Gallo et al. have put it regarding desirable modification practice, “chang-
NY: Holt, Rinehart and Winston (1979), p. 37; see also p. 290: “Evidence
es are made ‘by design,’ and not on an ad hoc basis; therefore, adaptation
of an instructional program’s worth is sought for use in making decisions
is a design feature aimed to enhance the trial, not a remedy for inadequate
about how to revise the program while it is being developed. In other
planning;” see Paul Gallo, et al. “Adaptive Designs in Clinical Drug Develop-
words, the evidence collected and interpreted during the phase of develop-
ment,” Journal of Biopharmaceutical Statistics, Vol. 16, Issue 3, June 2006,
ment is used to form the instructional program itself.”
pp. 275-283, esp. pp. 276. On the project management requirements this
6. Gagné and Briggs, op. cit., p. 293. See also Joseph S. Wholey, “Formative raises for the roll-out and maintenance of a training program, see John N.
and Summative Evaluation: Related Issues in Performance Measurement,” Fabac, “Project Management for Systematic Training,” Advances in Developing
Evaluation Practice, Vol. 17, Issue 2, Spring 1996, pp. 145 ff. and Greg Wang Human Resources, Vol. 8, No. 4, November 2006, pp. 540-547.
and Diane Wilcox, “Training Evaluation,” Advances in Developing Human
19. The general treatment of these cases is given in Emile Grunberg, “Some
Resources, Vol. 8, No. 4, November 2006, pp. 528-539, esp. pp. 529-530.
Methodological Observations on Macroeconomics,” Konjunkturpolitik, Vol.
7. Gagné and Briggs, op. cit., p. 38. 13, 1967, pp. 34-37. Cf. also E. Grunberg and Franco Modigliani, “The
8. Lawrence Friedman, et al., Fundamentals of Clinical Trials, Boston: John Predictability of Social Events,” Journal of Political Economy, Vol. LXII , 1954,
Wright, 1981, p. 6. pp. 465-478.
9. Derek Lowe, “Adaptive Trials,” Pharmaceutical Executive, Vol. 26, Issue 7, 20. Andrzej Granas and James Dugundji, Fixed Point Theory, NY: Springer-
July 2006, pp. 70-77, esp. p. 72. Verlag (2003); see also Stefan Banach, “Sur les opérations dans les
10. See, for example, Kathryn Webert, “Treatment Allocation in Clinical Trials,” ensembles abstraits et leur applications aux équations intégrales,” Funda-
Transfusion, Vol. 47, Dec. 2007, pp. 2187-2188, also Damian McEntegart, et menta Mathematicae, Vol. 3, 1922, pp.133-181 and Luitzen Brouwer, “Zur
al. “Blinded by Science with Adaptive Designs,” Applied Clinical Trials, Vol. Invarianz des n-dimensionalen Gebiets,” Mathematische Annalen, Bd. 72,
16, Issue 3, March 2007, pp. 56-64. A classic statement is found in Fried- 1912, SS. 55-56.
man et al., op. cit., pp. 58-67. 21. Abraham Wald, Sequential Analysis, NY: John Wiley, 1947, and his Statis-
11. Isabelle Boutron, et al., “Methods of Blinding in Reports of Randomized tical Decision Functions, NY: John Wiley, 1950; also Oskar Morgenstern,
Controlled Trials Assessing Pharmacologic Treatments,” PLOS Medicine, Vol. “Abraham Wald, 1902-1950,” Econometrica, Vol.19, No. 4, October
3, Issue 10, October 2006, pp. 1931-1939. 1951, pp. 361-367. See also Friedman et al., op. cit., pp. 48-52 and pp.
144-154.
12. Dean Fergusson, et al., “Turning a Blind Eye: The Success of Blinding
22. Derek Lowe, “Adaptive Trials,” op. cit., p. 72

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 53
Gordon Welty, Ph.D.

23. Scott Gottlieb, “Remarks,” Speech before the 2006 Conference on Adaptive
Trial Design, Washington, DC, July 10, 2006. See also Anna W. Mathews,
“FDA Signals It’s Open to Drug Trials That Shift Midcourse,” Wall Street
Journal, July 10, 2006. Article Acronym Listing
24. See also Paul Gallo, op. cit., pp. 281-282.
25. See also Paul Gallo, op. cit., pp. 280-281. ADDIE Analyze, Design, Develop, Implement, Evaluate
26. Steve Zisson, “Adapting To A New Kind Of Trial,” Clinical Trials Today, 17 FDA US Food and Drug Administration
July 2006. RCT(s) Randomized Clinical Trial(s)
27. See Paul Gallo, op. cit., pp. 278-279. SOJT Structured On the Job Training
28. Gagné and Briggs, op. cit., p. 290. SOP(s) Standard Operating Procedure(s)
29. Gagné and Briggs, op. cit., p. 291.

Originally published in the Summer 2008 issue of Journal of GXP Compliance

54 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Final Implementation of the Training Module

INTRODUCTION uct, and other US Food and Drug Administration regulated


The ADDIE model provides high-level guidance for the de- industries, the need for optimal delivery of the training is
velopment and revision of programs of all sorts, including constrained by the requirement that employees be trained
GXP training programs. The phases of the ADDIE model are before they are assigned to “touch” the product.
analyze, design, develop, implement, and evaluate. These At first glance, that requirement might seem to be trivi-
phases are sequential; each depends upon the successful al—just ensure that the training has been delivered “before,”
completion of the preceding phase. and be done with it. But the very dynamic of change that
The final implementation of a training module comes has driven manufacturing technologies as well as e-learning
among the last phases of the ADDIE model. A performance can create a climate of turbulence in process and procedure
gap or a training gap initiated the process, and a careful- that makes ensuring “before” quite risky, and raises the pros-
ly planned approach to address the performance gap has pect of serious compliance consequences if it turns out to
been prepared during the design phase. If management ap- be “after.” The requirement that employees must be trained
proves the design, the training program, including training before they touch the product becomes especially acute in
materials and assessment materials, has been created in the the case of final implementation of a training module, when
development phase. These training materials and assess- it is no longer a matter of selecting the trainees as it is in
ment materials are rolled out in a pilot implementation; a the case of a pilot. Each and every employee impacted by
proof of concept study highlighting the iterative feature of a new or revised procedure must be trained. This article
the ADDIE model. In the case of a pilot implementation, the examines that problem and considers several approaches to
results of the program evaluation are fed back, closing the addressing it.
loop, facilitating further refinement of the training program. First, we will review the scope and impact of the FDA
In the evaluation phase this is called a “formative evalu- regulations for pharmaceutical manufacturing in general,
ation”.1 Further design and development efforts follow, and about training in particular. The case for other regu-
until the module meets organizational needs. Then comes lated industries mentioned previously is the same. Next,
the final implementation of the training module. When the we will critically examine several approaches to ensuring
module has finally been implemented, it can be the object that employees are trained before they touch the drug prod-
of a “summative evaluation” that will estimate the relative uct, and find that each has shortcomings. Third, we will
cost and benefit of the finalized program in terms of orga- propose an alternative approach that facilitates the commu-
nizational goals. nication necessary to ensure the requisite training has taken
In this age of technological change, much attention has place in a timely fashion.
focused on the timing of training. On the one hand, train-
ing is optimally delivered close enough to task performance
to ensure that the skill enhancement is still relevant but not SCOPE AND IMPACT OF THE FDA REGULATIONS
yet forgotten. These requirements have led to just-in-time The FDA regulations for pharmaceutical manufacturing, set
training (JITT), which has benefited from e-learning and out in 21 CFR 211, are comprehensive in both scope and
other developments2. On the other hand, in the pharma- impact. Regarding scope, these regulations provide guid-
ceutical, biopharmaceutical, medical device, blood prod- ance for each person engaged in the manufacture, process-

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 55
Gordon Welty, Ph.D.

ing, packing, and holding of a drug product. The phrase The scope of this training will “relate to the employee’s
“each person” includes both employees and supervisors. functions;” the objective of this training will be “to enable
The phrase “manufacture, processing, packing, and hold- that person to perform the assigned functions.”
ing” is also comprehensive. It includes packing and labeling Moreover, 21 CFR 211.25(b) goes on to say that the su-
operations, testing, and quality control of drug products. pervisors of these persons shall be trained so as “to pro-
In sum we can say the scope of the regulations includes any vide assurance that the drug product has the safety, identity,
person who is touching the drug product or supervising the strength, purity, and quality (SISPQ) that it purports or is
persons who are directly touching the drug product. represented to possess.” In particular, these supervisors will
How do these FDA regulations impact on these persons? make the task assignments to the employees who will actu-
The regulations require that the pharmaceutical manufac- ally touch the product.
turer develop written standard operating procedures (SOPs) Three points follow from these stipulations. First, employ-
that provide guidance for a broad range of activities, includ- ees must have technical (or skill) training in their particular
ing the following: assignments. Second, the employees must have training in
• Equipment and facility cleaning CGMPS that constrain the exercise of skills. Third, supervi-
• Equipment calibration and maintenance sors are responsible for the SISPQ of the drug product and
• Handling of drug components and containers must be trained to fulfill that responsibility. All this training
• Production and process controls must take place in a timely fashion.
• Batch records
• In-process sampling and controls Training Or The Lack Thereof
• Quality lab controls How well have companies within the scope of 21 CFR 211
• Reprocessing of batches responded to these requirements? In a review of a sample
• Packaging and labeling operations of the FDA’s GMP warning letters sent during the five-year
• Warehousing period between January 2003 and December 20076, there
• Distribution were 25 warning letters that mentioned deviations regard-
• Returned drugs, among other activities. ing aspects of 21 CFR 211 during that time period. They
Moreover, these must be written procedures3. listed a number of observations that the FDA investigator
So a set of SOPs is required that will provide compre- had made during site visits to companies within the scope,
hensive guidance for dealing with the inputs, processes, including such issues as cleaning, contamination, sampling,
and outputs of drug manufacturing, as well as quality con- etc. Seven of these warning letters (over 25%) also cited
trol over this manufacturing. Not only are written SOPs inadequacy of training or inadequacy of the documentation
required; the regulations insist the quality unit approves of training, including inadequacy of skills training, training
them—they are controlled documents—and the procedures in GMPs, and supervisory training.
be followed4. This pattern is not a historical anomaly; FDA has been
Moving from the general to the particular, the FDA reg- concerned about the adequacy of training in the pharma-
ulations stipulate that all employees and supervisors be ceutical industry for some time. For example, regarding a
trained. 21 CFR 211.25(a) states that each person engaged somewhat earlier time period, FDA senior compliance of-
in the manufacture of a drug product shall be trained in the ficer Philip Campbell asked, “Are the employees trained?”
following5: He further inquired, “Are the supervisors trained?” Finally,
• I n the particular operations that the employee he asked “Are there records of that training, and is it ongo-
performs ing?”7.
• In current good manufacturing practices (CGMPs) The fact that more than a quarter of these FDA findings
• Including the CGMP regulations in chapter 211 point to problems in training should come as no surprise.
• The dozen or so written procedures required by these On the one hand, whenever there is a remediation (correc-
regulations. tive action and preventive action [CAPA]) for any deviation

56 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

investigation or audit observation, that remediation will and suggest retraining of the employee(s) as the corrective
usually involve a revision of procedure or other controlled action. This is goal displacement13; it places the unit’s goal,
document, which in turn almost invariably involves train- releasing the batch, above the organization’s goal, which is
ing to the revised SOP. As Carl Draper, Director of the FDA’s identifying the root cause and implementing a remediation
Office of Enforcement, has stated, “The implementation of that will ensure the deviation will not recur. This goal dis-
revised SOPs should include employee training”8. So train- placement results in a false alarm, where retraining is the
ing will be the indirect outcome of a remediation and will direct outcome of an investigation. The fallaciousness of
be the focus of some attention in the follow-up of the CAPA. retraining is amply demonstrated—retraining, retraining,
Thus we expect that any warning letter directly addressing retraining of the same employee(s), ad infinitum. As Philip
issues of cleaning, contamination, lab work, sampling, test- Lindemann points out, “Not identifying the cause of failure
ing, utilities, whatever may also include a call for training, may lead to additional failures”14. The investigator will rec-
or for better training. ognize this, as will upper management, if there are metrics
On the other hand, it seems that FDA has come to expect tracking CAPAs. The investigator and upper management
ineffective training9, or inadequate documentation of train- will thereupon question the adequacy of the organization’s
ing10. These expectations, along with the relative ease of investigations.
assessing the occurrence and documentation of training via Moreover, if “human error” were proposed as the root
the ubiquitous tracking systems and learning management cause of the deviation requiring retraining, then the actual
systems (LMSs), make the investigator’s focus on these areas root cause would be the following:
understandable. • Unreceptive trainee(s)
• Inadequate training materials
Training Versus Retraining • An unprepared or incompetent trainer
sequacy of training does not amount to a call for “retrain- • Ineffective interaction of trainee(s) and trainer
ing,” by which the employees that were originally trained • Some combination thereof 15.
are retrained to the same training materials, by the same For none of these cases would remediation be as simple as
trainer, in the same fashion. There is a substantial differ- retraining, because the trainee would need to be motivated,
ence between training as an indirect outcome of a CAPA the training materials would need to be revised, the trainer
and retraining as a direct outcome of an investigation as a would need to be qualified, or the interaction would need
CAPA itself. Investigators quickly recognize the fallacy of to be enhanced before the remediation could go forward.
retraining as a solitary or even major remediation11. For When John Levchuk calls for reinforcement training as
an example of such a fallacy, consider the FDA’s Adverse a remediation for “future skills deficiencies”16, he indicates
Determination Letter regarding the Baltimore manufactur- that refined or redefined training materials may be indicat-
ing facility of the American Red Cross, dated 27 July 2006. ed, because “usually, only those skills most likely to be for-
A Red Cross employee was not trained before touching the gotten or suffer compliance erosion over time would be tar-
whole blood product. When this problem was discovered geted for inclusion in a periodic reinforcement program”17.
two months after the event, the Red Cross conducted an in- Moreover, when he goes on to call for remedial training as
vestigation and concluded that this was a training problem. a remediation for “acquired skills deficiency,” he states that
“The corrective action was to fully retrain all employees” it would be “more appropriate and efficient if it were tar-
(12). FDA responded that “as a result of the incomplete geted to an incumbent’s specific skills deficiencies.” Thus
investigation, [the Red Cross] failed to determine all root Levchuk is not calling for retraining in either case.
causes of the problem”12. The Red Cross was then fined In this part we have reviewed the scope and impact of
more than $700,000. FDA regulations of pharmaceutical manufacturing in gen-
A manufacturing unit is strongly inclined to release an eral, and of training in particular, and found them to be
impounded batch by declaring that the catchall category comprehensive. Any person who touches the drug product,
“human error” was the root cause of the deviation or failure, or who supervises someone who directly touches the drug

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 57
Gordon Welty, Ph.D.

Figure 1
Three periods of SOP revision including training window.

Revision period Approval period Training window

Identify Revise Approve Implement


the SOP the SOP the SOP the SOP

product, falls within the scope of the regulations. These in time where the revision of the SOP is initiated and the
regulations impact on these persons via written SOPs that employees will touch the product.
provide comprehensive guidance for dealing with the in-
puts, processes, outputs, and the quality control of drug
manufacturing. These employees must be trained on THREE TYPICAL RESPONSES
these procedures insofar as they relate to the employees’ This section reviews three typical organizational responses
functions, so as to enable those persons to perform the as- to the necessity that all required training on the finalized
signed functions. As the process and procedures change, module has occurred before the employee is assigned to
the impacted employees must be trained in a timely fash- touch the drug product. Organizations tend to move through
ion, hence, the critical issues attending the rollout of a these three responses in turn; when one is found to be inad-
finalized training module. equate, the next is tried. The first proposal is typically the
The issue can be summarized as follows. An SOP is iden- notion that there is a “training burden” and suggestions are
tified as subject to revision. The SOP is revised and routed advanced to alleviate this burden. The second proposal is
for approval. At the point of approval, a “training window” to create a procedure for the supervisor’s checking of the
is opened (typically two weeks in length) before the SOP is employee’s training status. Last, various “technological so-
implemented. Within this training window, all impacted em- lutions” to the issue are proposed. None of these proposals
ployees (i.e., those who will be assigned to touch the product are adequate to the problem. They address the logistical
while executing the tasks controlled by the SOP) must be complications in part, but fail to take into account the lead-
trained. This is schematized in Figure 1, where the four criti- time and communication process that is necessary.
cal points are indicated on the timeline, delineating the three
periods: Revision, approval, and training window. Reduce The Training Burden
There are two major logistical complications that fol- Many organizations within the scope of 21 CFR 211 respond
low. First, all training and assessment materials must be to the requirements listed therein by focusing attention on
prepared before the SOP is approved; if not, they won’t be reducing the “training burden.” The organization’s manage-
available for the training that must be conducted within the ment believes that employees are spending too much time
training window. Second, the participants in the training in training—time removed from more productive activities.
must be identified and scheduled for training, again before Furthermore, the more time spent in training, or the more
the SOP is approved. This includes both the trainer(s) and training events that are attended, the greater the likelihood
the impacted employees who will be the trainees. In par- that a mistake will occur, and the untrained employee will
ticular, the employees must be those who will be assigned end up touching the product. The proposed solution is to
to touch the product under the guidance of the SOP. These reduce the number of SOPs required for any given position’s
two complications suggest that the timing of the requisite curriculum and reduce the training burden.18
training is critical, especially with reference to the points This is no solution at all. The number of procedures

58 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

required for a position depends upon the technological result of cost considerations, seeking to reduce rework and
sophistication (complexity) and proximity to the product reprocessing, or because of compliance concerns. In any
(criticality) of the tasks associated with that position.19 If case, a typical organizational response is to develop a new
the person is a janitor, there may be a few SOPs required. If SOP that calls upon supervision to check the employee’s
the person is a supervisor or SME of an aseptic filling line, training status. Such a controlled document can be called a
the number of SOPs may be very large. It is not a training task assignment procedure.
burden. It is a “technology burden,” and it is necessary to Such a procedure might require that the supervisor en-
face the implications of the fact that FDA regulates science- sure all necessary training and qualification requirements
based industries. These are technical operations issues, not have been completed and documented prior to assigning an
training issues. employee to touch the product. This check is typically per-
Managers are the only ones who can reduce the technology formed by looking at the employee’s training record in the
burden. Line management must review each and every SOP validated tracking system or LMS during task scheduling. If
that goes into a position’s curriculum. Essentially, the line employees have been trained on all the procedures listed in
manager must make a business case for each SOP versus each their curricula, the supervisor makes the task assignments.
job. Certain SOPs (e.g., one controlling the employee’s main- What if the supervisor makes a mistake in checking the
tenance of his or her locker) may well be eliminated. This is training records? What if the supervisor is not diligent,
where the following four elements would clear up these mis- overlooks a particular employee, or misses a page of the
understandings and mistaken senses of responsibility: training record? Referring again to the Red Cross example,
• Proceduralization of the training process where the employee was not trained before touching the
• Initiating an annual training plan product, the Red Cross concluded that, “the Education Co-
• Instituting an explicit risk analysis for every procedure ordinator failed to compare the employee’s previous train-
• The development of a training council (of line manage- ing transcript with the training requirements”.20
ment). Thereupon an organization might develop an even fur-
The only training issue is how to bring the employee up ther SOP that requires periodic checks by the quality unit
to task proficiency once an SOP is included in a given cur- of a random sample of employees found in GMP areas at a
riculum, and ensure that the level of proficiency (and docu- given time, to ascertain if they are in fact qualified for their
mentation thereof) is “audit proof” (i.e., will stand a regula- assigned job functions. Such a controlled document can
tory inspection). This means that line management, from be referred to as an “assignment monitoring procedure.”
the VP of Tech Ops on down, must factor in the time and Should discrepancies be found, the assignment monitoring
money (resources) to allow the training to take place. Via procedure would require the generation of a notice of event
the training council, the line managers who have identified (NOE) to inform management that a deviation has occurred.
the training needs will also approve the training resources, That NOE would need to address both the impact on the
for example, less training for janitors and more training for batch, to the extent the untrained employee had touched
supervisors of aseptic filling operations. No amount of talk the drug product, and the supervisory error itself.21
about a “training burden” will help ensure that the employ- There are several major problems with this organizational
ee is trained before being assigned to touch the product. approach. It presupposes that employees’ training curricu-
la, listed in the tracking system, correctly and currently in-
Creating A Procedure For The Checking Of The clude the procedures that are relevant to the tasks to which
Training Status the employees may be assigned. On the one hand, the cur-
In many cases, an organization will realize that it must take ricula may not correctly reflect the procedures. How does a
further steps to ensure that employees are trained on the supervisor ensure that every single procedure that relates to
relevant SOPs before they are assigned to touch the drug this task, or this process—regardless of who the originator
product. Sometimes this is a result of a deviation investi- of the SOP may be—has been included in this curriculum?
gation or an audit observation. Other times it may be the On the other hand, the curriculum may not currently reflect

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 59
Gordon Welty, Ph.D.

the procedures. How does the supervisor ensure that the the status of procedure initiation, the status of procedure
versioning up of each procedure has been the occasion for revision.
an update of the employee’s curriculum? Instead, the task assignment procedure should stipulate
These are hardly trivial questions. Change control and that the originator (or business owner) of any new or revised
change management are substantial problems in a regu- SOP should communicate with each and every impacted
lated industry subject to pervasive and persistent techno- functional area to determine who the impacted employees
logical development. As if that weren’t enough, procedures are (i.e., the training audience for the forthcoming SOP).
are versioned up to change a single word. Procedures are In terms of the timeline given in Figure 1, the originator of
versioned up and then found to be misaligned with higher the new or revised procedure must indicate to the manage-
corporate policies and standards; then they are versioned ment of each impacted functional area, the precise location
up still further to restore the status quo ante and alignment. on the document management timeline of that SOP. In par-
Procedures are versioned up, omitting key paragraphs; they ticular, the originator must indicate the SOP that has been
are subsequently versioned up to reinsert the omitted para- identified as subject to revision, and is in the process of
graphs. Multiple procedures coexist for similar functions being revised.
(e.g., gowning); these procedures are versioned up, one
by one, by their disparate business owners independent of Implement A Technological Solution
each other. Many organizations recognize that the proceduralization of
The remedy for the constant revision of procedures is a task assignment is inadequate and believe that a techno-
combination of making better business cases for proposed logical solution, usually in the form of a learning manage-
changes, and having critical review of the documents in ment system, will address the problem. For instance, Ed
process.22 But that remedy will not resolve the supervisor’s Cohen has recently identified three LMS “innovations” that
dilemma of task assignment. might jointly suffice to “effectively manage organizational
If the curriculum is either incorrect or not current, the su- compliance requirements.” The first innovation is where
pervisor cannot ensure the employee is adequately trained, “LMSs can automatically cascade all [procedural] changes
no matter how diligently the training record is checked, through the system; incorporate information and training
no matter how carefully the task assignment procedure is assignments tied to the change into development plans for
executed. And the assignment monitoring procedure will every employee impacted; and monitor employee records
most likely identify non-compliant task assignments after to determine whether system-assigned [training] tasks have
the fact. The only way to ensure compliance in this case is been completed”.24 This innovation is clearly the techno-
by over-training (i.e., by providing training to employees logical counterpart to the task assignment procedure; the
for whom the SOP may not be relevant). Of course, that is technological solution similarly begs the question by as-
not cost-effective training.23 suming that the employees’ training curricula correctly and
Moreover, over-training may result in employee resistance currently include the SOPs that are relevant to the tasks that
to training. Many times this occurs among high-perform- may be assigned.
ing individuals, say in a research and development institute, The second innovation discussed by Cohen is where
and presents special problems for organizational morale and an LMS protects the integrity of compliance related data.
productivity. “LMSs also are capable of safeguarding compliance-related
It is crucial to recognize the misspecification of task re- data through audit-trail functionality that allows any action
sponsibilities in the proposed task assignment procedure. impacting compliance data to be tracked and recorded”.25
This procedure places the key responsibility on the supervi- This innovation simply prevents an unauthorized interven-
sor for ensuring that employee training and qualification tion into the functioning of the LMS, so doesn’t bear on the
requirements are completed and documented prior to task issue of the supervisor’s task assignment.
assignment, while not giving that supervisor necessary in- The third innovation is the integration of the LMS with
formation about the accuracy and currency of the curricula, other business systems to ensure compliance. “For ex-

60 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

ample, a pharmaceutical industry employee who has not cess map of the development and utilization of a training
completed the necessary training required to operate a spe- outline, and the associated training audience list. Fourth,
cific drug dispensing machine cannot use his credentials we will discuss the use of the training audience list as the
to access and operate that machine until required training alternate approach to ensuring the requisite training occurs.
is completed”.26This innovation provides controls over the To anticipate, the training outline will be recommended as
execution of tasks by untrained employees. It does not fa- best practice to ensure that all requisite training on the fi-
cilitate task assignment. It is a matter of timing; the su- nalized module has occurred before employees are assigned
pervisor must ensure that training has occurred for revised to touch the drug product.
procedures prior to making the task assignments, not prior
to the employee’s executing the tasks. It is very late in the The Purpose Of An SOP
game when an employee is prevented from using creden- A procedure lists the necessary steps (tasks) that, taken to-
tials to enter a GMP area or to operate a specific machine. gether, are sufficient to produce the desired process result.
The employee may have been assigned a task for which the It can address several kinds of process: a person-to-machine
requisite training has been completed, yet would be denied process, a person-to-paper process, a person-to-person pro-
access to the area or machine because other training has cess, or some combination of the three types. An SOP, typi-
not been completed. Such an innovation might limit the cally in documentary form, indicates the sequence of tasks,
number of employees available, resulting in the shutdown the personnel or positions that are responsible for the tasks,
of the entire production line. The supervisor still does not and the standards that define the satisfactory completion of
have the necessary lead time and information to make task the tasks.27
assignments.
It is critical that none of the LMS innovations that Cohen The Purpose Of Training To A Procedure
discusses address the supervisor’s real-world problem of Training is a person-to-person process that prepares each
task assignment. Like the preceding proposal for a task as- employee (the trainee) to successfully execute the steps
signment procedure, this “technological solution” makes the (tasks) in a procedure, in the appropriate setting, stipulated
supervisor responsible for employee training and qualifica- order, mandated workgroup, and specified timeframe. Train-
tion requirements, while not giving that supervisor timely ing is the combination of trainee(s), training materials, virtual
information about the status of procedure initiation and the or actual trainer, and the interaction of these elements.
status of procedure revision. Thus procedures and training are different. The proce-
Instead, we must ensure that the originator of a new or dure is a controlled document subject to the quality unit’s
revised SOP has communicated with each impacted func- approval. Training is an interactive process. Of course a
tional area to determine the training audience for that forth- procedure can be the object of training, and training can be
coming SOP. And this brings us to the final part of this made into a procedure. But the two are distinct; reading a
paper, where we propose an alternative approach to ensur- procedure (a person-to-paper process) is not the same as
ing that the requisite training on the finalized module has being trained on that procedure28; being trained on a pro-
occurred before the employee is assigned to touch the drug cedure is not the same as being a subject matter expert on
product. that process.
How do we align the procedure and its associated train-
ing? How do we provide the supervisor with necessary in-
THE ROLE OF THE TRAINING OUTLINE formation about changes to relevant procedures so as to en-
This section addresses four topics. First we will compare sure that employee training and qualification are completed
and contrast the purpose of an SOP with the purpose of and documented?
training to a procedure. Next we will delineate the role of a
training outline as a brief summary of the training implica- The Role Of The Training Outline
tions of a new or revised SOP. Third, we will present a pro- The training outline is a controlled document that provides

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 61
Gordon Welty, Ph.D.

a brief summary of the training implications of a new or • Does the SOP or other document thoroughly identify
revised procedure.29 The following are the typical 12 fields CGMP aspects?
in a training outline: • Is all relevant training information covered in the train-
• Course title ing outline?
• Course number and version • Will all facilitators present the training and information
• Training audience consistently?
• Curriculum fit In the case of non-lifecycle documents and GMP
• Prerequisite courses and required skills regulatory training, the instructional designer can ask
• Trainers management about the range of the training audience;
• Behavioral objectives usually it will straightforwardly be all employees, all
• Training delivery method managers, etc.
• Course length In the case of a lifecycle document, the instructional
• Special instructions designer will review the SOP scope statement as well as
• Measures of effectiveness the task responsibilities, and generate a provisional train-
• Approval. ing audience list. This is the problematic case. These are
The training outline allows any employee to quickly as- the employees who must be trained to the new or revised
certain critical dimensions of training associated with a par- SOP, based on the finalized training module, before they
ticular SOP, including the behavioral objectives of the train- are assigned to touch the drug product.
ing, the training module’s fit in the larger curriculum, the The instructional designer will then attach the training
delivery method, assessment materials, and of course the outline and the associated (provisional) training audience
training audience. Figure 2 displays a process map of the list to the procedure’s change request. When the proce-
development and utilization of a training outline, and the dure and its training outline are circulated for review and
associated training audience list. approval, the training audience list will be circulated as
well. Management of each unit impacted by the proce-
Developing And Utilizing The Training Audience List dure will review the list and recommend limiting it or
When a performance gap or training gap is identified, man- expanding it, based on their direct responsibility for the
agement must decide on the appropriate corrective action task assignments of the listed employees.
and preventive action to respond to the gap. The following The instructional designer will then take those rec-
are two possibilities: ommendations into account as the procedure, training
• It involves a life cycle document or documents outline, and training audience list are reviewed and ap-
• It involves non-lifecycle or GMP regulatory training.(30) proved. Moreover, management in the impacted units are
In either case, the associated training will require the de- alerted for the approval and implementation dates of the
velopment or revision of a training outline. The instruc- SOP, and can accordingly schedule impacted personnel
tional designer (or originator of the procedure) will ask, for necessary training on the finalized module.
“Does a training outline exist?” If one already exists, the After the new or revised procedure has been approved,
training outline will be reviewed and revised as necessary. the training window opens, within which the impacted
If not, one will be prepared. employees can be trained to the SOP before the procedure
The instructional designer will review the following five goes into effect. It is critical that the training audience be
points: defined before that window opens, hence before the SOP
• Does the SOP or other document contain background is approved, so that all training on the finalized module
history or perspective of the process that would aid in will be completed before the implementation date.31 At
the training? this point, the other proposals noted above, especially
• Does the SOP or other document cover all related pro- the use of a task assignment procedure and various tech-
cesses? nological solutions such as a validated training tracking

62 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

Figure 2
Process map of the development and Training gap
recognized
utilization of the training outline.

Yes SOP-based No
training?

Is this Is this new


a new life-cycle Existing non-life-cycle Existing
document? GMP regulatory
training?

New New

No Does a training No Does a training


outline exist? outline exist?

Yes Yes

Prepare training Generate target Review training Prepare training Review training
outline audience list outline and revise outline outline and revise
as necessary as necessary

Attach training
outline and target
audience list to
change request

Route training
outline and target
audience list with
SOP for review

Review and revise


as necessary

Submit training
Tech docs outline to
Originator, dept,
forwards to GMP originator, dept,
and QA approval
training unit and QA for
approval

File training
outline in GMP
training unit

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 63
Gordon Welty, Ph.D.

system, will provide further assurance that all requisite before being assigned to touch the drug product. GXP
training has been completed. Thus, the risk of untrained
employees being assigned to touch the regulated product
will be minimized. ACKNOWLEDGMENT
The author would like to thank his colleague A.W. for
helpful comments.
CONCLUSION
This paper first reviewed the scope and impact of FDA
regulations of pharmaceutical manufacturing in gen- ENDNOTES
1. This process has been discussed (in thematic order) in Gordon Welty,
eral, and of training in particular, and found them to
“Framework for Continuous Improvement,” Journal of GXP Compliance, Vol.
be comprehensive. Any person who touches the drug 13, No. 1, January 2009, pp. 79-89; “The Role of Critical Review in the
product, or who supervises that person, falls within the Revision of Procedures,” Journal of GXP Compliance, Vol. 12, No. 5 (Autumn
2008), pp. 77-89; “The ‘Design’ Phase of the ADDIE Model,” Journal of
scope of the regulations. These regulations impact on GXP Compliance, Vol. 11, No 4, July 2007, pp. 40-52; “Developing Assess-
these persons via written SOPs that provide comprehen- ments of Trainee Proficiency,” Journal of GXP Compliance, Vol. 12, No.1,
October 2007, pp. 64-73; “Strategy and Tactics for Program Implementa-
sive guidance for drug manufacturing. These persons tion,” Journal of GXP Compliance, Vol. 12, No. 2, January 2008, pp. 12-19;
must be trained on these procedures insofar as they re- “Strategy and Tactics of Training Recordkeeping,” Journal of GXP Compliance,
late to the employee’s functions prior to their being as- Vol. 12, No. 3, April 2008, pp. 42-52; and “Formative Evaluation in the
ADDIE Model,” Journal of GXP Compliance, Vol. 12, No. 4, July 2008, pp.
signed to touch the drug product; hence the importance 66-73. On the distinction between formative and summative evaluations,
of ensuring that the final implementation of the training see Michael Scriven “The Methodology of Evaluation,” in Peter Taylor and
Doris Cowley (eds) Readings in Curriculum Evaluation, Dubuque: IA: Wm. C.
module includes all these employees. Brown (1972), pp. 28-48, esp. pp. 29-33.
Next we considered several organizational responses 2. See Carol Watson and Sanford Temkin, “Just-In-Time Teaching: Balancing
the Competing Demands of Corporate America and Academe in the Deliv-
to the need to ensure employees are trained before be- ery of Management Education,” Journal of Management Education, Vol. 24,
ing assigned to touch the drug product. One took the No. 6, December 2000, pp. 763-778; Michael Jones, “Just-in-time Training,”
form of trying to reduce the training burden. A sec- Advances in Developing Human Resources, Vol. 3, No. 4, 2001, pp. 480-487;
and Bob Mosher, “E-Reference: The Real Just-in-Time Training,” Chief Learn-
ond took the form of a procedure requiring that the su- ing Officer Magazine, Vol. 4, No. 11, November 2005.
pervisor ensure all necessary training and qualification 3. As an example of the failure to meet the requirement of written procedures,
consider the FDA’s Warning Letter to Greer Laboratories, Inc. dated 24 June
requirements in the employee curricula are completed 2005: “Your firm failed to establish written procedures applicable to the
and documented prior to assigning an employee to a function of the quality control unit;” available at www.fda.gov/foi/warn-
ing_letters/archive/g5395d.pdf.
task. The third took the form of proposing a techno- 4. As an example of the failure to follow these written procedures, see the
logical solution. There are several problems with these FDA’s Warning Letter to Intermax Pharmaceuticals, Inc., dated 13 May 2003:
approaches, especially the failure to provide the super- “Although your firm has a written procedure for training; it was found that
these procedures are not followed;” available at www.fda.gov/foi/warn-
visor with timely and necessary information about the ing_letters/archive/g6159d.pdf.
accuracy and currency of the employee curricula, and 5. For biopharm personnel, 21 CFR 600.10; for non-clinical lab personnel,
21 CFR 58.29; for medical device personnel, 21 CFR 820.25; for human
the revision status of the SOPs. tissue recovery personnel, 21 CFR 1271.170. For further itemization of
Finally, an alternative response was presented where- the content of training, see FDA, Guidance for Industry: Sterile Drug Products
Produced by Aseptic Processing, Rockville, MD: CDER (2004), page 13;
by the training outline, a controlled document includ- “Fundamental training topics should include aseptic technique, cleanroom
ing a training audience list, is employed by the origi- behavior, microbiology, hygiene, gowning, patient safety hazards posed by
nator of a new or revised procedure to communicate a nonsterile drug product, and the specific written procedures covering
aseptic manufacturing area operations.” See also FDA, Guidance for Industry:
with each impacted functional area to determine which Quality Systems Approach to Pharmaceutical CGMP Regulations, Rockville,
employees require training. Those employees’ curricula MD: CDER (2006), page 13; “Typical quality systems training should ad-
dress the policies, processes, procedures, and written instructions related to
are revised to correspond to the new or revised proce- operational activities, the product/service, the quality system, and the de-
dure, and supervision is alerted to the opening of the sired work culture (e.g., team building, communication, change, behavior).”
6. Available at www.fda.gov/foi/warning.htm.
training window before the changes are effective, ensur- 7. Cited in “Production Procedure, QC Unit Citations Top FDA-483 List,” Gold
ing the employees are trained on the finalized module Sheet, Vol. 38, No. 5 (May 2004), pp. 3-4. For FDA inspections conducted

64 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
Gordon Welty, Ph.D.

from 2001 to 2003, inadequacy of training was the seventh most cited g4683d.pdf.
observation, with 173 observations out of a total of 1933. 22. See G. Welty “The Role of Critical Review in the Revision of Procedures,”
8. See the FDA Warning Letter dated 24 June 2005 to Greer Laboratories, Inc., Journal of GXP Compliance, op. cit.
available at www.fda.gov/foi/warning_letters/archive/g5395d.pdf. 23. This is obvious for skill training; as Michael Swartz and Ira Krull, “Train-
9. John Levchuk, “Training for GMPs—A Commentary,” Presented at the ing and Compliance,” LCGC North America [Liquid Chromatography—Gas
Pharmaceutical Manufacturers Association program, Training for the 90s Chromatography], Vol. 22, No. 9 (2004), pp. 906-912, esp. p. 906, have
(Arlington, VA: Sept. 1990). expressed it for training in CGMP regulations: “It is of little value to train or
10. See David Gallup, et al, “Selecting a Training Documentation/Recordkeep- educate an employee on all of the regulations if there is no impact on the
ing System in a Pharmaceutical Manufacturing Environment,” PDA Journal job that person fulfills every day.”
of Pharmaceutical Science and Technology, Vol. 57, No. 1, 2003, pp. 49-55, 24. See Ed Cohen, “Learning Management Systems Mitigate Risk,” Talent Man-
esp. pp. 49-50 for an insightful discussion of FDA requirements for training agement, Vol. 4, No. 12, December 2008, pp. 34-35.
documentation; also Vivian Bringslimark, “If Training Is So Easy, Why Isn’t 25. It is important that the functionality of “electronic signatures” not result in
Everyone in Compliance?” Biopharm International, Vol. 17, No. 1, January costly overbuilding of the training tracking system; see Tammala Woodrum,
2004, pp. 46-53, esp. pp. 51-52. “21 CFR Part 11: The role of predicate regulations and associated internal
11. See James Vesper, “Performance: The Goal of Training—or Why Training policies,” Drug Information Journal, Vol. 37, No 2, (2003), pp. 159-164,
Is Not Always the Answer,” BioPharm [Eugene, OR], Vol. 14, Part 2, 2001, also G. Welty, “Strategy and Tactics of Training Recordkeeping,” Journal of
pp. 44-46, esp. p. 44. Also Tony Warchut, “Retraining—Correctly Ap- GXP Compliance, op. cit, pp. 44-46, and G. Welty, “Training Documentation
plied to CAPA,” Presented at the GMP TEA Meeting, Boehringer Ingelheim, and Part 11 Compliance,” Conference Proceedings, SALT 2008 Interactive
Ridgefield, CT (20 Mar 2009), where retraining as a default CAPA is called a Technologies Conference, Society for Applied Learning Technology: Arlington,
“FDA Red Flag Warning.” VA (20 Aug 2008).
12. Available at www.fda.gov/ora/frequent/letters/ARC_20060727_ADLetter.pdf. 26. See Cohen, “Learning Management Systems Mitigate Risk,” op.cit., p. 35.
See also Nicole Fuller “FDA Fines City Red Cross in Training Irregularity,” 27. John DiLollo, “The Use of SOPs in a Pharmaceutical Manufacturing Envi-
Baltimore Sun, August 2, 2006. ronment,” Journal of cGMP Compliance, Vol. 4, No. 3 (2000), pp. 33-35.
13. Robert Merton, Social Theory and Social Structure, Glencoe, IL: Free Press, 28. As Katherine Beauchemin et al., “‘Read and Understand’ vs ‘A Competency-
1957; also John Bohte and Kenneth Meier, “Goal Displacement: Assessing based Approach’ to Designing, Evaluating, and Validating SOP Training,”
the Motivation for Organizational Cheating,” Public Administration Review, PDA Journal of Pharmaceutical Science and Technology, Vol. 55, No.1 (2001),
Vol. 60, No. 2 (March 2000), 173-182. pp. 10-15, esp. p. 11 have accurately put it, “Clearly the ‘read and under-
14. Philip Lindemann, “Maintaining FDA Compliance in Today’s Pharmaceutical stand’ method does not meet the criteria set out for validity and reliability;”
Manufacturing Facilities,” presented at the PharmTech Annual Event, Somer- see also Bringslimark, op. cit., p 46.
set, NJ: 13 June 2006, p. 15. 29. See also the discussion in Gordon Welty, “The ‘Design’ Phase of the ADDIE
15. As Levchuk, op. cit. has commented, however, “usually, available informa- Model,” Journal of GXP Compliance, op. cit, esp. pp. 49-51.
tion is inadequate to establish a specific reason beyond failure to have a 30. See Welty, “The ‘Design’ Phase of the ADDIE Model,” op. cit., esp. pp. 42-44.
training program, failure to follow the written training program, or failure to 31. James Vesper, “Defining your GMP Training Program with a Training Proce-
ensure that personnel received training.” dure,” Biopharm [Eugene, OR], Vol. 13, Part 11 (2000), pp. 28-32, esp. p. 29.
16. Levchuk, op. cit.
17. See also Vesper, “Performance: The Goal of Training,” op. cit., p. 46.
18. The very real, but very different, problem of poorly written and overlapping
SOPs has been discussed under the heading “Consolidation of SOPs” in G.
Welty, “The ‘Design’ Phase of the ADDIE Model,” op. cit, pp. 44-45.
19. See G. Welty, “The Role of Critical Review in the Revision of Procedures,” Article Acronym Listing
op. cit., pp. 79-80.
20. See www.fda.gov/ora/frequent/letters/ARC_20060727_ADLetter.pdf. CAPA Corrective Action and Preventive Action
21. Assuming for just a moment that each employee’s curriculum is correct CGMPs Current Good Manufacturing Practice
and current, this proposed approach presupposes that recourse to a notice
FDA US Food and Drug Administration
of event is an adequate organizational response for supervisory error.
Regulators typically find this unacceptable, because recourse to a NOE also JITT Just-In-Time Training
requires a list of immediate and specific corrective actions that will be taken. LMS Learning Management System
As an example of the failure to meet this requirement for NOEs, consider NOE Notice of Event
the FDA’s Warning Letter to Pharmaceutical Formulations, Inc. dated 05 May
SOPs Standard Operating Procedures
2004: “Process failures resulting in the rejection of substantial quantities of
drug products were not investigated and there is no documentation to show SISPQ Safety, Identity, Strength, Purity, and Qualit
any corrective actions;” available at www.fda.gov/foi/warning_letters/archive/

Originally published in the Spring 2009 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 65
David E. Jones — BONUS ARTICLE

Suggestions for Better Training Reports

Database administrators will likely agree that the me-


Upgrade your training reports. Consider ways chanics of maintaining training data is relatively straight-
to make your electronic data report writer work forward once the fields within a database record have been
for you. defined. The specifics of a training event are entered into
a training record. Employee identification, typically name
and employee number, is vital information. Identification is
INTRODUCTION coupled with coding for the training, i.e., the date, the SOP
Training and the documentation of training are important number and revision, and the department or functional area.
tasks in Food and Drug Administration (FDA)-regulated The more challenging task is assembling these hundreds or
firms. Firms with relatively few employees can and do thousands of “single events” into a meaningful and accu-
function satisfactorily with paper-based training records. rate report. The training report may be printed to provide
However, as the number of employees increase and the “written documentation” that listed employees are qualified
number of Standard Operating Procedures (SOPs) grows, to perform the tasks identified. These reports, which typi-
many firms rely on some form of electronic database man- cally contain a list of personnel, can be easily understood
agement software to capture and report the training of when properly titled and their references to relevant train-
personnel. The on-going task of capturing data and then ing curricula are also cited and/or provided. What may be
generating meaningful and timely reports can be daunting adequate “titling” for internal personnel may be confusing
and time-consuming. This article directs attention to the to an outside auditor or an FDA investigator. Each training
output of such database management programs-the train- report should be self-explanatory.
ing report-and offers several suggestions for improving the
value of such reports.
From the Box to the Report
Training and Documentation: Training and the accurate capture of training records is an
FDA Requirements on-going and time-consuming component of the training
The requirement to provide appropriate and adequate effort of any firm. The process should be appropriately con-
training to personnel who manufacture, package, test trolled, that is, there should be dedicated personnel and
and hold drug products is a clearly defined current Good restricted access to the records and report generation. Re-
Manufacturing Practice (cGMP) requirement.1 There is cords should be reviewed-as a separate step-to ensure that
an implicit expectation and regulatory requirement that data have been captured accurately. Many firms use training
written evidence be available (or retrievable) for the forms that are “input friendly” to speed the keying of data
training provided to personnel. This evidence, which is into the firm’s database management program. The original
the output of such software, is often collectively referred source documents should be archived for an appropriate
to as “training records.” This article invites users and cus- period-based on a firm’s defined retention schedule-to an-
tomers of database management programs to consider swer questions that might be raised about the accuracy of
what reports may be needed and how they should be the computer records.
titled before the selected software is validated. Many firms opt for configurable off-the-shelf (COTS)

66 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
David E. Jones — BONUS ARTICLE

software that is specifically designed for training records. Figure 1


Most of these software programs offer validation assistance Report Title and Descriptive Information Should Be Improved
in the form of test scripts that are incorporated into a ge-
neric protocol. The software is configured by qualified per-
sonnel to meet a particular firm’s requirements, which are
based on functional organization and the training curricula
ABC Laboratories, Inc.
of various personnel – to name a few. Validation is then Training Record - Packaging Department
typically focused on the faithful reproduction of the records Date: 22-Nov-2006 Time: 11:48 AM
into reports created or configured to meet a particular firm’s
needs. Page 1
In the all-too-typical rush to get software validated so it
can be used, user requirements may not be thoroughly ex-
plored and incorporated into the design of reports. Even
when company personnel have accurately identified their
needs, an outsider-such as an auditor-may have slightly
different needs. This outside perspective can improve the
“stand alone” quality of the various reports. Figure 2
Report Title and Descriptive Information Is Better

Suggestions for Improving Training Reports


Here are suggestions that the author believes to be impor- ABC Laboratories, Inc.
tant components of computer-generated training record re-
Training Record - Blister Packaging Line #2
ports.
1.Before designing or configuring the reports, consid- Date: 22-Nov-2006 Time: 11:48 AM
er the capabilities of the database management pro- Page 1 of 3
gram. Should a report be sorted alphabetically by last
name? Should a report be sorted by employee number?
Should the report display both employee name and
employee number? [Imagine the challenge of working
with a training report listing 200 individual names that
are not sorted alphabetically.]
2. A lthough the curriculum for any employee will typical- the contents of the report. The title should appear on
ly include both general plant training and task-specific the first page of the report-preferably in a font and size
training, will it be easier to generate two reports? Or, that is easily decipherable -and be included on each
will all training be incorporated into a single report? subsequent page along with the page number. The
3.A classic task for an auditor is to verify that each per- page numbering should be in classic CGMP style of
son who has signed a batch record for a particular part “Page n of nn pages.” Some firms also include a report
of the manufacturing or packaging-or quality testing- number as a further refinement of the naming process.
has been trained in the relevant SOPs. A departmental For instance, “Packaging Department” is descriptive
training report can be a particularly useful report for (See Figure 1), but “Blister Packaging-Line #2” is even
this function versus having to verify that each of thirty more descriptive (See Figure 2). The goal in titling re-
employees have been trained on the latest version of ports is to clearly define the personnel who are quali-
each of the relevant SOPs. fied to perform a task and/or use specific equipment.
4. E ach report should have a title that is descriptive of (Although “Packaging Department” is selected for the

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 67
Figure 3 example; the specificity of titles applies to all depart-
Report Title and Descriptive Information Is Clear ments or functional areas.) See Figure 3.
The reader now knows the list contains information on
personnel who are qualified to perform any task related to
Blister Packaging Line #2 and the curriculum on which per-
ABC Laboratories, Inc. sonnel have been trained. The “training valid” date shows
Training Record - Fully Qualified List
the period during which the listed personnel were trained
for Blister Packaging Line #2
Report #Pkg-FullQ on all applicable SOPs.
5. When a functional area training report is created, it
Ref: Curriculum Pkg 22.1 …. should refer to a readily available list of Standard Op-
Training Valid: Oct 15-Nov 20, 2006 erating Procedures (SOPs) and revision numbering of
the SOPs. If training records are grouped according to
Report Print Date: 22-Nov-2006 a training module as opposed to a functional group,
Time: 11:48 AM the report should clearly state this. See Figure 4.
Page 1 of 3 6 Further, and related to #4 above, a subtitle should indi-
cate the period for which the training is valid. For ex-
ample, “Blister Packaging #2, Fully Qualified Personnel
(those who can perform any task in the department) for
November 2006 (if that is the valid period). The header
or footer (or equivalent) should indicate when the par-

Figure 4
Example Training Report by Functional Area with SOPs and Revision Numbering

SOP Number Revision Date of Revision Title


P-101 02 07-01-06 Verification of Packaging Order
P-102 03 03-12-06 Label and Packaging Material Reconciliation
P-103 02 02-10-06 Cleaning of Acme Blister Machine 2022 and
Blister Line #2
P-104 03 04-12-06 Blister Packaging batch record documentation
practices
P-105 02 09-18-05 Zipco Label reader set up and checklist
P-106 02 08-12-06 Acme Blister Machine 2022 Setup and checklist
P-107 03 04-12-06 Uniform and dress code for Blister Packaging
Line #2
P-108 02 05-12-06 Environmental controls and checklist for Blister
Packaging Line #2

Others Various List is for example and will likely be much


longer for specific firm requirements

68 SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model
ticular report was printed. CONCLUSION
7. A common industry practice is to have operators both The design of reports that document the training of person-
sign and initial the batch record as they begin work on nel is a task performed by database management admin-
the batch. This can be very useful in identifying a par- istrators and/or other qualified individuals. The relatively
ticular operator especially when initials are common- simple “tweaking” of the title of reports along with appro-
ly used to verify various tasks throughout the batch priate pagination and reference to training curricula can im-
record. It is useful to “standardize” the name shown prove the clarity-and thus the value-of such reports to the
on the batch record and on training records. Here are user. GXP
three examples of confusing, non-standardized name
use: (1) An operator is entered in the training records REFERENCE
1. Code of Federal Regulations, Part 21. “…§ 211.25 Personnel qualifications.
as Robert K. Smith; the operator signs the batch record
(a) Each person engaged in the manufacture, processing, packing, or holding of
register as “Robbie” Smith. (2) Two operators, Thomas a drug product shall have education, training, and experience, or any combina-
Baker and Terrance Blackwell each initial their entries tion thereof, to enable that person to perform the assigned functions. Training
shall be in the particular operations that the employee performs and in current
as “TB.” (3) An operator is entered into the training good manufacturing practice (including the current good manufacturing practice
records as Maria Lopez, but signs the batch record as regulations in this chapter and written procedures required by these regulations)
as they relate to the employee’s functions. Training in current good manufacturing
Maria L. Rodriguez. practice shall be conducted by qualified individuals on a continuing basis and with
8. T
 raining records change over time as personnel are sufficient frequency to assure that employees remain familiar with CGMP require-
added and deleted and as training lapses and training ments applicable to them….”

is added. Imagine the difficulty of reviewing the status


Article Acronym Listing
of training when the latest generated record is more
than 90-days old and new revisions of SOPs have been COTS Configurable Off-The-Shelf (software)
introduced during this period. The process of updat- CGMP Current Good Manufacturing Practice
ing a record should be relatively easy and done on a FDA Food and Drug Administration (U.S.)
timely basis. SOP Standard Operating Procedure

Originally published in the January 2007 issue of Journal of GXP Compliance

SPECIAL EDITION: The ADDIE Model, and Instructional Generic Design Model 69