Marceau, R., Jackson, C., Diedrich, F., & Riccio, G. (2010). Development of General Measures for Students.

In: Riccio, G., Diedrich, F., &

Cortes, M. (Eds.). An Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based
Requirements (Chapter 12). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in
text.]

207

Marceau et al.
Chapter 12. Development of General Measures for Students
Ryan Marceau, Cullen Jackson, & Fred Diedrich
Aptima, Inc.
Gary E. Riccio
The Wexford Group International
12.1 Intent

In March of 2008, the Asymmetric Warfare Group (AWG) initiated a scientific and technical
effort to help define and develop Outcomes-Based Training and Education (OBTE) (Darwin,
2008; Cornell-dEchert, 2009). With respect to measures development, this initial effort focused
primarily on instructor application of OBTE within field training (see chapter 2). To further this
research, the AWG initiated a follow-on effort aimed at defining observable performance
measures for OBTE based learning that are focused on the student in either a field-based or
classroom setting. The intent was to develop measures that can be used to provide feedback to
students and their instructors about student performance in the context of important
characteristics of the learning environment (Bransford, Brown & Cocking, 2000). Such measures
provide a means for instructors, and students, to reflect on the behavior of students and what they
mean for instruction that realizes (or fails to realize) OBTE. Hence, they are intended to support
formative feedback to students and instructors that enables reflection on student behavior, the
actions that instructors can take to influence those behavior, and the meaning of those behavior in
terms of student progress toward key learning objectives and developmental outcomes in the
context of OBTE (see chapter 3; Pellegrino, Chudowsky, & Glaser, 2001).
12.2 Performance Measure Development Process
We employed a method that focuses on behavior that relates to the acquisition of knowledge,
skills, and competencies in the context of OBTE. The process for developing performance
measures grounds assessment in the behavior of experts and individuals who have attained
proficiency. This helps ensure that behavior-based measures are meaningful and pertain to
constructs for which data can be reliably collected. More specifically, our methodology for
measure development combined subject matter expertise with established psychometric practices
to produce measures of observable behavior. Our approach was based loosely on Aptimas
COMPASS design process (e.g., MacMillan et al., in press). In this case, we employed a fourphase approach to operationalize the principles and practices of OBTE in terms of student
behavior that can be observed and measured. This approach relied heavily on collaborative
discussions with subject matter experts that were conducted in four sequential steps.
12.2.1 Phase One: Define Performance Indicators (PI)
The first phase was to conduct a workshop with the experts to define high level performance
indicators for students in OBTE. This first workshop was conducted in a group setting with the
progenitors of OBTE from the AWG. The first workshop involved an intensive discussion of the
principles and practices of OBTE and how they are manifested in the behavior of learners. More
specifically, in this workshop, subject-matter experts identified behavior on which the
performance measures would be based. These performance indicators (PI) are types of behavior
that allow an observer to rate the quality of a Soldiers performance in individual or collective
tasks. For this workshop, it is critical to identify behavior that can be observed rather than
Asymmetric Warfare Group

Development of Student Measures

208

inferred from little or no observable evidence. The PI provide a solid basis on which to develop
measures that are not sensitive to individual rater differences and therefore can be rated reliably
by multiple expert observers. Identifying the PI is a key milestone from which to develop specific
measures of performance within a learning environment.
12.2.2 Phase Two: Translate PI into performance measures
While some PI are readily translated into performance measures, more detailed information is
generally needed to create behaviorally-anchored performance measures from the PI identified in
the first workshop. To develop these behavioral anchors, it is necessary to identify specific
behavior that is relatively effective and behavior that is relatively ineffective for each PI.
Therefore, a second workshop involved one-on-one interviews with subject matter experts to
deliberate about behavior that could illustrate expert, average, and novice performance for each of
the PI.
The second workshop consisted of individual interviews with some of the AWG OBTE
progenitors. The interviews were focused on identifying the novice, average, and expert behavior
for each PI. The topic of conversation focused on the actions of a student that demonstrate
development of intangibles (e.g., confidence, initiative, and accountability), as well as effective
learning. The information gathered in this workshop served to inform the creation of behavioral
anchors appropriate for the measures.
12.2.3 Phase Three: Measure refinement
The goal of the third workshop was a detailed review and modification of a set of draft
performance measures. Subject-matter experts were asked to guide the refinement of measures to
reduce ambiguity and promote inter-rater agreement. This phase relied on expert input to review
and revise all of the performance measures according to the following criteria:

Relevance: Does the measure assess an aspect of performance that is important for
mission readiness?
Observability: Does the measure assess a behavior that is truly observable?
Question wording: Does the measure make sense to other subject matter experts?
Scale type: Is the scale used appropriate for differentiating behavior?
Scale wording: Do the behavioral anchors make sense to other subject matter experts?

The third workshop presented the experts from the first workshop with a draft set behaviorally
anchored performance measures for each PI. Performance measures were reviewed to ensure that
they were seen as meaningful, relevant, and reflective of the community in which training will be
embedded. Upon completion of the third workshop, we produced a comprehensive set of
performance measures for the assessment of student behavior with respect to the principles and
practices of OBTE.
12.2.4 Phase Four: Retranslation of Measures
During this final phase, subject matter experts were asked to read each measure and come up with
key words that describe what the particular question aims to measure. This workshop ensured that
the final measures are consistent with the PI developed in Phase one. If a measure was assigned
key words that were not relevant to its associated PI, that measure was regrouped under a
different PI or dropped. In so doing, we sought to understand and document how individual

Asymmetric Warfare Group

209

Marceau et al.

measures lined up with and support the principles of OBTE. In this case, this workshop was
again conducted in a group setting with some of the progenitors of OBTE from the AWG.
12.3 Product of Measure Development
The product of our measure development effort with the OBTE instructional designers yielded a
set of 31 observable measures that are focused on learner behavior. Learner behavior is not,
however, independent of the behavior of the instructor. In most cases, learner behavior is viewed
as being couple to the environment created by an instructor. Thus, we believe that instructors
should utilize these measures as an aide to establish and maintain an effective learning
environment. Overall, these measures define OBTE at a level sufficient for rigorous verification
of student learning in OBTE environments and, thus, they are useful both for program evaluation
and formative feedback to instructors and students.
More specifically, the observable performance measures cluster around four performance
indicators:
1. Learner Perception of the Teacher and Course
2. Learner Engagement in the Course
3. Learner Relationship with the Teacher
4. Learner Results and Understanding
The following sections provide detailed descriptions, sample measures, and example applications
of each PI presented by subject matter experts during discussions. A complete set of measures is
available in Appendix B.
12.3.1 Learner Perception of the Instructor and Course
The learners perception of the instructor and the course plays a critical role in the value that the
individual places on the experience. It is important that instructors exhibit a reasonable level of
expertise in the subject matter, but they do not need to know everything. Rather, instructors need
to know how best to leverage their expertise and how to find additional resources as needed for
addressing new challenges. It is critical that the learner has trust and confidence in the instructor
to be in engaged in learning and to be prepared for ongoing self-development (Darwin, 2008).
This trust and confidence is not born out of an assumption by the learner that the instructor knows
all there is to know about a topic. Rather, we believe it comes from a perception that the
instructor truly cares about the student and wants the student to understand the material. The
instructor only needs to know enough about the domain to serve as an effective agent who can set
the conditions to facilitate and guide the students learning.
Students should perceive their instructors as mentors. This perception comes from the
reinforcement of initiative and awareness in the students. Students should feel comfortable
challenging the instructors assertions, when appropriate. Students who perceive the course as
valuable and trust the instructor are more likely to take the initiative to engage the instructor
during and outside of formal periods of instruction (e.g. before class, after class, or during
breaks). This active seeking of additional interactions regarding course content with the teacher
and peers demonstrates that the student has confidence in the meaningfulness of the course and
trust in the teacher.

Asymmetric Warfare Group

210

Development of Student Measures

Example Measures
2. Do the students behave as if they have confidence in, and respect, for the teacher?

Students display negative

attitude towards teacher, do
not pay attention; comment
negatively to other
students; verbally challenge
teachers expertise in an
inappropriate way

Students quietly takes

notes, but do not fully
engage in discussion or
dialogue (enforced respect)

Students challenge
teachers assertions
appropriately; engage
teacher during breaks;
ask appropriate, thoughtprovoking questions
(nurtured respect)

3. Do the students engage with the teacher outside formal periods of instruction (e.g.
before/after class, during breaks)?

None of the students

engage with the teacher

Some of the students

engage with the teacher

Most or all of the student

engage with the teacher

Example Vignette:
A new teacher had been successfully instructing a course for a few months. His instructional
techniques resulted in conversations between himself and the students. Upon consistent requests
from students, who had been conditioned by past learning experiences, he decided to begin
handing out his teaching notes as study guides. Upon doing so, the teacher noticed a sharp decline
in student retention and test performance. He had not changed any of the content, yet the
performance dropped.
The teacher in the hypothetical vignette may have changed the instructional environment from
one of collaborative learning to one of memorization by handing out his notes. Once the students
had the notes, they did not feel motivated to take any further initiative toward understanding the
material. They already had the answers and they had been previously conditioned to memorize
and recall specific information in order to pass tests. This perceived norm was reinforced as soon
as they received the list of notes. Previously, when the cue was not present, the students behaved
differently. Through collaborative interaction with the instructor they were forced to think more
critically about the content in order to derive the key points. The instructor served as a guide in
this process of discovery (Dewey, 1915/2008; Hmelo-Silver, 2004). This concept of norms in
learning is important to note. Changes to the norm may induce a negative reaction in some
students who become frustrated when the teacher does not tell them what to know. However, it

Asymmetric Warfare Group

211

Marceau et al.

is expected that after they begin to see that the teacher truly wants them to understand the
material, the students will become more enthusiastic about the course and instruction.
12.3.2 Learner Engagement
We believe that engagement is largely the observable behavioral manifestation of motivation in
students. Behavior such as responsiveness, alertness, attentiveness, and demonstration of
initiative are all indicative of high levels of student engagement. It is critical that the instructor
fosters engagement (Gagne, 1985). The instructor should provide an environment that encourages
and inspires students to think and participate. For example, practical applications and exercises
should require the learners to discover solutions to problems. Behavior demonstrating student
engagement should be reinforced continuously in order to ensure students understand that their
actions will make a difference. High engagement can be seen in students who try to take the next
step or advance an idea further. Such behavior shows that they are interested and desire to learn
more. One key indicator that students are engaged is when they constructively challenge an
assertion made by the instructor or another student. This shows that a student is not only attentive
but also is actively trying to understand and contrast the instruction with his or her past beliefs.
This behavior can also be seen in spontaneous side conversations in which students
collaboratively reflect on an assertion or idea posed by the instructor. These discussions should be
encouraged and even anticipated at critical points during the program of instruction.
Example Measures
1. Do the students demonstrate initiative?

Students wait for prompting

before responding with needed
action or avoid action

Students responds with

action without prompting,
but do not further dialog
with additional ideas

Students correctly respond

as needed without
prompting, and extend
concepts/actions one step
further

2. Do the students ask and answer questions?

Students asks questions on

procedures and processes to
meet course requirements, or
fail to ask questions even
when not understanding

Asymmetric Warfare Group

Students ask questions for

clarification; focus on
current topic and
application

Students ask questions to

advance the idea to other
situations; apply
discussion to applications
outside of test items

Development of Student Measures

212

Example Vignette:
A new instructor in a Basic Officer Leadership Course is lecturing his students. After his
introduction, he is aware that many are not paying full attention. As he continues, he realizes that
more and more of the students are losing interest. His instruction is rather dry, and he has not had
a lot of practice teaching this particular block of instruction. Students are beginning to fidget and
their body language is indicating that they are becoming disengaged (e.g. little eye contact,
slouching, etc.).
Recognizing this, the instructor chooses to use a Tactical Decision Exercise (TDE) as a way to
bring the students back into the discussion. This particular exercise requires the students to react
to a difficult and ethically challenging problem. At the end of the exercise students are asked to
discuss the course of action they would have chosen and why. Students slowly begin to volunteer
their choices. Eventually the students are all participating and highly engaged in this controversial
issue. Students are debating their approaches and rationale for their decisions.
During this time, the instructor is listening to the conversation, but he is not participating due to
his lack of experience in the chosen scenario. He is unable to reconcile the different points of
view that have emerged in the classroom. As a result, the instructor decides to stop the
conversation after a few minutes in order to return to the PowerPoint presentation that covers the
terminal and enabling learning objectives. The students halt their debate and do not engage each
other for the rest of the day.

The instructor in the above hypothetical vignette demonstrated a high level of awareness
regarding his students level of engagement during the instruction. His choice to leverage a TDE
to increase engagement suggests that he understood learner state, and wanted the students to
understand the content and learn. However, the instructor made a critical mistake by using a TDE
that was outside of his own domain of expertise and understanding. As a result he was unable to
participate effectively in the discussion. He could have played an active role in guiding the
students through the discussion had he chosen a topic for his TDE that he understood more
thoroughly or chosen one in which he could have recruited outside expertise or pointed students
toward critical resource. The instructor could have acted as a mentor and guide in the students
pursuit of the solution. By doing so, the instructor would have created an environment in which
collaborative discussion and debate was seen as an acceptable and encouraged method of coming
to know. Instead, the instructor created the opposite effect by halting the discussion and setting
the implicit precedent that this type of discussion is not appropriate.
12.3.3 Student Relationship with Teacher
It is important that students feel they can have an effect on the direction and progression of the
course. Students should not have full control of the course, but instructors should be flexible, as
appropriate, to allow students to take ownership of their learning. Students should feel
comfortable asking questions, especially when they are feeling lost or overwhelmed. Questions
posed by students, whether they serve to advance the current idea, or are seeking clarification,
should cause a noticeable effect in the classroom. This perceived change on the learners part will
create an understanding that they have some control over the classroom environment. This
perception of control is known as an internal locus of control. Individuals with internal loci of

Asymmetric Warfare Group

213

Marceau et al.

control are more likely to exhibit high levels of motivation to learn because they more likely to
accept feedback and take action to correct performance problems (e.g., Goldstein & Ford, 2002;
Guay, Ratelle, & Chanal, 2008).
Additionally, instructors should act as mentors as the students mentally struggle toward an
understanding of the training content or problem. It is important to note that struggling, in this
sense, describes a willing drive to overcome an obstacle or problem on the behalf of the student.
This drive may be manifested as motivation and engagement in the learner. Instructors should
support students on their paths toward discovering the solution by asking key questions designed
to stimulate further thought, providing reminders of learning points when needed, and giving
specific suggestions on paths toward solutions if needed. Instructors should also encourage
students to relate training to their past experiences and ask questions when they are struggling.
Questions like What if we did it this way or What about this situation show that students
are aware and are relating training content to their past experiences and assumptions. Once
students have discovered the answer or solution, instructors should ensure that students are able
to articulate their thought processes. Understanding where they were momentarily wrong or right
will further increase their understanding.
Example Measures
1. Do the students take initiative to catch up when feeling overwhelmed or left behind?

Students do not speak up and

ask for help; do not respond to
teacher queries

Students do not speak up,

but respond to teacher
when queried

Students proactively speak

up and ask questions when
falling behind or confused

2. Do the students struggle to discover a solution and solve unique problems?

Students execute standard

procedures even when not
appropriate for problem; fail
to solve problem

Asymmetric Warfare Group

Students make some

progress on unique
problem, but fail to
effectively solve the
problem

Students adapt training to

discover an effective
solution to a unique
problem

Development of Student Measures

214

Example Vignette:
An experienced instructor posed a unique problem to his students. The students then began to
discuss and propose potential solutions to each other. The instructor, who had facilitated this
exercise several times before, monitored the discussion and participated as needed to keep
students from going too far down the wrong road. The students struggled mentally with the
problem but enjoyed the challenge and were highly engaged. As expected, the students eventually
discovered the solution through trial, error, and collaboration.
Next, the instructor facilitated a discussion to elicit explanations about how the students figured
out the answer. The students began to describe a decision-making process and method of
investigation which was consistent with the instructors expectations. However, one student, who
had come to the same solution, proposed a completely different path to discovery. The instructor,
after listening to his explanation, had to pause for a moment. He had never thought about the
problem in this particular way before. After thinking about it from the students unique
perspective, the instructor told the student that she also was correct. At this point the instructor
changed the focus of the discussion to a conversation about the two different, but equally correct
approaches.

Engaged students are continuously relating course content to past experiences. The hypothetical
vignette above describes a case in which one students past experiences allowed her to approach a
problem in a very unique way. That student felt confident and comfortable enough in his learning
environment to challenge what his instructor and peers were proposing as the one method for
arriving at the solution. The instructor, having recognized that the student was also correct,
reinforced the students initiative and validated her unique approach. The students perspective on
the problem served to enrich the discussion and shape the direction of the course. The other
students benefited from this discussion through exposure to the diversity of thought.
The instructors desired outcome for the exercise was to allow his students to build confidence in
their ability to solve difficult problems. He achieved this outcome and also leveraged the
unexpected answer to increase awareness regarding problem solving and to reinforce initiative.
12.3.4 Student Results
Student retention of factual information is important but this alone does not create knowledge.
Scoring 100% on a multiple choice test only shows that the students remembered enough
information long enough to recall it and use it on the test. It is possible that much of this inert
information will not be retained after training.
Outcomes for training should include student learning, awareness, understanding and
development. Students best demonstrate knowledge and awareness through the application of
learned information to unique problems or contexts, rather than through mere reproduction of
isolated facts on tests. Hence, instructors should provide opportunities for students to apply
information in situations that are different from those used as exemplars during instruction
(Tobias & Duffy, 2009). Through these opportunities, students will have the chance to integrate
lessons learned with their own unique knowledge to achieve individual solutions. The presence of

Asymmetric Warfare Group

215

Marceau et al.

these opportunities should have a positive effect on transfer by allowing learners to relate training
to past knowledge as well as to additional contexts, ideas, and environments.
Example Measures
1. Do the students retain and apply factual information (demonstrate a technical and contextual
understanding)?

Students do not retain

information (e.g., cannot
answer question)

Students retain information

(e.g., give textbook
answer)

Students retain and apply

information to create
knowledge (e.g., show
application to mission,
technical and contextual
understanding)

2. Do the students apply lessons learned to improve performance?

Students do not apply lessons

learned to change behavior;
continue to make same errors

Students change behavior

in accordance with lessons
learned

Students integrate lessons

learned with existing
behavioral strengths to
yield optimized individual
approach

Example Vignette:
A Private, while learning how to assemble and disassemble his weapon, remembers a slightly
different method his father had taught him when he was younger. The instructor allows the
Private to try this method because he knows it is a safe and acceptable alternative. The Private
integrates the new method with his past knowledge to come up with an alternative successful
procedure. The Private is then able to safely assemble and disassemble his weapon faster than the
other Privates.
Further along in training the Privates are learning how to assemble and disassemble another
weapon. A second Private, while trying to take a shortcut, unknowingly puts himself into a very
dangerous situation by performing steps out of order. The Drill Instructor recognizes the danger
and immediately stops the Private from injuring himself and explains what could have happened
and why it was unsafe. Recognizing his error, the Private follows the standard procedure next
time to successfully disassemble and assemble the weapon.

Asymmetric Warfare Group

216

Development of Student Measures

The hypothetical vignette above describes two situations in which a soldier acquired knowledge.
In both cases, the end result was a soldier who is able to complete the task to standard (e.g.
assemble and disassemble his weapon). The first private became more knowledgeable about his
weapon by applying past knowledge resulting in a positive result (e.g. performing the task more
efficiently). The second private experienced a negative result (e.g. put himself in an unsafe
situation) but still acquired knowledge. As a result of the experience, the second Private became
more aware of why the process exists.
Instructors should always allow students to take ownership of their learning and, with some
guidance and constraints, discover solutions to problems. Small failures can lead to success and
overall positive learning results. Conversely, too many failures can be discouraging, hurt
confidence, and have an overall negative effect on learning. It is important that instructors find an
appropriate balance between allowing students to struggle and providing guidance when
necessary.
12.3.5 Self-Report Measures
In addition to the observer-based measures described above, much information can be gained by
asking students about their impressions across numerous areas ranging from what they believe
they learned to impressions regarding instructors. Hence, although in the sections above we focus
on observable measures, our process also developed a set of self report measures as a tool for
assessing students perceptions. The measures are related to items such student perceptions of the
instructor and course, perceived value of the course, and level of perceived individual
development. These measures are useful to the instructor in determining if the desired learning
environment is being created or if in-stride adjustments are needed. Additionally, they can help
the student become more aware of the learning objectives by making the teachers goals
transparent to the learner.
Example Self-report measures:
1. Do you have trust in your teacher to develop your individual skill set?

I dont think the teacher

cares if I do well or not

I feel that the teacher is

interested in my success in
this course

I feel that the teacher is

genuinely interested in
my success in my
mission

2. Do you feel that you are a better problem solver as a result of your last practice exercise?

Not at all; no different than

before I participated in the
exercise.

Asymmetric Warfare Group

Yes, but only within the

context of this course

I feel I am a better
problem solver overall as
an individual

217

Marceau et al.

It is important to note that an instructor should utilize self-report measures at multiple times
throughout her courses. While such feedback at the end can be helpful, feedback during the
course can help guide real-time adaptations that enable the course to be more student-centered. In
addition, these measures do not necessarily need to be administered in paper survey form; they
also could be used to frame a discussion. Of course, these self-report measures can be tailored to
the learning objectives of the course or particular block of instruction to provide the instructor
with detailed information regarding student learning. The overall point of these measures,
however, is to complement observer-based measures of student behavior with self-report
measures that can help instructors triangulate on student attitudes. Multiple kinds and sources of
feedback can minimize the risk of misinterpretation and help determine the urgency of an instride modification or deviation from a plan.
12.4 Conclusion
The student measures presented in Appendix B, and summarized in this chapter, include both
self-report and observer-based items that relate to key themes such as learner perception of the
teacher and the course, learner engagement, learner relationship with the teacher, and learner
understanding. Taken together, they capture key elements of student behavior that are useful for
guiding instruction. However, not all of the observable performance measures and self-report
measures are ideally suited for every program of instruction. Instead, instructors and observers
are encouraged to refine or adapt these measures to construct their own assessment tools that are
focused on their desired outcomes and observable behavior within their training environments.
Such adaptations will be consistent with OBTE if (a) they take into account long-term results and
developmental outcomes, and (b) there is careful documentation of the changes and the reason for
them. In doing so, instructors and observers will have demonstrated both initiative and
accountability in taking ownership of quality assurance for the course.
As models for key elements of assessment, the measures described in this chapter serve essential
objectives. First, along with measures of instructors (chapter 2), they define OBTE based on a
verifiable set of expected actions and observable behavior. These measures can therefore be
useful for course designers and instructors who have the desire to adopt an outcomes-based
approach. Second, they can provide feedback to course designers and instructors based on
observable learning and behavior of students. Hence, an instructor or observer can use these
measures to gauge the level of learning taking place in a field or classroom environment. This
pre-planned feedback can guide course refinement or real-time adjustments. Finally, they can be
useful for providing feedback to students related to course-specific learning objectives and
longer-term developmental outcomes as in a progress map (Pellegrino, et al., 2001). The
measures provide observable behavior that can be used to gauge student state, not relative to other
students but relative to the students further development. Accordingly, they represent a key tool
in creation of an assessment-centered environment in which assessment is an integral part of
learning.
12.5 References
Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, experience, &
school. Washington, DC: National Academy Press.
Cornell-dEchert, B. (2009). Outcomes-based training and education: Implementation guide. Ft.
Meade, VA: Asymmetric Warfare Group.

Asymmetric Warfare Group

Development of Student Measures

218

Darwin, M. (2008). Outcomes-based training and education: fostering adaptability in full

spectrum operations (Briefing, December 2008). Ft. Meade, VA: Asymmetric Warfare
Group.
Dewey, J. (2008). The school and society, the child and the curriculum. Chicago, IL: Centennial
Publications of The University of Chicago Press. (Original work published 1915)
Gagne, R. (1985). The Conditions of Learning (4th ed.). New York: Holt, Rinehart & Winston.
Goldstein, L., & Ford, K. (2002). Training in organizations: Needs assessment, development, and
evaluation. Belmont, CA: Wadsworth Thomson Learning.
Guay, F., Ratelle, C. F., & Chanal, J. (2008). Optimal learning in optimal contexts: The role of
self-determination in education. Canadian Psychology, 49, 233240.
Hmelo-Silver, C. (2004). Problem-based learning: What and how do students learn? Educational
Psychology Review, Vol. 16(3), 235-266.
MacMillan, J., Entin, E. B., Morley, R., & Bennett, W. (in press). Measuring team performance in
complex dynamic environments: The SPOTLITE method. Military Psychology.
Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The
science and design of educational assessment. Washington, DC: National Academy Press.
Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure. New York:
Routledge

Asymmetric Warfare Group

ix

Riccio, Diedrich, & Cortes (Eds.)

TABLE OF CONTENTS
.

page
Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1
Historicity of our Research on OBTE ..........................................................................................1
The Approach and Lessons Learned from the Research..............................................................3
Documentation of the Research ...................................................................................................4
Section I. Development of Stakeholder Requirements for OBTE..............................................6
Chapter 1. Preparation for Full Spectrum Operations ......................................................................7
1.1 Requirements of Full Spectrum Operations ...........................................................................8
1.2 Outcomes-Based Training and Education (OBTE)..............................................................10
1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11
1.2.2 OBTE as a Multifaceted Instructional System .............................................................12
1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13
1.3.1 A Systems Engineering Framework for Integration and Development of OBTE ........13
1.3.2 Preparation for Validation and Verification .................................................................14
1.4 References ............................................................................................................................17
Chapter 2. Formative Measures for Instructors ..............................................................................20
2.1 Development of Formative Measures ..................................................................................20
2.1.1 The COMPASS Methodology ......................................................................................20
2.1.2 Development of Measures for OBTE ...........................................................................21
2.2 Description of Formative Measures .....................................................................................21
2.2.1 Results of the COMPASS Process................................................................................21
2.2.2 Elaboration on the Description of Measures.................................................................23
2.3 OBTE Performance Measures: Planning for Training.........................................................23
2.3.1 Define Outcomes ..........................................................................................................23
2.3.2 Create a Positive Learning Environment ......................................................................25
2.3.3 Create the Parameters of Learning................................................................................27
2.4 OBTE Performance Indicators: Training Execution............................................................28
2.4.1 Communicate the Parameters of Learning....................................................................28
2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31
2.4.4 Facilitates Learning of Concepts ..................................................................................32
2.4.5 Creates a positive learning environment.......................................................................34
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation ................................36
2.4.7 Uses scenarios to facilitate learning..............................................................................38
2.4.8 Instructors exhibit intangible attributes in own actions ................................................40
2.4.9 Hotwashes and Mini-AAR............................................................................................42
2.5 Uses of the Measures ...........................................................................................................43
2.5.1 Formative Measures for Instructors ..............................................................................44
2.5.2 Quality Assurance and Instructor Education ................................................................44
2.5.3 Continuous Improvement of Assessments....................................................................45
2.5.4 Program Evaluation and Organizational Change..........................................................46
2.6 References ............................................................................................................................46

Asymmetric Warfare Group

Table of Contents

Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50

3.1 Multifaceted Inquiry.............................................................................................................50
3.1.1 Interaction with Progenitors of OBTE..........................................................................51
3.1.2 AWG Documents on OBTE .........................................................................................52
3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52
3.1.4 Interaction with Stakeholders .......................................................................................53
3.2 Essential Characteristics of OBTE.......................................................................................53
3.2.1 The Meaning of Developmental is a Critical Difference..............................................53
3.2.2 The Definition of Outcomes is a Critical Difference....................................................56
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58
3.2.4 The Meaning of Experience is a Critical Difference ....................................................61
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62
3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65
3.3 Toward a Grounded Theory for OBTE ................................................................................66
3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68
3.4.1 Leadership and Enculturation of Soldiers.....................................................................68
3.4.2 Robust and Adaptable Plan...........................................................................................70
3.4.3 Instructors as Role Models ...........................................................................................70
3.4.4 Collaborative Identification of Outcomes and Measures .............................................71
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72
3.5.2 Task Relevance of Planned Instructional Events..........................................................72
3.5.3 Reveal Operational Relevance of Training...................................................................73
3.5.4 Incorporate Stress into Instructional Events .................................................................73
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74
3.6.1 Collaborative Reflection and Problem Solving ............................................................75
3.6.2 Communication.............................................................................................................75
3.6.3 Nature and Extent of Guidance.....................................................................................76
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76
3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77
3.7.1 Soldier Motivation and Development of Intangibles....................................................77
3.7.2 Plan for Development of the Individual .......................................................................78
3.7.3 Get Students to Take Ownership ..................................................................................78
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79
3.8 References ............................................................................................................................79
Chapter 4. Grounded Theory for Values-Based Training & Education .........................................86
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86
4.1.1 Fundamental Units of Analysis.....................................................................................87
4.1.2 Nested Time Scales and Adaptability ...........................................................................88
4.1.3 Adaptability and Ambiguity .........................................................................................90
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92
4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93
4.2.1 Ecological Psychology..................................................................................................93
4.2.2 Self-Efficacy Theory.....................................................................................................97
4.2.3 Positive psychology ......................................................................................................98
4.3 A More Integrated Scientific Infrastructure .......................................................................101
Asymmetric Warfare Group

xi

Riccio, Diedrich, & Cortes (Eds.)

4.3.1 Self Determination Theory .........................................................................................101
4.3.2 Situated Learning Theory ...........................................................................................103
4.3.3 Existential Psychology................................................................................................105
4.4 Building on the Scientific Infrastructure for OBTE...........................................................109
4.4.1 Triadic Frameworks ....................................................................................................109
4.4.2 Further Development ..................................................................................................112
4.5 References ..........................................................................................................................112

Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................118
5.1.1 An Alternative to Individual versus Collective ..........................................................118
5.1.2 Cognition and Reality .................................................................................................119
5.2 Conscious Experience and the Dynamics of Thinking ......................................................122
5.3 Emotion, Information, and Engagement ............................................................................125
5.3.1 Ecological Perspective on Emotion ............................................................................125
5.3.2 Emotion as Engagement .............................................................................................126
5.3.3 Implications for Training and Education ....................................................................129
5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129
5.4.1 Toward a More Integrated Theory..............................................................................129
5.4.2 Emotion and Decision-Making ...................................................................................130
5.4.3 Emotion and Nested Time Scales ...............................................................................131
5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132
5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133
5.5 Beyond Science ..................................................................................................................134
5.5.1 Existentialism..............................................................................................................134
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135
5.6 References ..........................................................................................................................137
Section II. Verification and Validation of OBTE as a Service System ..................................142
Chapter 6. Initial Impressions of Participation in CATC .............................................................143
6.1 Methods..............................................................................................................................143
6.1.1 Participants..................................................................................................................143
6.1.2 Procedure ....................................................................................................................143
6.1.3 Analyses......................................................................................................................144
6.2 Results ................................................................................................................................144
6.3 Implications for Service System Development: Peer Review ...........................................146
6.4 References ..........................................................................................................................147
Chapter 7. Local Development of Measures of Effectiveness .....................................................149
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training? .................149
7.2 Measure Development Process ..........................................................................................150
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151
7.4 Implications........................................................................................................................156
7.5 Conclusions ........................................................................................................................158
7.6 References ..........................................................................................................................159

Asymmetric Warfare Group

Table of Contents

xii

Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160

8.1 Methods..............................................................................................................................160
8.1.1 Participants..................................................................................................................160
8.1.2 Procedure ....................................................................................................................160
8.1.3 Analyses......................................................................................................................161
8.2 Results ................................................................................................................................163
8.2.1 Behavior of DS ...........................................................................................................163
8.2.2 Behavior and Performance of Privates .......................................................................165
8.2.3 Patterns of Communication ........................................................................................168
8.2.4 Potential Influence of Instructor Behavior on Performance of Privates .....................170
8.3 Implications for Service System Development..................................................................171
8.3.1 Verification of OBTE .................................................................................................171
8.3.2 Validation of OBTE....................................................................................................172
8.4 References ..........................................................................................................................173
Chapter 9. Impact on Rifle Marksmanship Training....................................................................174
9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................174
9.1.1 Method ........................................................................................................................174
9.1.2 Assessment..................................................................................................................175
9.1.3 Results An Overview ...............................................................................................177
9.1.4 Evidence for Influence of OBTE ................................................................................178
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180
9.1.6 Behavior of Privates....................................................................................................182
9.1.7 Patterns of Communication ........................................................................................186
9.1.8 Summary .....................................................................................................................186
9.2 Attitudes Toward an OBTE in Basic Training...................................................................187
9.2.1 Method ........................................................................................................................187
9.2.2 Results.........................................................................................................................187
9.4 References ..........................................................................................................................191
Chapter 10. Influence of CATC in an Operational Setting ..........................................................192
10.1 Methods............................................................................................................................192
10.1.1 Participants................................................................................................................192
10.1.2 Procedure ..................................................................................................................192
10.1.3 Analyses....................................................................................................................193
10.2 Results ..............................................................................................................................193
10.2.1 Downstream Impact on Marksmanship ....................................................................193
10.2.2 Downstream Impact on Training in the Units ..........................................................194
10.2.3 Downstream Impact on Self Efficacy .......................................................................195
10.3 Implications for Service System Development: Validation.............................................196
10.4 References ........................................................................................................................197
Chapter 11. Implications for Service System Development.........................................................198
11.1 Lessons Learned about Transfer of OBTE.......................................................................198
11.2 Implications for Service System Development................................................................199
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199
11.2.2 Further Development of OBTE as a Service System ...............................................199
11.2.3 Further Verification and Validation of OBTE ..........................................................201
11.3 References ........................................................................................................................203

Asymmetric Warfare Group

xiii

Riccio, Diedrich, & Cortes (Eds.)

Section III. Further Development of OBTE as a Service System ..........................................206

Chapter 12. Development of General Measures for Students ......................................................207
12.1 Intent ................................................................................................................................207
12.2 Performance Measure Development Process...................................................................207
12.2.1 Phase One: Define Performance Indicators (PI).......................................................207
12.2.2 Phase Two: Translate PI into performance measures ...............................................208
12.2.3 Phase Three: Measure refinement.............................................................................208
12.2.4 Phase Four: Retranslation of Measures ....................................................................208
12.3 Product of Measure Development....................................................................................209
12.3.1 Learner Perception of the Instructor and Course ......................................................209
12.3.2 Learner Engagement .................................................................................................211
12.3.3 Student Relationship with Teacher ...........................................................................212
12.3.4 Student Results .........................................................................................................214
12.3.5 Self-Report Measures ...............................................................................................216
12.4 Conclusion........................................................................................................................217
12.5 References ........................................................................................................................217
Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219
13.1 Intent ................................................................................................................................219
13.2 Observing OBTE in the Classroom Environment............................................................219
13.2.1. Participants...............................................................................................................219
13.2.2. Procedure .................................................................................................................220
13.2.3. Measures ..................................................................................................................220
13.3 Utility of OBTE Measures in a Classroom Environment ................................................220
13.3.1 Generality of Measures .............................................................................................220
13.3.2. Implications for Improvement of Measures.............................................................221
13.3.3 Implications for improvement of course design .......................................................222
13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223
13.4.1 The Role of a 360 Review in OBTE .......................................................................223
13.4.2 Narrative of a Participant Observer ..........................................................................225
13.5 Learning, cognitive load and motivation..........................................................................228
13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228
13.5.2 Results.......................................................................................................................229
13.5.3 Implications ..............................................................................................................230
13.6 Conclusions ......................................................................................................................230
13.7 References ........................................................................................................................231
Chapter 14. Organizational Climate and Creation of Durable Change ........................................233
14.1 The Need ..........................................................................................................................233
14.2 Initial Indications of Possible Resistance to Change .......................................................234
14.3 Models and Considerations for Sustainable Change........................................................235
14.3.1 The Change Transition Period ..................................................................................235
14.3.2 Organizational Culture..............................................................................................237
14.3.3 Clarity of Mission and Shared Understanding..........................................................237
14.3.4 Relevant Observations During the Current Investigation.........................................238
14.3.5 Organizational Support and Incentives.....................................................................238
14.4 Conclusions ......................................................................................................................239
14.5 References ........................................................................................................................239

Asymmetric Warfare Group

Table of Contents

xiv

Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242
15.1 Background ......................................................................................................................242
15.2 An Emerging Consensus ..................................................................................................244
15.2.1 What Part to Balance?...............................................................................................244
15.2.2 Improving Training, by Design ................................................................................245
15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246
15.2.4 Future Orientation, Unknown Requirements............................................................247
15.2.5 The Quality Instructor Challenge .............................................................................247
15.2.6 Purpose and Design are Key .....................................................................................248
15.2.7 A Natural Advantage ................................................................................................249
15.2.8 Task Specialization or Generalized Competency .....................................................249
15.3 Conclusion........................................................................................................................251
15.4 References ........................................................................................................................252
Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254
Toward Values-Based Standards for Army Doctrinal Requirements ......................................254
Nested Standards and Quality Assurance.................................................................................256
Needs and Opportunities for Staff & Faculty Development ....................................................259
A Role for Science and Measurement .................................................................................259
Toward Best Practices in Instructor Education....................................................................260
Critical Considerations for Further Scientific Investigation ....................................................263
The Necessity of Long-Term Studies ..................................................................................263
False Dichotomy of Objective-Subjective ...........................................................................264
Clarity About What Is Evaluated.........................................................................................265
Next Steps ............................................................................................................................266
References ................................................................................................................................268
Section IV. Appendices...............................................................................................................270
Appendix A. OBTE Principles & Practices: Instructor Measures................................................271
A.1 Genesis of Formative Measures for Instructors ................................................................271
A.2 Principles of Outcomes-Based Training & Education ......................................................272
A.3 Guide to Using Measures of Instructor Behavior..............................................................276
A.4 Complete Menu of Instructor Measures............................................................................279
Appendix B. OBTE Principles & Practices: Student Measures ...................................................318
B.1 Guide to Using Measures of Student Behavior .................................................................318
B.2 Complete Menu of Student Measures ...............................................................................319
Appendix C: A Commanders View of Outcomes-Based Training and Education .....................340
Summary ..................................................................................................................................340
Definition .............................................................................................................................340
Description...........................................................................................................................340
Elements of OBTE. ..................................................................................................................341
Developing the Outcomes....................................................................................................341
Developing the Training Plan ..............................................................................................341
Conducting Training ............................................................................................................342
How Training is Assessed....................................................................................................344
Conclusion................................................................................................................................344

Asymmetric Warfare Group

xv

Riccio, Diedrich, & Cortes (Eds.)

Appendix D: Warrior Ethos..........................................................................................................345

Analysis of the Concept and Initial Development of Applications..........................................345
Current Understanding of Warrior Ethos.............................................................................345
Purpose.................................................................................................................................348
Approach..............................................................................................................................348
Expansion of the Definition of Warrior Ethos.....................................................................348
The Tenets of Warrior Ethos ...............................................................................................349
Clarifying the Definition of Warrior Ethos..........................................................................351
Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353
References ................................................................................................................................355
Supplementary Work Product from Warrior Ethos Project .....................................................355
Appendix E: Indicators of Warrior Ethos.....................................................................................356
Methods....................................................................................................................................356
Participants...........................................................................................................................356
Instruments and Facilities ....................................................................................................356
Procedure .............................................................................................................................356
Results ......................................................................................................................................358
Qualitative Findings.............................................................................................................358
Quantitative Findings...........................................................................................................358
Discussion ................................................................................................................................359

Asymmetric Warfare Group

Evolution of the investigation as reflected in the chapters of this monograph.