Writing Effective Tests

 The following slides summarize the work

of Lucy Jacobs and Clinton Chase from
their book Developing and Using Tests
Effectively: A Guide for Faculty

 The book is a compilation of years of

research on how to develop effective
test instruments.
Writing Tests
 Most higher education faculty never took a
course in how to write tests. Their experience
comes from the many tests they have taken over
their many years in school. This personal
experience is not a reliable way to learn to
develop tests that will determine so many
important outcomes for our students.

 Even with diligence, time and practice writing

tests, most tests given in college have
weaknesses and are not highly reliable.

 However, the following presentation can

significant help to improve the quality of the tests
Writing Fair and Effective
Tests
 Purpose of  Benefits to
Tests Students
 Grades  Learn how they
 Planning content are tested
presentations  Engage in deep or
 Facilitate learning surface learning
 Motivate students based on tests
 Feedback on
learning
Test Benefits to Instructors
 Focus on essential  Feedback if course
course content objectives were
 Message to achieved
students on what’s  Feedback on
important effectiveness of
 Valuable feedback instruction
on learning  A motivational
strategy
Problems with College Tests
 Don’t focus on  Too short to
what’s important sample content
 Too little feedback fairly
given to students  Purpose of test not
 Questions made clear-so
ambiguous or students see it as
unclear penal
 Too few tests-
don’t have a broad
view of students
knowledge
How Long Should a Test Be?
 True-False 30  Matching 30
sec. sec.
 Multiple choice 1  Short Essay
min.  10-15 minutes.
 Completion 1  Extended essay
min.  30 minutes
 Short answers 2
min.
 Multiple choice
90sec. higher
level thinking
Number of Test Items
 Fifty Minute Tests
 40-45 multiple
choice
 60-80 true-False
 Fastest students
finish in half the
time of the slowest
 Tests should allow
almost every
student to attempt
every item
Test Item Difficulty
 Tests should  Difficulty can be:
produce a spread  A. content it asks
of scores
 Test items should  B. cognitive skill
be of average level being
difficulty measured
 Items that 50-70%
of the students
can answer
 C. Construction of
correctly the
 If they didn’t study item
Test Reliability
 Long enough to  Directions should
sample content be clear.
well  Scoring procedure
 Most student needs to be free of
should be able to errors.
finish  All students should
 Score range take the same test
should be wide
 Items free of
ambiguity and
tricks
Test Validity
 Samples the  What limits
course objectives validity?
in proportion to  A. Directions not
their importance clear
 Use a test matrix  B. Tests skills not
to check part of the
distribution of course objectives
questions by  C. Too short
content and
cognitive level.
 D. Subjectivity of
scoring
Multiple Choice Tests
 Write a stem in the  Do not pad the
form on an stem-write them
incomplete simple and clear.
statement or  Distracters must
question that be plausible
could be answered  Avoid using humor
without look at the
four choices
 Make certain there
is only one best
 Always put a verb answer
in the stem
Multiple Choice Tests
 Make distracters  Avoid giving
fairly grammar clues
homogeneous-this  careful with
will force students specific modifiers--
to be ie. Never, always
discriminating  Limit use of all of
 Don’t give the above and
irrelevant clues or none of the above
tricks-these
reduce validity
 Make answers
Multiple Choice Tests
 Arrange options in  Avoid asking trivia
logical(alpha )  Ask students to
order choose the best
 Keep correct answer not the
response letter correct answer.
( number)
balanced
 Don’t lift stems
verbatim from text
 Arrange answers
in vertical columns
True-False Tests
 Advantages  Limitations
 A. Sample many  A. Difficult to write
more bits of so there not
information ambiguous or
 B. Ask more obvious
question in 50  B. Difficult to make
minute time clearly true or
period helps clearly false
reliability statements
 C. Research  C. 50-50 chance
affirms True-False  D. Educated guess
True-False Tests
 Avoid use of  Write questions
specific that are clear and
determiners ( all, simple to read.
never)  Make it clear
 Avoid use of where the answers
qualifiers(sometim are to be placed--
es, usually) don’t use + or -
 Avoid use of
indefinite terms
(long time ago)
True-False Higher Level
Thinking
 Use propositional  Setting up a
logic--if then problem that
approach. requires analysis.
 Example-If the  Example --John
gross national bought a car…it
product goes up won’t start. Which
and inflation is are possible
3.4%…then the reasons?
stock market will
likely go up.
Short Answer or Completion
Items
 Requires recall  Questions should
rather than have a single word
recognition of answer.
information.  Make certain there
 Doesn’t test is only one right
higher level answer
thinking  Delete only key
 Best used to see words
what information  Make blanks same
students have length
collected.  Don’t lift
Matching Tests
 Stimulus column  Put stimuli in
on the left alphabetical order
numbered--  Use 10-15 items
response column  Keep entire test on
on right lettered. one page.
 Provide space to  Have at least five
right answer to the more stimuli than
left of stimuli. responses
 Use only
homogeneous
material
Essay Tests
 Advantages  4. Can’t simply
 1. Can assess recognize an
complex learning answer
 2. Easy to write  5. Doesn’t allow
 3. Emphasize for guessing--
communication except bluffing
skills as  6. Can see how
fundamental students select,
organize and
evaluate ideas and
apply them
Essay Tests
 Limitations  4. Score
 1. Difficult to score influenced by
 2. Scores are less readers overall
reliable impression of the
student
 3. Sample only a
limited amount of
 5. Not favorable to
content developing good
writing skills
Essay Reliability Concerns
 Less reliable than  A second reading
objective tests often results in a
 Environmental different grade.
factors influence  Expectations of
scoring ie. Time of students’
day, mood of performance
reader where  Physical elements
paper is in the of the paper--
stack handwriting
 Previous paper can  Limited number of
influence current questions luck of
paper the draw
Making Essay Tests Better
 Restrict essays to  Keep question
assess only focused on a
outcomes that single objective
require complex  All students should
higher level respond to the
cognitive same questions-
functions. don’t give choices
Examples  Grammar and
 1. Compare spelling count only
-Contrast if taught as an
 2.Present objective in the
Making Essay Tests Better
 Directions need to
be very clear and
include type of
response
expected--Example
outline, list,
discussion
 List the point value
of each question
Scoring Essay Tests
 Conceal students  Use a prescribed
names reading procedure
 Skim a few before  A. Key procedure
starting to grade  lays out ideas that
to get a sense of should have been
responses in the essay
 Read only one  B. Ranking
item across all Procedure
papers  Makes piles based
 Reshuffle the stack on quality
Open-Book Tests
 Advantages  Limitations
 Used in classes  Students tend not
where formulas, to prepare well
tables or graphs  No evidence to
are needed to find suggest students
solutions do any better
 Student
demonstrates they
know how to use
references
 Reflect real life
Take Home Tests
 Advantages  Limitations
 1. More thorough  1. Question rarely
exploration of sample adequately
ideas can take course material
place  2. Much more
 2. No time grading time
restrictions  3. Students often
 3. Usually better unsure of how
written thorough to be.
Re-takes of Tests
 Advantages  Limitations
 Retakes do result  A large test bank
in better learning is needed
 Lower test anxiety  Grading time
 Students learn increases
from their  Students don’t
mistakes often prepare well
 It eliminates the for the first test
need for make-up
tests
Helping Students Prepare for
the Tests
 Review the scope  Review make-up
of the test policy
 Use a practice test  Provide a review
 Clarify time limits session
 Clarify what  Make provisions
material will be for last minute
needed and questions
allowed  Allow for breaks
 Review grading during long exams
procedure