Test Format

Introduction

The typical teacher will spend one-quarter to

one-third of his or her professional time
involved in assessment-related activities
without question, teachers need to know
and understand the principles of sound
assessment.
(Stiggins, 2004)

Planning a Test

Goals for our session:

To understand what makes for good test items & how
to write them.
FOCUS: Multiple-Choice Question (MCQ) &
Constructing Essay Questions
(Based on Blooms cognitive taxonomy)

Planning a Test

Use a test matrix or blueprint.

Identify major ideas and skills

rather than specific details.

Use Blooms cognitive taxonomy

or something appropriate for
your context.
(Nilson, 2010)

Learning Outcomes: Using Blooms Taxonomy

The basis of all exam questions should be
learning outcomes.
Using Blooms Taxonomy can help you
clarify what you want students to learn.
By setting learning outcomes at different
levels of Blooms Taxonomy, you will
target exactly what you want students to
know, be able to do, or have experienced
as a result of learning activities in class.

Blooms Taxonomy

Examples of questions for different levels:

http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-bloom.php
6

TYPES OF TEST ITEMS

SELECT
TYPE

Multiple Choice

Matching

SUPPLY TYPE

Short Answer
Structured Response
Free Response
Completion/Cloze
Essay

Test Items vs Cognitive Levels of Learning

Comprehension
Knowledge
MCQ
True/False (TF)
Matching
Completion
Short Answer

Application

MCQ
Short Answer
Essay
Structured Response (MCQ, TF, Likert
Scales)
Free Response (Essay, Interview Qs, Fill
in the blanks)
Performance

Synthesis
Analysis
Evaluation
MCQ
Short Answer
Essay
Free Response

TEST TYPES - General

Structure
Structured
Response
Advantages

Great Depth
Quick Scoring

Free
Response
Advantages
Provide full and
independent responses
gives a better
indication of the
competence and
intelligence of the test

Disadvantages
Limited Depth
Difficult to assess higher
levels of skills
Guessing/Memorization vs.
Knowledge

Disadvantages

the difficulty, bias, and time

effort required in grading them
the quality of a free response
answer is often determined more
by ability with the language in
which it is given than
competence or intelligence
8

TYPES OF TEST TYPES

Choose the correct word to
complete the sentence.
Multiple Choice
Questions

Gap-filling

Cook is ________today for being

one of the Britains most
famous explorers.
A. recommended
B. recognised
C. reminded
D. Remembered

Can be used to test:

grammar, vocabulary,
reading, listening etc.

Complete the sentence.

Can be used to test:

grammar, vocabulary,
Very effective at testing
listening for specific
words.

Check the exchange _________to see

much your money is worth.

Match the word on the left to the word

with the opposite meaning.

Matching

fat
young
dangerous
short

old
tall
thin
safe

Can be used to test:

- Most often used to test
vocabulary

TYPES OF TEST TYPES

Complete the text by adding a word to
each gap.
Cloze

This is the kind________test where a

word ________omitted from a passage
every so often. The candidate must
_______the gaps, usually the first two
lines are without gaps.

Can be used to test:

Very effective for testing grammar,
vocabulary & intensive reading

Decide if the statement is true or false.

True-False

Can be used to test:

Most often used to test vocabulary

England won the world cup in 1966.

T/F
Answer the question (after reading or
listening.)

Open
Questions

Why did John steal the money?

Can be used to test:

Very useful for testing any of the four
skills. But less useful for testing
grammar or vocabulary

Error
Correction

Find the mistakes in the sentence and

correct them.

Manchester United was the more

better team on the night.

Etc.....
(other Test
types)

Can be used to test:

Useful for testing grammar &
vocabulary as well as listening and
reading

Translation
Essay
Dictations
Sequencing/ordering of words /
phrases
Sentence construction

10

Multiple-Choice Tests

11

When to Use
To assess breadth of learning
To test a variety of levels of learning
When you have a large number of individuals
taking the test
When you have time to construct the test items
When time is limited for scoring
When it is not important to determine how well
individuals can formulate their own answer
When you want to prepare individuals for future
assessments that use a similar format

(Clegg & Cashin, 1986)

12

Multiple-Choice Tests
Students select the correct answer from
alternative responses. Each item has:
item stem
correct or keyed option
several distractor options
Format:
complete question
incomplete question

(Clegg & Cashin, 1986)

13

Terminology: Multiple Choice

1

1. The capital city of Canada is

a. Vancouver
3 b. Montreal
c. Toronto
*d. Ottawa
2

1. Stem: presents the problem

2. Key: correct or best answer
3. Distracters or Alternatives: appear to be reasonable answers
to the examinee who does not know the content
4. Level of difficulty and topic or chapter reference

14

Constructing Items MCQ Stem should:

Clearly defines a problem/question
(specific is terrific!)
Provide sufficient information
Avoid extra information / clues (a, are, is,
etc.)
Be grammatically correct
Avoid clues that help give away answers
Avoid negatives and absolutes

15

Constructing Items
MCQ Responses should:

Be plausible/logical/reasonable
Be grammatically consistent with the stem
Be approximately the same length or other logical order
(alphabetical, numerical, time series..)
Make alternatives mutually exclusive (i.e. do not overlap)
Provide only one correct answer
Avoid all or none of the above
Incorrect answers (distractors) should not be too obvious but
should be justifiably incorrect
When composing incorrect alternatives, include any
misconceptions that students are likely to hold
Do not lift a statement exactly from the textbook

16

Advantages
Multiple-choice items can provide:
versatile in measuring all levels of cognitive
ability,
highly reliable test scores,
scoring efficiency and accuracy,
objective measurement of achievement or ability,
a wide sampling of content or objectives,
a reduced guessing factor compared with truefalse items,
different response alternatives which can provide
diagnostic feedback.
(Ory & Ryan, 1993)

17

Limitations
Multiple-choice items:
are difficult and time-consuming to construct,
lead an instructor to favour simple recall of
facts,
place a high degree of dependence on the
students reading ability and instructors
writing ability, and
are particularly subject to clueing. (Students
can often deduce the correct response by
elimination.)

(Ory & Ryan,

1993)

18

Objectives at Different
Levels

Level: Knowledge

Level: Comprehension and

Application

Objective: State the average

effective radiation dose from
chest Computed Tomography
(CT).

Objective: Compare the

radiation exposures from
different radiologic
examinations.

What is the average effective

radiation dose from chest Computed
Tomography (CT)?
A.
1 mSv (millisievert )
B.
8 mSv
C.
16 mSv
D. 24 mSv

Which of the following imaging

examinations is associated with the
highest effective radiation dose?
A.
Abdominal and pelvic
multidetector CT
B.
Coronary artery
multidetector CT
C.
Conventional pulmonary
angiography
D. Digital pulmonary
angiography

19

Analyse the following sample:

1. This floor is made of ___________
A ciment
C
cemment
B cemnet
D
cement
2. Skinner developed programmed instruction in
_____.
A. 1953
B. 1954
C. 1955
D. 1956

20

About Essay Questions

21

Essay Questions
Provide

a freedom on response that

facilitates assessing higher
cognitive behaviors (e.g., analysis
and evaluation)

Allow

respondent to focus on what

they have learned and do not limit
them to specific questions
22

Essay Questions
Types of Essay Questions
Extended Response Question
Great deal of latitude on how to respond to a
question.
Example: Discuss essay and multiple-choice
type tests.
Restricted Response Question
More specific, easier to score, improved
reliability and validity
Example: Compare and contrast the relative
advantages of disadvantages of essay and
multiple choice tests with respect to:
reliability, validity & objectivity
23

Constructing Items: Essay Tests

Make the essay items reflect course objectives

Should measure an objective at Blooms - Synthesis/Creating level
Include specific directions such as compare, contrast, define,
discuss, formulate, etc...
Prepare students for the test and provide essay test taking tips
Plan ahead, and manage your time wisely
Have each student answer all the questions
State the number of points each question is worth
For controversial topics, indicate clearly that the student should
logically present a position rather than take a specific stand
Limit the scope of the question
Example: Write a two-page statement defending the importance
of conserving our natural resources.
(your answer will be evaluated in terms of organisation, its
comprehensiveness and the relevance of the arguments
presented)

24

Essay Questions

Advantages:

Measure higher learning levels (synthesis,

evaluation) and are easier to construct than
objective test items
Students are less likely to answer an essay
question by guessing
Require superior study methods
Offer students an opportunity to demonstrate
their abilities to:
Organize knowledge
Express opinions
Foster creativity

25

Essay Items/Questions
Disadvantages
May limit the sampling of material covered
Tends to reduce validity of the test
Subjective unreliable nature of scoring
halo effect good or bad students
previous level of performance
Written expression
Handwriting legibility
Grammatical and spelling errors
Time Consuming
26

Scoring Essay Items

Write an outline of the key points (use outline to

design a rubric analytic or holistic)
Determine how many points are to be assigned to
the question as a whole and to the various parts
within it.
If possible, score the test without knowledge of
the students name
Score all of the answers to one question before
proceeding to the next question
Consistent standard

27

Alternative Methods of
Assessment

Research/Term Papers
Research Reviews
Reports
Case Studies
Portfolios
Projects
Performances
Peer evaluation
Simulations

28

Mistakes in Writing Essay Questions

The main mistakes instructors make when
writing essay questions are:
Writing vague, non-specific questions.
Asking for too many parts in an answer.
Asking questions over material that
students did not read or cover in class.
Asking questions that are difficult to grade
(e.g. asking for students opinions).

29

Tips for Writing an Effective Essay

Question

To write an effective essay question:

1.

Decide exactly what you want to know about the students

learning. Do you want to know that they can tell you how a
complex economic system forms? If so, ask that.

2.

Cue students to the material from texts and class that forms
the basis of your questions.

3.

Put points in parentheses after each component of the

question so students will know exactly which parts to
answer and how much those parts will count.

4.

Use key words so students will understand exactly how you

want them to answer.

30

Tutorial Task

You are going to design a quantitative research project to

look at students motivation in classrooms. You want to
study ways their past experiences, current expectations,
and self- efficacy influence their study behaviors this
semester.

1.

Using what you learned from your textbook, list your

independent and dependent variables (2 marks);
explain for each how you would operationalize that
variable (5 marks);
Finally, discuss at least two ways you would analyze the
data that you collect (10 marks).

What are the reasons this is an effective essay

question? Justify your answer.

31

Resources

1.

2.

3.

4.

5.

6.

7.

Brown, H. Douglas, (2010). Language Assessment Principles and

Classroom Practices. (2nd ed). New York, Pearson Longman.
Clegg, V. L., & Cashin, W. E. (1986). Improving multiple-choice tests.
Idea Paper No. 16, Center for Faculty Evaluation and Development,
Kansas State University.
http://www.idea.ksu.edu/papers/Idea_Paper_16.pdf.
Davis, B. G. (2009). Tools for teaching. (2nd ed.). San Francisco, CA:
Jossey-Bass.
Gronlund, E. Norman, (2006). Assessment of Student Performance.
(8th ed). Boston, Pearson Allyn and Bacon.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review
of multiple-choice item-writing guidelines for classroom assessment.
Applied Measurement in Education, 15(3), 309-334.
Nilson, L. B. (2010). Teaching at its best: A research-based resource
for college instructors. (3rd ed.) San Francisco, CA: Jossey-Bass.
Ory, J.C. & Ryan, K. E. (1993). Tips for improving testing and grading.
Vol. 4. Newbury Park: Sage Publications.

32

Further Reading
Svinicki, M. & McKeachie, W. J. (2011). McKeachie's teaching tips: Strategies,
research, and theory for college and university teachers. Belmont, CA:
Wadsworth, Cengage Learning.
University of Oregon, Teaching Effectiveness Program. Writing Multiple Choice
Questions that Demand Critical Thinking. Web site: h
ttp://tep.uoregon.edu/resources/assessment/multiplechoicequestions/mc4cr
itthink.html
University of Minnesota, Office of Measurement Services. Writing Multiple
Choice Items. Web site:
http://oms.umn.edu/fce/how_to_write/multiplechoice.php
University of Texas at Austin, Instructional Assessment Resources. Writing
Multiple Choice Items. Web site:
http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method
/exams-mchoice-write.php

33

REFLECTION
A good test needs an instrument of quality.
An instrument of quality needs items that are
of quality.
Items that are of quality need item builders
who are of quality.
DO YOU HAVE ALL THOSE QUALITIES?

(Haladyna, Downing &

Rodriguez, 2002)

34

ISL
MCQ: Writing Item
Develop as many effective choices as you
can, but research suggests three is
adequate.
Make sure that only one of these choices is
the right answer.
Vary the location of the right answer
according to the number of choices
Place choices in logical or numerical order.
Keep choices independent; choices should
not be overlapping.
(Haladyna, Downing &
Rodriguez, 2002)

35

ISL
MCQ: Writing Item
Phrase choices positively; avoid negatives such
as NOT.
Avoid giving clues to the right answer, such as:

o specific determiners including always, never,

completely, and absolutely
o choices identical to or resembling words in the stem
o conspicuous correct choice
o pairs or triplets of options that clue the test-taker to
the correct choice
o ridiculous options
(Haladyna, Downing & Rodriguez,
2002)

36

ISL
MCQ: Writing Item

Keep choices homogeneous in content and

grammatical structure.
Keep the length of the choices about equal.
None-of-the-above should be used carefully.
Avoid All-of-the-above.
Make all distractors logical.
Use typical errors of students to write your
distractors.
Use humor if it is compatible with the
teacher and the learning environment.
(Haladyna, Downing & Rodriguez, 2002)
37