8/9/2014

1
Evaluating Evaluation:
Materials Evaluation Tools in Question
Mian Md. Naushaad Kabir, PhD
Assistant Professor
English Department
East West University
naushaadk@gmail.com
What I am going to present
Evaluation tools, their types & nitty-gritty
Aspects/ criteria: core (general) & peripheral
Ready-made tools vs. home grown materials
Issue of Customization of tools
Locally produced tools vs. locally produced materials
Recent curriculum guidelines (2012) vs the tool
A Framework for analysing & evaluating local
materials/coursebooks
Formaldehyde Meter
Evaluation tools, their types & nitty-gritty
Checklist including statements
Allwright (1981), Cunningsworth (1995),
McGrath (2002)
Checklist with MCQs, yes/no questions, & open-
ended questions Cunningsworth (1984)
Checklist with questions on basic assumptions, and
shape and design Dubin & Olshtain (1986)
Types of tools depends on
- The agent and number and type of evaluator
The WHY of evaluation
- for updating/ adapting materials, for finding gaps
and problems, for measuring effectiveness
The WHAT of evaluation
- GOAL, CONTENT, METHOD and GUIDANCE (Allwright
1981); Integration of grammar & notion, themes &
topics and communicative & sociocultural functions
Continued from the previous page
Pedagogical validity (theoretical assumptions against
changing needs)
Psychological validity (how to learn) and
Process & content validity
8/9/2014
2
Continued from the previous page
Differences in evaluation for purpose, evaluator,
modality and time (Tomlinson 2003)
3 types of evaluation:
Pre-use
While-use
post-use
(Ellis 1997; McDonough & Shaw 1993/ 2003;
McGrath 2002; Tomlinson 2003)
Aspects: General & Peripheral
Importance of criteria to be determined by evaluators
by reaching a consensus regarding that Ur (1999)
General criteria (essential) and specific criteria
(contextual) Ur (1999)
Essential, desirable, additional and inclusion of
negative factors Chambers (1997)
Media-specific, content specific, age specific and local
criteria McGrath (2002); Tomlinsons (1999)
7 general criteria (Cunningsworth 1984)
language content
selection and grading of language items
presentation and practice of new language items
developing language skills and communicative
abilities
supporting materials
motivation and the learners and
finally overall evaluation
Peripheral Criteria (Dubin & Olshtain 1986)
Questions about shape and design:
1. Length and size
2. Internal format
3. Sequencing and grading
4. Processes: linear, cyclic or spiral?
5. Consumable: If materials allow learners to
write on them.
6. External format: Graphic devices.
7. Ancillary: additional elements e.g. word lists,
glossary, indexes etc.
8. Visual and auditory aids: pictures, art work,
realia etc.
9. Options: If they allow alternatives or
adaptations?
10. Evaluation: how will learners mastery be
evaluated?
Ready-made tools vs. home grown
materials
Based on some books
Suitable for its own
culture
Various purposes
Not tailor-made
Can have andragogical
implication
Based on context
Suitable for its own
culture
For learning
Most often tailor-made
Have pedagogical
implication
8/9/2014
3
Issue of Customization of tools
Who = Agent/ Evaluator
What = Content i.e., text, grammar,
function, illustration, vocabulary etc.
When = pre-/ while-/ post
Why = CPD/ reflective practices/
professional practices/ betterment of
the learners
How= Individually/ collectively/ in
peer/ in pair etc.
Locally produced tools vs. locally
produced materials
Based on home-grown materials
Familiar with the context
Customized as per the necessity
Culture specific
Relevant and hence valid and reliable
Great opportunity for collective professional
self development through CPD
Always for the betterment of the learners
Recent curriculum guidelines (2012)
vs locally produced tool
Conclusion
Evaluate course book for
successful learning and teaching
professional development
being updated
to place voices of learners & teachers in
education
Book Map: EfT Classes IX-X (Old)
Unit 1: Hello and Welcome! Lesson 1: Im Becky
Topic Skills Function Grammar/
Structure
Vocabulary
Im
Becky
Reading,
Writing
&
Speaking
Greeting,
introducing,
exchanging
personal
information
Present
simple, Wh-
questions
Jamboree,
newsletter,
interview
A Framework for analysing & evaluating
local materials/coursebooks
Analysis of Physical Aspects
1. All units are of same length Yes/ No
2. Page quality is suitable for the learners of that
age-group Yes/ No
3. The printing is clear Yes/ No
6. Any other comments:
8/9/2014
4
For Book
7. Visual aids e.g. pictures, diagrams, charts, bubbles etc.
are clear Yes/ No
8. Please look at the pictures from page 100 to page 126
and reply the following questions:
a. How many men and how many women do you find?-
------
b. How many people of the learners age (irrespective
of sex) do you find?-------------
d. Do they highlight any specific religion? If yes,
which religion?-------------------
e. Do they highlight any specific class (e.g. lower class,
middle class etc.)? Yes/No
if yes, which class?--------------
Analysis of linguistic, pedagogical and psychological, content and
process aspects
9. Topics can make bridge between the native culture and target
culture 4 3 2 1
10. Topics are relevant to learners 4 3 2 1
11. Topics are interesting to learners
a. linguistically 4 3 2 1
b. thematically 4 3 2 1
c. culturally 4 3 2 1
12. The book provides learners with a variety of materials (e.g. reading
texts, pictures, tasks) 4 3 2 1
14. a. All language skills i.e. listening, speaking, reading and
writing are given equal importance in every unit/lesson Yes/No
b. If the answer is no, write them in order of priority
1 2 3. 4
Unit Evaluation: (for answering this set, take into
account all the 5 lessons of Unit 11)
29. The lesson objectives are honestly followed in the
lesson 4 3 2 1
30. The lesson begins effectively to draw learners attention
4 3 2 1
31. The activities are interesting to engage them
4 3 2 1
32. Recapitulation of things learnt is there at the end of the
lesson 4 3 2 1
Task Evaluation (Please look at 3 or 4 tasks of
different lessons in the unit)
46. Tasks can be administered ensuring the
participation of each learner in class time
4 3 2 1
47. Tasks provide enough scope for the teacher to
give feedback to each learner 4 3 2 1
48. Objective (s) is/are fulfilled at the end of the
task 4 3 2 1
Analysis of linguistic, pedagogical and psychological, content
and process aspects
Grammar and composition is
60. taught in context. 4 3 2 1
61. integrated with all skills. 4 3 2 1
62. taught using audio-visual aids. 4 3 2 1
63. is approached communicatively 4 3 2 1
64. taught through topics. 4 3 2 1
Unit Evaluation: Grammar
65. a. Is the unit interesting? Yes/No
b. why?
66. The unit provides with enough scope of practice 4 3 2 1
67. Grammar is connected to real life use 4 3 2 1
68. The level of grammar is compatible with learners level
4 3 2 1
70. It enhances
a. internalization through understanding b. memorization
through rote learning
8/9/2014
5
Unit Evaluation: Composition
78. Describe how composition is taught on the
basis of the unit?
79. What do you like about it?
80. What do you dislike about it?
Overall Assessment
84. The book is self-sufficient for teachers for
teaching grammar 4 3 2 1
85. The book is self-sufficient for learners for
teaching grammar 4 3 2 1
86. The book is self-sufficient for teachers for
teaching composition 4 3 2 1
Thank you