ACHIEVING POSITIVE WASHBACK WITH COMMUNICATIVE LANGUAGE TESTS

Dr Roger Hawkey Visiting Professor, Centre for Research in English Learning and Assessment University of Bedfordshire
3rd Annual IELTS Conference Beijing, 25 November 2010

Which means looking at:

what people mean by washback and related concepts how these relate to communicative language tests Some evidence from related research how positive outcomes might be achieved from the relationships

and my argument will be .

that test washback is a complex matter but needs pursuing because its key to test validity that valid communicative language tests facilitate, but cannot ensure positive washback .
3

Washback (Backwash)?
Washback refers to an exams influences on teaching, teachers, learning, curriculum and materials (eg

Alderson and Wall 1993, Hamp-Lyons 2000, Hawkey, 2006, 9), Shohamy, Donitsa-Schmidt and Ferman, 1996).

Hughes (e.g. 2003) and Green (2007) call the same phenomenon backwash
4

Samuel Messick, test validity guru, refers to washback as . . . the extent to which the introduction and use of a test influences language teachers and learners to do things they would not

otherwise do that promote or inhibit language learning (1996: 241)

Impact
Impact is concerned with wider influences, broader social contexts of tests; washback with micro contexts of the classroom and the school (Hawkey 2006 and Hamp-Lyons 2000). Impact generally agreed to cover the total effect of a test on the educational process and on the wider community (McNamara 2000: 133).
6

Washback : Impact overlap

The impact : washback distinction is useful but does not mean the two are distinct (Saville 2009) Complex relationships between individuals, their institutions and society more generally of crucial importance in understanding how impact works
7

i.e. the stakeholders

Impact at macro-level: positive developments for all stakeholders affected by the introduction of an assessment: learners, teachers, school management, and receiving institutions and organisations, education authorities governments..
8

 all the

stakeholders in all their roles

Government agencies Professional bodies

Context Stakeholders in the Testing Community Testing system Cambridge ESOL

Test construct

Learners Parents/carers Teachers, Heads School owners

Learners Teachers, Heads School owners Test writers Consultants Examiners Test centre administrators Materials writers Publisher inter alia

Test format
Test scores

Receiving institutions Government agencies Professional bodies Employers

Test conditions Test assessment criteria

Academic researchers inter alia)

Input to test design provided by stakeholders

Context of test use where decisions are made by stakeholders using test scores

Backwash, Washback, Impact and consequential validity?

The washback of the test on the learning and teaching that precedes it, as well as its impact on institutions and society more broadly (Taylor (Ed) forthcoming) are key aspects of the consequential validity of the test (Weir 2005) An assessment is valid if: it has the intended positive impact it does not have unintended negative impacts
10

So, test washback and impact are

key aspects of test validity, validation
Impact at micro-level (washback): positive changes in the classroom to content, teaching, and learning outcomes. Impact at macro-level: on stakeholders affected by the introduction of an assessment: learners, teachers, school management, education authorities and government.
11

Impact seen as separate category ?

Cambridge ESOL VRIP = o Validity o + Reliability o + Impact o + Practicality (now termed VRIP Q = VRIP + Quality Assurance)
12

or an element in an integrated model of

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment. (Messick 1989) Test validity seen as including the use and consequences of test results.

a tests usefulness, fitness for purpose?

13

So, how about impact by design?

Both the construct validity of our scorebased inferences and the impact, or consequences, of test use need to be considered from the very beginning of test design, with the test developer and test users working together to prioritise the relative importance of these qualities emphasises Bachman (2005).
14

So, its important to study impact

but by no means straightforward Alderson agrees test consequences are important and may relate to validity issues However, the myriad factors impacting on a test for example, for example teachers linguistic ability, training, motivation, course hours, class size, extra lessons and so on Alderson and
Banerjee (1996)
15

But complex as they are, impact studies are needed, or how do we know what the impacts, part of the validity of a test, are and how they compare with what they should be? If we dont try to find out about the likely impacts, how can we plan for achieving positive washback with communicative language tests?
16

WHERE DOES IMPACT STUDY FIT INTO THE PROCESS OF EXAMINATION DEVELOPMENT?
Washback of test on teaching, learning, materials Washback of test on learner and candidate performance

New / Revised test

Impact of test on other stakeholders, institutions

Impact on receiving institutions, employers, policy makers, testers

Reactions to washback/ impact study, decisions on programme or test

17

Two-way washback?
Washback is not necessarily unidirectional, i.e. from exam to textbook and teaching rather than bi-directional, i.e. also from textbook and teaching to exam (Wall
2005, and Hawkey 2009)

Changes in approaches to teaching and assessment also have washback on exams.

18

Some washback / impact implications from an IELTS study?

A smallish study (see Hawkey 2006), following up on previous joint initiatives between Cambridge ESOL and Lancaster University But with some interesting food for thought on the IELTS washback and impact fronts.
19

Impact Study Participants?

572 IELTS candidates, pre- and post-test (largest subgroup from East Asia and Pacific) 83 teachers completing the teacher questionnaire 43 teachers completing the instrument for the evaluation of textbooks 120 students, 21 teachers and 15 receiving institution administrators in face-to-face interviews, focus groups

20

KEY QUESTION : IS IELTS A FAIR

TESTOF PROFICIENCY?
No 28%

Yes 72%

If No, why not?

1 2 3 4 5 Opposition to all tests Pressure, especially time Topics Rating of writing and speaking No grammar test
21

CANDIDATES (post-IELTS) TOP LIKES (n=132)...

1 VALIDITY?
recognition (7), language and study skills (2) fair (17), 4-skills/ comprehensiveness (15)

41 17 16 15 14 13
22

2 SPEAKING 3 STRUCTURE, ORGANISTION, FORMAT 4 WRITING 5

INCENTIVE, CHALLENGE, INTEREST, VARIETY

6 LISTENING

AND DISLIKES? (n=138)

1 TIME PRESSURE 2 READING 3 LISTENING 4 WRITING 5 COMPLICATED QUESTIONS 6 LANGUAGE DIFFICULTY 7 SPEAKING 50 41 18 16 9 8 7
23

WHAT IS MOST DIFFICULT ABOUT IELTS?

Learners Reading Writing Listening Speaking Teachers Most difficult IELTS module (%)

49 24 18 9

45 26 20 9
24

IELTS SKILLS: STUDENTS PERCEIVED DIFFICULTY AND PREP COURSE TIMINGS:

28

17

20

19

Tim e

49

24

18

Difficulty

20 Reading

40 Writing

60 Listening Speaking

80 Grammar Vocab

100

120

25

DID YOUR IELTS PREP COURSE PROVIDE YOU WITH THE LANGUAGE KNOWLEDGE AND SKILLS YOU NEED?
17%

Yes No

83%
26

STUDENTS WORRY AT TAKING IELTS

A bit 19%

V little 9% Very much 41%

Quite a lot 31%

27

AND TO THE PREP COURSE TEACHERS?

No Don't know No answer 27 11 1

Does the IELTS test cause stress for your students? Yes 44

Does the IELTS test provide motivation for your students?

Yes No Don't know No answer 70 8 4 1

28

Conditions that need to be in place for positive washback

include: making the test tasks direct and criterionreferenced optimising overlap between test and target language domain demands emphasising the importance, demandingness, but attainability of the test ensuring the test-takers and teachers are familiar with the test providing optimal support for teachers. But the effect of the many intervening variables always remains.
29

Food for thought .

that test washback is a complex matter but needs pursuing because its key to test validity that valid communicative language tests facilitate, but cannot ensure positive washback .
30

Thank you. Roger Hawkey

rogerhawkey@btinternet.com Hawkey.r@cambridgeesol.org

31

References
Alderson J C & Wall D (1993) Does washback exist? Applied Linguistics 14(2): 115-129 Alderson J C and Banerjee J (1996) How might impact study instruments be validated? Paper commissioned by the University of Cambridge Local Examinations Syndicate (UCLES) as part of the IELTS Impact Study Bachman L (1990) Fundamental Considerations in Language Testing, Oxford: Oxford University Press. Bachman (2005) Building and supporting a case for test use, Language Assessment Quarterly 2, 1, 1-34 Bachman L and Palmer A (1996) Language Testing in Practice, Oxford: Oxford University Press. Green A (2007) IELTS Washback in Context: preparation for academic writing in higher education, Cambridge: Cambridge University Press Hamp-Lyons, L (2000) Social, professional and individual responsibility in language testing, System 28 (4), 579-591. Hawkey, R (2006) Impact theory and practice: studies of the IELTS test and Progetto Lingue 2000, Cambridge: Cambridge University Press Hawkey R (forthcoming) Consequential Validity, in Taylor, L (Ed) Hughes A (2003) Testing for language teachers, Cambridge: Cambridge University Press.
32

McNamara, T. 2000. Language Testing. Oxford: Oxford University Press. Messick, S (1989) Validity, in Linn, R L (Ed.) Educational Measurement (3rd ed.), New York: Macmillan, 13-103. Messick, S (1994) The interplay of evidence and consequences in the validation of performance assessments, Educational Researcher 23 (2), 13-23. Messick S (1996) Validity and washback in language testing, Language Testing November 1996 vol. 13 no. 3 241-256 Saville N (2009) Developing a model for investigating the impact of language assessment within educational contexts by a public examination provider, unpublished PhD thesis. Saville N., & Hawkey, R (2004). The IELTS Impact Study: Investigating Washback on Teaching Materials. In L. Cheng & Y Watanabe (Eds.) 2004: 97-112. Shohamy, E, 1999 Language Testing: Impact in B. Spolsky, ed., Concise Encyclopaedia of Educational Linguistics ((oxford 1999) 711-714 Shohamy E (2001) The Power of Tests: A critical perspective on the uses of language tests, Harlow: Pearson Education. Shohamy, E. Donitsa-Schmidt S & Ferman I.(1996). Test impact revisited: Washback effect over time. Language Testing, 13 (3) 298-317 Taylor L Ed (forthcoming) Examining Speaking, Cambridge: Cambridge University Press. Wall D (2005) The Impact of a High-Stakes Examination on Classroom Teaching: A Case Study Using Insights from Testing and Innovation Theory Studies in Language Testing 22 UCLES Cambridge: Cambridge University Press. Weir C (2005) Language Testing and Validity Evidence: Oxford:. Palgrave.

33