Academic Writing Assessment 1

Running head: ACADEMIC WRITING ASSESSMENT

Academic Writing Assessment

Assessment Final Project

Eman M. Eltukri

University of Southern California

EDUC 527: Assessment in the Language Classroom

December 9th, 2009

Academic Writing Assessment

 Academic Writing Assessment 2

With the growing number of Libyan students granted scholarships to study in the United

States, many courses are currently offered in Libya to assist students with their academic

reading, writing, listening and speaking skills. As a future member of the teaching staff at the

University of Garyounis in the city of Benghazi- Libya, I designed a course in the last spring

semester as a part of the ‘Course Design’ class for teaching academic writing to Libyan

students at Garyounis University. The course aims at improving students’ academic writing

in order to prepare them to study towards their higher studies in the U.S.

In the “initial needs assessment” stage of the course design, I suggested testing the

students’ academic writing ability on the second day of the course to diagnose areas of

strengths and weaknesses to help me shape the course accordingly. This would be a great

chance to design this diagnostic test to help determine what they need to work on during the

course to help them succeed in their studying experience in the U.S.

From this perspective, the current paper seeks to present a proposal for designing a

formative diagnostic test that is related to the aforementioned academic writing course. The

test can be characterized as formative because it evaluates test takers’ competencies with the

goal of helping them, and it gives them analytical feedback. The current paper provides

information about the test design process, test operationalization as well as a hypothetical

analysis for test results.

Test Design

Test Purpose

The purpose of the proposed test is to make inferences about students’ ability of

constructing academic essays. It will evaluate content, structure and organization. The test

can be classified as a low stake test because students will not be affected by the test results;

however, they will help the teacher in identifying what the students’ needs are.

Description of TLU Domain and Task Types

 Academic Writing Assessment 3

In order for the test to be useful, the test tasks should reflect the language used in

the target language use (TLU) (Bachman & Palmer, 1996). The TLU domain of the

current test can be initially identified as “English for academic purposes”. As for the

typical TLU task type, test takers will choose between three topics in order to develop

an essay. They have to state their opinions clearly on the subject matter chosen and

support their arguments with examples. They also have to pay attention to

organization, i.e. they have to provide an introductory paragraph with a clearly stated

thesis statement, three body paragraphs with their topic sentences and supporting

examples, and a conclusion that restates their opinion and summarizes the main

points. Test takers would very likely encounter similar tasks in the TLU domain as

.they will have to synthesize different essays

Characteristics of Test Takers

The test will be designed for Libyan students whose ages vary between 22 to 30. The total

number of the test takers is 20; males and females. Their native language is Arabic. They are

enrolled in general English classes and their current level of English proficiency is upper-

intermediate. All of the test takers have graduated from different schools at Garyounis

University. Some of them are currently teaching assistants in their departments, and others

hold master’s degrees and have already started to teach at the same university. All of them

are granted scholarships to study towards their master’s or doctoral degrees in the U.S.

Test takers’ topical knowledge will relatively vary because of their different majors.

However, the prompts given on the test to be chosen between them are general in nature, i.e.

they are not related to a specific field of study, like Physics or Literature. Therefore, test

takers will not be influenced by their topical knowledge as well as their affective schemata

due to the general nature of the prompts. In addition, they have an acceptable amount of

language knowledge- both organizational and pragmatic knowledge- due to their current
 Academic Writing Assessment 4

English language proficiency level. That is, they have knowledge of general writing skills.

For instance, they have good note-taking skills, they can fill out an evaluation form, and they

can write reflections and short paragraphs. Nevertheless, they lack academic writing

composition skills.

Definition of Construct

The construct to be measured is related to the “linguistic competence”. It will be measured

in terms of the following components of language knowledge:

• Knowledge of syntax: accurate use of syntactic structures.

• Knowledge of essay organization: introduction, body and conclusion.

• Knowledge of cohesion: accurate use of transitions and cohesive devices.

• Knowledge of register: knowledge of a range of markers of formal and informal

registers (Bachman & Palmer, 1996).

Resources Available for Creating and Administering the Test

The test will take place in the classroom where the students are going to have their

academic writing course. The developer and the rater of the test is the teacher. The designated

time for the test is the class session which is two hours. The materials that will be used are

papers, pencils, erasers and pencil sharpeners.

A Plan for Evaluating the Qualities of Usefulness

According to Bachman and Palmer (1996, p. 16), “the most important consideration in

designing and developing a language test is the use for which it is intended, so that the most

important quality of a test is its usefulness.” They proposed the following six qualities which

determine test usefulness: reliability, construct validity, authenticity, interactiveness, impact,

and practicality. Weigle (2002, p. 48) emphasized that “it is virtually impossible to maximize

all of them […] [however,] the test developer is to determine an appropriate balance among

the qualities for the specific situation.” Unfortunately, there was no chance for piloting the
 Academic Writing Assessment 5

current test in order to evaluate the quality of usefulness. However, given the construct

definition and test purpose, the following expectations can be drawn in terms of evaluating

the quality of usefulness:

• Reliability

The test consists of a single test task with a narrow range of construct components so

realistic to expect high level of reliability. Test takers are very likely to receive the same

score across different characteristics of the testing situation such as different prompts and

different raters.

• Construct validity

The test can be considered a valid one since the construct is clearly defined as “English for

academic purposes” and relevant to the purpose of the test which intends to measure test

takers’ ability in constructing well developed academic essays.

• Authenticity

The test task is relevant to the characteristics of TLU domain. Test takers would encounter

similar tasks beyond performance on the test to language use TLU domain. They will be

required to write different academic papers during their academic studies. Therefore, the test

can be considered authentic.

• Interactiveness

A degree of involvement of test takers’ characteristics, especially their language ability,

can be expected because the test task prompt is general in nature. They will be required to

develop an essay through giving their opinions on a topic related to a general theme (see

Appendix A). However, as mentioned above, test takers’ topical knowledge and affective

schemata will not be affected.

 Academic Writing Assessment 6

• Impact

Test scores will not have an impact on the test takers because it is a diagnostic test, and

test results will not affect them. However, the test results will have an impact on the teacher

because they will help her identify what the course should focus on. As for washback, the test

will not have a washback on the test takers because there is no prior preparation for the test

needed. On the other hand, since it is a diagnostic test, the information that “washes back” to

the test takers will help them know their areas of strengths and weaknesses (Brown, 2004)

which can be a positive form of washback.

• Practicality

The test can be said to be very practical because the resources for developing and

administrating the test are accessible. In addition, the test is easy to administer and within

appropriate time constraints.

Test Operationalization
Test Blueprint (Test Structure)

Test Description and Purpose

The test will consist of a single task. Test takers will have to choose between three

different prompts to construct an essay. The purpose of the task is to measure test takers’

ability in constructing well developed essays that include an introductory paragraph with a

clearly stated thesis statement, body paragraphs with main ideas and supporting examples and

a conclusion that summarizes the main ideas. Furthermore, the task will measure test takers’

use of grammar, register, punctuations, transitions and spelling. The rationale for choosing

this single task is that it helps in making clear assumptions about students’ writing ability. In

other words, writing a five-paragraph essay, can clearly demonstrate test takers’ writing
 Academic Writing Assessment 7

quality in terms of different components like grammar, style, and their knowledge of easy

organization.

Setting and Time Allotment

The test will take place in the classroom where they will be taking their academic writing

class at the Garyonuis Language Center in the city of Benghazi in Libya. The time allotment

of the test is two hours which is the class session duration. Therefore, the test can be

classified as “a power test” (Bachman & Palmer, 1996, p. 51) because enough time is allowed

to accomplish the test task.

Test Instructions

The instructions will be given in English explicitly on the test paper. Test takers will be

asked to write a five-paragraph essay that includes an introduction, a body, and a conclusion

to answer ONE of the three questions provided (see Appendix A). In addition to the main

instructions, additional directions regarding how their response to the task should be are

stated on the test paper. They are clearly instructed to provide (a) an introduction that states a

claim or position statement, (b) three body paragraphs, each with different examples of

evidence to support their positions, an appropriate topic sentence for each of the three

supporting paragraphs, (c) a concluding paragraph that restates your claim and summarizes

the main points as well as their opinions, and (d) accurate grammar, transitions, punctuations

and correct spelling.

Input Characteristics

The input channel is visual, i.e. written. It will be presented in the target language. The

input is given as a prompt, relatively medium in length, in order to elicit an extended

 Academic Writing Assessment 8

production response. It provides a context for the writing task and specifies a number of

criteria that the test takers’ writing response should meet.

Expected Response

Students will write their answers in English. The channel of response is visual; extended

writing. Responses are supposed to be lengthy. Organizational and pragmatic characteristics

of responses are likely to be simple.

Scoring Method

The scoring method that will be followed is the criterion-referenced. Test takers’ writing

responses will be scored from 0-5 on separate criterion-referenced scale for range and

accuracy of use of syntax, essay organization, cohesion, register, punctuations and spelling.

Test takers will be provided with an evaluation form that gives in details how their

performance was (see Appendix B). The reason for doing that is to make them aware of what

they need to work on.

Test Analysis

Test Results

Test takers will be provided with an evaluation form (see Appendix B) that provides them

with separate scores for each portion of the test and a total score. The highest score is five

and interpreted as ‘excellent’; whereas 1 is the lowest and interpreted as ‘poor’, followed by 0

which stands for ‘missing’. Each element being rated is worth 5 points; the total of the points

is 115. In addition to the evaluation form, marginal and end-of-essay comments will be

provided on test takers’ pieces of writings in order to ensure beneficial washback to the

students.
 Academic Writing Assessment 9

Generally speaking, test takers’ responses to the test task are expected to have the

following characteristics. Due to their similar level of test takers’ language ability as well as

their similar background of general English writing, test takers’ scores are expected to be

close to each other. However, it should be mentioned that there would certainly be slight

variation because every individual has his/her strengthens and weaknesses in some areas of

language, like the use of register, grammar and punctuations. Furthermore, it is expected that

the essay organization would probably fall under the categorization “needs improvement”

(see Appendix B) for most test takers because of the different writing style between English

and Arabic. Test takers are more likely to be affected by their first language in this respect.

Not having piloted the test can be considered as a limitation to this test design proposal.

Piloting the test would assist in making modifications and improvements to the test task and

the test instructions before conducting the actual test. I would love to have the opportunity to

pilot the test in the near future to put these ideas into practice.
 Academic Writing Assessment 10

References

Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford, UK: Oxford

University Press.

Brown, H. D. (2004). Language assessment: principles and classroom practices. White

Plains, New York: Pearson Education, Inc.

Weigle, S. C. (2002). Assessing writing. Cambridge, UK: Cambridge University Press.

 Academic Writing Assessment 11

Appendix A

A Sample of Writing Assessment

Student's Name ________________________________________________

Topic chosen (check one of the boxes): A □

B□
C□

1. Write a five-paragraph essay (introduction, body, conclusion) answers the

following question:

A. Do you agree or disagree with the following statement? Parents are the best
teachers. Use specific reasons and examples to support your answer.
B. Some people believe that university students should be required to attend
classes. Others believe that going to classes should be optional for students.
Which point of view do you agree with? Use specific reasons and details to
explain your answer.
C. Some people believe that the Earth is being harmed (damaged) by human
activity. Others feel that human activity makes the Earth a better place to live.
What is your opinion? Use specific reasons and examples to support your
answer.

* Make sure to provide…

a) an introduction that states a claim, or position statement,

 Academic Writing Assessment 12

b) three body paragraphs, each with different examples of evidence to

support your position,

c) an appropriate topic sentence for each of the three supporting

paragraphs,

d) a concluding paragraph that restates your claim and summarizes the

main points as well as your opinion,

e) accurate grammar, transitions, punctuations and correct spelling.

Appendix B
A Sample of the Evaluation Form for the Writing Assessment

Writing Test Evaluation Form

___________ :Name: ______________________________________ Final Grade

Excellent 4 = Very Good 3 = Good 2 = Needs Improvement 1 = Poor 0==5

Missing
Content Total

1. Introduction
Hook____
Relevant to topic____
Thesis statement____

2. Body
Paragraph 1
Topic Sentence____
Example____
Support____
Concluding sentence or transition____
Paragraph 2
Topic Sentence____
Example____
Support____
Concluding sentence or transition____
Paragraph 3
Topic Sentence____
Example____
Support____
Concluding sentence____
 Academic Writing Assessment 13

:Conclusion .3
Transition____
Restate claim, summarize main points____
Opinion____
:Overall .4
Coherence____
Transition____
Grammar____
Register____
Punctuations____
Spelling____

.…………………………………………………………………………………… :Comments