ABRACADABRA∗

Evaluation of a Balanced Text and Word-

level Reading Intervention – Winter 2006

Preliminary Findings, June 2006

∗
The authors gratefully acknowledge the financial contributions towards the research and development of
ABRACADABRA from the following agencies; Industry Canada, The Chawkers Foundation, Inukshuk, and
the Fonds Québécois de la Recherche sur la Société et la Culture. This project would not have been
possible without this support. For further information or to view the software please visit
http://grover.concordia.ca/abra/version1/
 Executive Summary

ABRACADABRA (http://grover.concordia.ca/ABRA/version1/abracadabra.html) is a web-

based application that implements a balanced reading curriculum in a digital
environment, allowing student to develop word, text, fluency, and eventually writing
skills. In its entirety, this application will provide an opportunity to enrich students’
reading as well as provide full support for those who teach these students.

Created by the Centre for the Study of Learning and Performance (CSLP),
ABRACADABRA has experienced an iterative process of research and development
that proves its efficacy, ensures its usability and defines its practicality. A
multidisciplinary team consisting of researchers, practitioners, school administrators,
instructional designers and developers are involved in all phases of this process in order
to accomplish our goal of creating this evidence-based resource for students and
teachers.

This report contains the exciting preliminary results from our Winter 2006 study,
examining the effectiveness of the tool as a resource for teaching reading. Small groups
of students were randomly assigned and worked with trained facilitators on combinations
of activities. The comparison group did not work on the ABRACADABRA program but
stayed in the classroom and received their regularly scheduled language arts instruction.
Significant effects in areas of letter-sound knowledge and blending demonstrate
that this tool can give students building blocks to become good readers.
Furthermore, significant effects in students’ listening comprehension give us
strong cause to continue our research in the power of technology as a tool for
learning. Positive trends in other reading sub-skills also demonstrate that we are on the
right track to developing an effective literacy tool for teachers and students.

During this same period a sub-study was conducted in two kindergarten classrooms. In
order to determine if ABRACADABRA can be used as an early intervention, small
groups of students worked with a facilitator on the ABRACADABRA activities appropriate
for this age group. This intervention reported significant effects in letter-sound
knowledge and blending vowel/consonant words. On a combination of all reading
tasks the intervention groups read nearly two times as many words as those who
did not partake in the sessions.

The results of this preliminary analysis give us strong reason to continue practicing this
process of research and development so that we may meet one of our primary
objectives: To transfer scientific knowledge to the educational community in the form of
practical tools. A more thorough analysis of the data will take place over the next several
months, as we feed such information into the design of ABRACADABRA, ultimately
refining its effectiveness as a literacy resource. These encouraging findings were
made possible by our partners in the community who allowed us access to their
schools, their teachers, and of course, their students.

2
ABRACADABRA (http://grover.concordia.ca/ABRA/version1/abracadabra.html) is a
web-based application that implements a balanced reading curriculum in a digital
environment, allowing students to develop word, text, fluency, and eventually writing
skills. In its entirety, this application will provide an opportunity to enrich students’
reading as well as provide full support for those who teach these students. Developed
by the Centre for the Study of Learning and Performance, this web-based tool, is a work
in progress and only slightly over two years old. It has undergone a cyclical process of
research, development, formative evaluation and further research to reach its present
stage. This report highlights the preliminary results from our Winter 2006
research, in which two of our partner school boards participated, allowing us
access to 190 students. Our aim is to continue to work with our partners, and others,
in order to continue to refine this tool, aid in its integration into the classroom, and learn
more about how to help our students learn how to read.

Results from this year’s study clearly show strong effects and the positive trends
define ABRACADABRA as a resource that can help children learn the skills they
need to have to become strong readers.

Our primary goal is to evaluate the effectiveness of the ABRACADABRA application as

a tool that will help build literacy skills for emerging readers. Our secondary objectives
are many and will expand as the tool matures. These include answering questions such
as how this tool can help struggling readers, how it can help second language learners
and eventually how teachers can use ABRACADABRA as part of their reading
program.

History
The first ABRACADABRA prototype was released in February of 2003. Since then,
different versions of ABRACADABRA have undergone a series of evaluations in order
to gain feedback on its use. Some of these experiences have been a pilot study during
the summer of 2003 at the Writer’s Workshop (SWLSB) and Sinclair Laird (EMSB).
Exercising our philosophy of using field experiences to refine our tools we developed
more activities and created more stories in order to develop a larger intervention and
test the effects of ABRACADABRA on children’s reading development. We, therefore,
designed a study in which small groups (4) of students were guided by trained
facilitators to work on various activities in the site.

Description of the 2004/2005 ABRACADABRA Pilot Study

Small groups of students (4) from 2 schools, each randomly assigned to Intervention R
(Rime) or SP (Synthetic Phonics), were removed from their class during language arts
period to work on the ABRACADABRA program. The 2 intervention syllabuses were
identical in terms of time (20 minutes 4 times per week), group size, and the children
were of equivalent initial ability (across all classes). Each group also received identical
comprehension, fluency tasks but differed in subtle ways on aspects of alphabetics
exercises. A comparison group stayed in the class and received regular classroom
teaching.

3
Results of the 2004/2005 ABRACADABRA Study

Preliminary results demonstrated moderate effect sizes in the following areas:

Effect Effect Definition ABRACADABRA

Size
Decoding Skills Moderate These are skills such as segmenting a All children
syllable into its sounds (e.g. ‘c’ ‘a’-‘t’) improved here but
or sounding out unfamiliar words. children taught
explicit decoding
skills (Intervention
SP) improved in
them to a modest
degree
Processing Speed Moderate These are ‘pure’ processing tasks Only children in the
such as naming objects or letters as interventions
fast as possible. Efficiency here is improved here.
closely linked to fluency in reading and Children in
spelling. Intervention R did
particularly well!
Reading vocabulary Moderate These are skills such as isolated word All children
reading and reading isolated words for improved here.
meaning. However, within
this there was a
strong effect of a
test of reading
words for meaning
for Interventions R
and SP.
Text-level Small These are skill such as independently Children taught
comprehension reading and then responding to explicit decoding
questions on a sentence or a passage skills (Intervention
SP) showed
strongest effects
here.

Results from this study led us to improve both the design of the study and the design of
the ABRACADABRA application. Multiple additions and changes were made to the
software before returning to the field to conduct our Winter 2006 study.

4
 Description of the Winter 2006 ABRACADABRA Study
As always, our primary goal is to evaluate the effectiveness this tool. Improvements
were made based on user feedback, focus groups and results of the 2004/2005 study.
The design for the second round of research was similar to the 2004/2005 pilot. Our
secondary objectives were to analyze the effectiveness of the software on children who
are demonstrating difficulties with concentration, and understand the effects of
introducing sound and letter recognition to kindergarten children.

Main Study: Cycle 1, Level 1

The main study included 145 participants from 6 schools. Small groups (4) of cycle 1,
level 1 students, each assigned to intervention SP or R, were removed from their class
to work on the ABRACADABRA program with a trained facilitator. The 2 intervention
syllabuses were identical in terms of time (20 minutes 4 times per week), group size (4
students), and the children were of equivalent initial ability (across all classes). They
received identical comprehension and fluency tasks, but differed in subtle ways on
aspects of the alphabetics. Once again, it should be noted that the approaches in
intervention SP and intervention R are both proven effective in improving reading skills.

Intervention SP: Along with comprehension, fluency and writing tasks, these groups
of students focused on blending and segmenting activities.

Intervention R: Along with comprehension, fluency and writing tasks, these groups
of students focused more on a progression from sentences to words
to rhymes.

Comparison group: A group acted as unseen control for the duration of the study.

Measurements: WRAT, Arithmetic, Woodcock-Johnson Word Attack (nonsense words),

Reading Fluency, Phonemic Segmentation, Spelling, Blending Words, Rapid Object
Naming, Rapid Digit Naming, Test of Everyday Attention for Children (TEA-ch), GRADE
– Sentence Comprehension, Listening Comprehension, Word Reading, Word Meaning
(This measurement was group administered)

Preliminary Results from the Main Study

Our overall preliminary findings are extremely positive. The process of analysis and
redesign based on our pilot study and our work with our educational partners including
observations and focus groups have contributed to further success in the effectiveness
of ABRACADABRA.

5
The following graphs show the preliminary results from this study.

**Please note that this preliminary report will be followed by a more in-depth analysis of
the data. This lengthy project will begin in the fall of 2006.

24.5
24
23.5
23
Phoneme
22.5
Rime
22
Control
21.5
21
20.5
Pre-test Post-test

Letter Sound Knowledge

Students who participated in the ABRACADABRA program displayed more letter-sound

knowledge than those who did not participate. The results show a significant effect for the Rime
group.

110
108
106
104
102 Phoneme
100 Rime
98 Control
96
94
92
Pre-test Post-test

Blending

Students in the ABRACADABRA program were able to blend sounds better than those who did
not work on the program. Particularly strong effects are seen in the Phoneme group.

6
 6

4
Phoneme
3
Rime
2 Control
1

0
Pre-test Post-test

Listening Comprehension

Both groups of students who used the ABRACADABRA program showed strengths in listening
comprehension when compared to the students who were tested but did not work on
ABRACADABRA. The Rime group showed particularly strong effects.

92

91

90
Phoneme
89
Rime
88 Control
87

86
Post-test

Word Reading and Comprehension

These results show that the Phoneme group in the ABRACADABRA program demonstrated very
strong word reading and comprehension skills. The Rime group was not as strong, but still
outperformed those who did not participate in the sessions.

7
Kindergarten Intervention

The intervention period consisted of short highly structured literacy activities using the
ABRACADABRA software. Students in two kindergarten classes rotated through the
computer center in groups of 4 or 5 and worked with the guidance of the researcher. All
students had the opportunity to work with ABRA software 3 times a week.

Intervention A These students had 40 (20 weeks, 15 minutes per session, 2 x per
week) sessions of phoneme-based activities in addition to alphabet
exposure.

Intervention B These students had 40 (20 weeks, 15 minutes per session, 2 x per
week) sessions of syllable and onset-rime base activities in addition
to alphabet exposure.

Comparison group: A third group acted as unseen control for the duration of the study.

Measurements: Letter-sound Knowledge, Blending and Segmenting, Common Unit

Phonological Task, Nonsense Word Reading, No Rime Word, Peabody Picture
Vocabulary

Preliminary Results from the Kindergarten Study

The overall results from this study are extremely positive, demonstrating that
ABRACADABRA has strong potential as a resource for early reading intervention.

Comparison of 3 intervention Groups

on Letter Sound Knowledge
Identified Correcly

25
Number of Letters

20
15
10
5
0
Alpha Alpha Alpha
Control Phoneme Rime
Intervention Groups

Both intervention groups outperformed the control group on correctly identifying letters and their
corresponding sounds.

8
 Comparison of 3 intervention groups
on Blending VC words

words correctly
Number of VC
8
6

blended
4
2
0
Control Phoneme Rime
Intervervention Groups

Both intervention groups outperformed the control group on blending VC (ex. i-ce) words. The
Phoneme intervention experienced the greatest gains.

8
7
6
5
4 Intervention
3
Control
2
1
0
Average
Number of
Words Read

On a combination of all reading tasks intervention groups read nearly two times as many words
as those who did not partake in the sessions.

9
Preliminary Conclusions

These combined results show exceptional promise in the ABRACADABRA application

as an effective tool for students in Kindergarten and Cycle 1, Level 1.

We have now demonstrated statistically significant (and, particularly in the kindergarten

study), very large effects of the ABRACADABRA reading intervention on children’s
literacy and closely related skills. This effect is evident despite a relatively brief
intervention of a total of 13 hours exposure per child (cycle 1 level 1 study, 10 hours for
the Kindergarten study). These patterns could therefore represent a major contribution to
the development of these children. The longer-term impacts of these improvements on
children’s development should, we feel, now be investigated to evaluate this.

The results from our pilot study (2004/2005) demonstrated areas of development that
were needed to create a more effective tool. We thus, set out to improve the alphabetics
module by creating more activities, more levels and more content for the students. The
results of these efforts are evident in the results from this year’s study.

We will continue to examine the results of this study and will feed them into the
development of the ABRACADABRA application. Two areas of weakness that exist
within the system are the Fluency and the Comprehension activities. We will address
these components of ABRACADABRA and improve their content. As well, we will be
developing the Teacher Environment where a teacher can access student reports and
professional development material.

Our medium-term goal is to continue this close process of research and development so
that more students and more teachers can benefit from a successful and proven reading
intervention like ABRACADABRA.

Team Members

Philip C. Abrami
Dr. Robert Savage
Erin Comasky
Denise Silverstone
Geoffrey Hipps
Director of the Centre for the Study of Learning and
Performance; ABRACADABRA Project Lead
Principal Investigator; Professor, McGill University
Researcher for the Kindergarten Study, McGill University
Research Coordinator, Centre for the Study of Learning
and Performance
Literacy Coordinator, Centre for the Study of Learning and
Performance

Partners

Sir Wilfrid Laurier School Board

English Montreal School Board

10