Jenna Hutto Kayse Morris Niki Taylor

Program Evaluation

FRIT 8425 Fall 2012 Georgia Southern University December 3, 2012

Executive Summary Introduction to report Focus of the evaluation Brief overview of evaluation plan and procedures Presentation of evaluation results Conclusions and recommendations Appendices

Page 3 Page 4 Page 6 Page 8 Page 9 Page 15 Page 18

The purpose of this evaluation is to determine the impact of the program Read Naturally, and its role in aiding student achievement based on oral reading fluency and comprehension. This program evaluation was conducted at Bacon County Elementary school with ten students from each third, fourth, and fifth grades. Currently this program is offered to students who score below the twenty fifth percentile based on Lexile scores. In determining the effectiveness, the following evaluation questions were posed:

Do students who use Read Naturally show significant gains in reading fluency and comprehension?

Do teachers who implement Read Naturally see improvement in students on an academic level?

Data sources were used from the Read Naturally program at Bacon County Elementary in Alma, Ga. These reports included students weekly probes based on achievement, areas of weakness, and grade level. We evaluated CRCT scores and instructional folders to review approaches for response to interventions. Students who were placed in the lower twenty fifth percentile were nominated for tier three interventions and the Read Naturally, program based on success rates. Students final recommendations for promotion were reviewed before making a final decision on probing subjects. Questionnaires were distributed to classroom teachers and parents of students who participated in the program.

Since the introduction of Read Naturally, in the fall of 2009, at Bacon Country Elementary, the program has demonstrated success based on individual student achievement. This program has helped many students enhance their reading capabilities. Read Naturally, focuses on the five major points of reading: vocabulary, fluency, phonics, phonemic awareness, and comprehension.

Read Naturally's structured intervention programs combine teacher modeling, repeated reading, and progress monitoring three strategies that research has shown are effective in improving students' reading proficiency. Using audio support and graphs of their progress, students work with high-interest material at their skill level. (http://www.readnaturally.com)

Read Naturally, also focuses on primary steps when assessing and aiding in better reading. 1.) 2.) 3.) 4.) 5.) 6.) 7.) Select a story Key Words Prediction Cold Timing Graph the cold timing score Read along Practice 4

8.) 9.) 10.) 11.) 12.)

Answer the questions Pass Graph the hot timing score Retell Practice and pass the words list

The goal of Read Naturally, since 1991, is to help students become better readers. The program is designed to help students increase reading fluency by using three main elements, teacher modeling, repeated reading, and progress monitoring.

The program has been implemented into the regular school day schedule in 30 minute sessions for 14 weeks at Bacon County Elementary School. The students selected to participate in the program are students that receive a low score on the AimsWeb Screening or are currently in any tier of intervention. AimsWeb is a universal screening of oral reading fluency in which students read a passage to check for a students accuracy and determine their number of correctly words read per minute. Students on tier intervention were returning users of the program. Some of the students were new to the program Read Naturally.

Read Naturally is an individualized program. It allows students to progress at their own pace through different reading levels. Students participating the program become more independent and develop a feeling of ownership for their success. The program offers highinterest nonfiction stories that are age-appropriate depending on the reader.

The purpose of the evaluation of the program Read Naturally is to focus on reading accuracy, reading fluency, and reading comprehension. The following questions were used to guide the evaluation: 1. Is Read Naturally a suitable program for students?

2. Do students who use Read Naturally show significant gains in reading fluency and comprehension? 3. Do teachers who implement Read Naturally see improvement in students on an academic level?

To assess the impact of Read Naturally on students at Bacon County Elementary School, information was collected detailing student gains between cold timing scores and hot timing scores and the perceptions of the stakeholders over a 14 week period. The information was collected through student evaluations, student questionnaires, parent questionnaires, teacher questionnaires, and system data analysis.

The evaluators collected qualitative data by providing evaluations and questionnaires to students, parents, and teachers. The program system data analysis was collected to show quantitative data. The evaluators examined the data of 60 students, 10 third graders, 10 fourth graders, and 10 fifth graders. Again, some students were familiar with the program and some were new to it.

Data of the 60 students was used to determine the gains between the cold and hot timing scores. Periodic reports were collected for the 60 students. The reports presented the students goal while working with the Read Naturally program (Appendix A,B,C). Fluency details were also collected for each of the 60 students (Appendix A,B,C). Student evaluations were distributed to the 60 students from third, fourth, and fifth grade (Appendix F). Questionnaires were also distributed to the students (Appendix E) and their parents by the evaluators (Appendix D). The data collected in the program evaluation of Read Naturally was obtained from the 60 students at Bacon County Elementary School. 8

The counselor and RTI coordinator at our school analyze previous CRCT test scores and reading fluency scores. Also, they meet with teachers and discuss students that need an intervention in reading. After completing a list of which students meet the requirements for interventions, the intervention team sends home letters explaining that the students need help in the area of reading. The letter also explains they will be using Read Naturally, a computer based program, for 30 minutes a day. The child has to remain in the intervention until they meet the grade level goal. Below are the recommended goals for fluency for grades three through five.

Recommended Reading Fluency Rates in Connected Text

Grade 3rd 4th 5th Fall 79 90-99 105 Winter 84-93 98-112 110-118 Spring 100-114 105-118 118-128

Below are the charts that illustrate the information for ten students from each grade level. It depicts the starting date, ending date, starting WPM rate, ending WPM rate, stories read, and comprehension correct.

3rd Grade
Student 1 2 3 4 5 6 7 8 9 10 Starting Date 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 Ending Date 11/30/12 11/28/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 Starting Goal (WPM) 50 50 50 50 50 50 65 50 50 85 Ending Goal (WPM) 80 100 80 50 70 75 125 100 60 90 Stories Read 36 22 41 61 50 13 52 36 34 30 Comprehension Correct 76% 101% 87% 80% 83% 71% 82% 82% 70% 85%

After examining the third grade results, we noticed that 1 out of 10 students did not increase in reading fluency and that 3 out of 10 students do not meet the standard of 80% on comprehension correct. We recommended that teachers who are implementing Read Naturally pay attention to what the students are doing to maybe cause no increase in WPM or less percentage in comprehension. It may be due to the students not properly knowing how to use the program or making mistakes. This may be something that can be improved with the help of the teacher applying help or it may be that the student just does not understand and is missing a step prior to being placed on Read Naturally. Another problem was that most of the third grade students, with the exception of three, did not meet the recommended WPM goal for winter which is 84-93. However, looking at their starting WPM goal, they are well below grade level already. In the fall they should have been reading 79 words and most of them started at either 50 or 60. Therefore, this tells me some of students still improved a tremendous amount.

10

4th Grade
Student 1 2 3 4 5 6 7 8 9 10 Starting Date 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 Ending Date 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 Starting Goal (WPM) 65 85 80 65 85 80 85 50 65 75 Ending Goal (WPM) 90 115 100 100 100 120 130 90 100 79 Stories Read 93 33 30 86 72 41 52 30 92 52 Comprehension Correct 70% 84% 84% 79% 81% 81% 82% 83% 80% 78%

When examining the fourth grade results, all of the students increased in WPM and only three of the students did not reach the recommended goal of 80% in comprehension answered correctly. Once again, this could be a problem that can be helped by the teacher or may be that the student needs other intervention. However, most of the students in fourth grade did reach the recommended reading fluency for winter which was 98-112. This shows better improvement than third grade. Also, more of the students in fourth reached the 80% goal of comprehension. However, there was still one student who fell short of the WPM goal and the comprehension goal. This student may need a different type of reading intervention. Keep in mind some students to not cooperate and learn well on computers. Therefore, this student may need to be in some type of reading intervention class.

11

5th Grade
Student 1 2 3 4 5 6 7 8 9 10 Starting Date 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 8/15/11 Ending Date 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 11/30/12 Starting Goal (WPM) 50 130 120 75 100 70 85 75 80 80 Ending Goal (WPM) 120 140 130 110 100 110 120 110 115 80 Stories Read 30 68 22 44 77 75 106 140 57 29 Comprehension Correct 87% 77% 87% 84% 80% 90% 77% 84% 80% 77%

Fifth grade results look awesome. Eight out of the ten students started well below WPM for their grade level. However, after completing the program, eight out of the ten students were well above the WPM goal for winter which is 110-118. Also, only three out of the ten students had a comprehension that was below 80%. Several of the students had a large gain in WPM after using Read Naturally and maintained an 80% or above on comprehension questions correct. By the Spring fluency check, most of the fifth grade students should be reading at or above average recommended WPM, which is 118-128. Four of the students are already above the recommended WPM for Spring. The other six will have time to continue to increase their WPM in reading fluency and hopefully reach the grade level goal.

12

Stakeholder Perceptions After discussing the program with teachers, parents, students, and staff members, the overall information indicates that Read Naturally has had a positive impact on the students who participated. Some of the strengths they named were self-paced, easy to use/navigate, and computer correction. These were some of the most important strengths. They believe that Read Naturally being self-paced helps the students not feel intimated by others. Another great feature is that they read on their own so other students cannot hear them or worry about who is more advanced in the program. The parents really liked the fact that the program was offered during the school day and that students could use the program while at school without getting further behind on their actual classwork. They also liked that while working individually on computers, there was also a teacher present so if the students needed guidance. Another feature the parents really liked about Read Naturally is that it gave the students immediate feedback on the story and words read per minute. Also, the program allows them to read, reads to them, and allows them to read again. It is teaching them how to read the passage, not just letting them read it alone. Some features that the teachers and staff liked are the read- word and comprehension questions. If while reading a student comes to a word they do not know, they can simply click on the word and the program will tell them how to pronounce the word. Also, the comprehension questions are multiple choice and if the student gets the question wrong, it will not allow them to move on to the next question, it brings the story back up and allows them to go back in the story and find the correct answer. Even though they answered incorrectly and the question counts against them, they have to find the correct answer before they can move to

13

the next question or next step of the program. This teaches them skills of how to read back in the story to find an answer. The students were also asked what they thought about Read Naturally and how it has benefited them in reading. The answers speak for themselves. Some students said I can read better and I can read faster now. The students who really struggle in reading and want to increase their reading speed and accuracy were overjoyed working on the program that it helped them succeed. The only downfall was that some students who are on it very frequently said that it does get boring because you are doing the same thing each day even if you are reading different stories. Program limitations cited by this group were concerned with student ability and staffing concerns. More students could benefit from this program but without the staff to allow the students to participate in the Read Naturally program, some of the students are not chosen because they dont quite make the requirements for the program. Also, some students just do not have the ability to use the program and actually increase in words per minute (fluency) and comprehension. Overall, the program definitely has more positives than negatives. There are more strengths in the program data and the surveys from teachers, parents, and students than downfalls in the program. This program may not be perfect and may not help every student succeed in reading faster and more accurately, but it is a great start to help a child improve their reading skills. Read Naturally focuses on increasing fluency and comprehension and at Bacon County Elementary School it has done that in many ways. The program has more than proved its effectiveness to our staff, parents, and students.

14

Based upon the evaluation of the program data relating to student needs and gains of the current program Read Naturally, this evaluation study concludes:

Program Strengths: The teachers manual is well-organized with clear objectives linking research with practice, plus detailed information for proper implementation and effective use of the program. Students receive explanations as to why they perform various activitiescold timing, reading along, and practice readings. Progress monitoring drives instructional practice, helps with instructional decisions, and alerts teachers to reading problems. Comprehension questions and the retell step are included with every story, reminding students that their ultimate goal is to understand what they read. Students are instructionally engaged in the act of reading and their time on task is high. Students work in a self-directed manner, freeing the teacher to work with other students. Teacher assistance is required for the pass step. Timed reading and graphs of results can be very motivating for students. Most students increase in WPM over the time period while using Read Naturally.

15

Program Weaknesses: Lack of ample time and staff to productively implement Read Naturally as recommended. As seen in the results, it does not always work on every student, there are exceptions. Amount of comprehension questions answer correctly is not as high at least 80% for every child. Recommendations: Work with scheduling at the school to allow more time for students to work on Read Naturally each week. Work individually with the students who have not increased in WPM to see what they may be doing or what their problem may be. Try to help the students who are not increasing to at least 75% on comprehension questions. Therefore, for the most part, Read Naturally is a very effective program. As with every program though, it has its weaknesses. For the most part, the school needs to get a different schedule and try to incorporate the students using Read Naturally more during the week. Also, the school and teachers need to figure out what individual students are not increasing in WPM are struggling with. It may be that they do not fully understand the program or are having trouble navigating, especially if they were not taught properly. Also, the students who do not have a comprehension of at least 80%, which is 12 students out of the 30, may have some issues also. The staff members who are with the students when they are working on Read Naturally may have to work closer one-on-one and watch to see what the students are missing. As always,

16

there are some students who need further instruction or help than others. Therefore, the lower comprehension and no increase in WPM may be a problem that can also be improved.

17

Appendix A 3rd Grade Results Letter and Fluency Graph Student 1

18

19

Appendix B 4th Grade Results Letter and Fluency Graph Student 1

20

21

Appendix C 5th Grade Results Letter and Fluency Graph Student 1

22

23

Appendix D Parent Questionnaire

24

Appendix E Student Questionnaire 1.) Do you enjoy reading?

2.) Do you believe Read Naturally, has helped your reading accuracy and fluency?

3.) How often do you read?

4.) When do you read best?

5.) How has Read Naturally, helped you?

25

Appendix F Student Evaluation

26