Honestly, I really like it [Academic Advisory].

It reminds
me what I need to do and my grades are important.
-IFHS Student

Evaluation of
IFHS Academic
Advisory Program
Prepared for Idaho Falls High School
AAD Committee

Wendi Straub
EdTech 505, Fall 2014

SUMMARY
This report evaluates Idaho Falls High Schools Academic Advisory Days (AAD) program
implemented three years ago to minimize loss of class credits by providing additional support
during the school week to any students with an F or D. Additional goals include providing an
academic advisor to each student, to build in time during the school week for extra help or makeup work and to motivate student performance. Students with low grades are identified and
assigned for additional help with their classroom teachers up to three 25 minute Monday sessions
per month. Concerns about the programs effectiveness and efficiency led to the development of
a voluntary AAD committee and this evaluation.
Program effectiveness was assessed by tracking grade changes before and after AAD
interventions and at the end of the trimester for 100 randomly selected students entitled to AAD
services at midterms. The shared document used to assign students and monitor student
attendance was also studied to ensure that measurements were restricted to students that actually
participated in AAD. In addition, 113 randomly selected students and about third of the teachers
completed surveys asking about their perceptions of the programs value and experiences.
Data collected showed some positive correlation between the AAD program and better grades
and fewer Fs, but the overall effectiveness of the program is called in to question by low
participation due more often to teacher error than student resistance. Although most students had
better grades after AAD, almost a third still did not receive credit despite this intervention,
suggesting that the impact is less than desired. This is coupled with an unexpected break down in
the assignment process by advisory teachers. A significant percentage of students were not
assigned to AAD despite their need and did not receive program services. As both the
effectiveness and efficiency of the program are hindering program goals, significant reworking
of the program is recommended.
DESCRIPTION OF THE ACADEMIC ADVISORY PROGRAM
This report evaluates the Academic Advisory Days (AAD) program at Idaho Falls High School
(IFHS), now in its third year. All students at IFHS are randomly assigned to one advisory teacher
as they matriculate. Each teacher monitors weekly grades for ~25 students, assigns students to
AAD, and assists seniors with their graduation project. The primary focus of the current AAD
program is to provide additional instructional support to students at risk for failure in their
classes because loss of credit is associated with lower graduation success. Any student with a
grade under 70% is assigned to an extra 25 minute class up to three times per month. Students
with passing scores are released for a longer lunch during this time as an incentive to maintain C
or better grades.
From its inception, the AAD program has been vulnerable to needs of other programs. In 2011, a
temporary committee of teachers and administrators, charged with finding a way to advise

seniors on their Senior Project without additional pay to staff, recognized an opportunity to
provide academic advising during the regular school week. Modeled on a neighbor school, AAD
originally occurred on Tuesdays and Thursdays, but was reduced to one day per week after its
first year to make time for weekly professional development on district literacy goals. It was
moved from Thursday to Monday this year to avoid three bell schedules per week as that
complicates arrangements for buses, students attending classes in other locations, etc.
The program was implemented in fall 2012 with one two-hour training session for teachers
during the opening professional development meetings. Continued training has been as needed
for new teachers and as teachers requested. No formative assessments have been conducted on
the AAD program and although progress was informally monitored by the previous
administration, no concrete data had been aggregated.
Program Objectives:
The AAD program revolves around three philosophical ideas: 1) students benefit from additional
instructional time for courses in which they are underperforming; 2) at risk students are best
identified by teachers who review students total academic schedules on a weekly basis; and 3)
all students are better served by a relationship with one advisory teacher throughout high school.
The goals are

To reduce the number of Fs and Ds so fewer students lose class credits towards graduation.
To provide mandatory additional instructional time for underperforming students during the
regular school schedule
To motivate students to maintain a 70% or higher in classes
To provide time for students to make-up work from absences
To provide all students with an academic advisor who reviews students progress regularly
and guides senior project

Program Components:
Students are assigned to AAD in a shared Google spreadsheet on each Friday prior to an AAD
Monday. After reviewing PowerSchool grade reports for their advisory students on relevant
Fridays, teachers assign students to Monday AAD. To minimize credit loss, students with any Fs
are assigned to the class teacher with whom they have the highest F; students without Fs are
assigned to their lowest D (minimizes the risk of the lower grade becoming an F). The AAD
spreadsheet is opened on Friday from 5am 5pm and coupled with a PowerSchool grade freeze
to avoid grade changes that affect AAD assignments.
Students report to their advisory teacher every Monday for attendance. On AAD days, students
are given grade reports and AAD assignments and released for either longer lunch if no grades
below a C or to report to their assigned classroom teacher. The remaining 20-25 minutes are at
the discretion of individual teachers to help their assigned students, plus any students who
choose extra time/help despite acceptable grades. After students depart AAD, teachers enter

attendance on the shared spreadsheet. Administrators track attendance, meet with students who
skipped AAD and decree Saturday school when warranted.
EVALUATION METHOD
This evaluation employed a Decision-Making Model with goal-based summative elements
because the recently formed AAD committee is primarily interested in whether the program is
meeting its goal to positively influence the grades of under-performing students. Secondary
concerns deal with the processes of identifying and assigning students appropriately by
advisory teachers, attitudes toward the program, and student attendance in advisory. If indeed
the interventions are having some impact, then the secondary concerns may relate to program
improvements. If the program is not having any impact, the question becomes whether there
is a correlation with systematic deficiencies in the process.
Participants:
IFHS is a four year high school with ~1200 students. According to the IFHS 2014 Executive
Summary, approximately 15% of IFHS students are provided some type of academic
accommodation. Roughly 30% of the student body had one or more Ds or Fs at midterms.

For tracking advisory assignments, attendance and grade improvement, the target population
was the subset of students with one or more low grades. 100 students were randomly selected
from the midterm PowerSchool Grade Report using Excel RANDBETWEEN and tracked for
AAD assignment and follow-up grades.

For student surveys, existing advisory groups were used. New students are randomly
assigned by a computer program to advisors, with each teacher having 4-6 students from
each grade level. Although the advisory groups were selected for convenience, there is no
correlation between the teachers willingness to help and the students assigned. 113 students
from all four grades (23-26% from each) responded to the survey

All certified staff (60+) full-time at IFHS serve as advisory teachers and may have students
assigned to them during AAD. The AAD Committee is an informal volunteer group of teachers
directed by one vice-principal. Approximately 18 teachers are currently involved in the AAD
committee. All teachers were invited to participate in the survey, but only 23 responded.
Procedures:
The evaluation formally began following an AAD committee meeting in October to collect
generate ideas about the current program and its future. Informal feedback from teachers,
students and administrators had already suggested that the AAD process is inefficient,
undervalued and likely having little impact on students. Information collected from this meeting
and initial meeting with the vice-principal helped define the goals of the program and informed
the design of this evaluation.
To assess whether AAD assignment is positively correlated with fewer Ds and Fs, 100
randomly selected students with at least one low grade were tracked for grade changes, AAD

assignment and AAD attendance during the first trimester. Two goals, providing instructional
time during the scheduled day and providing all students with an academic advisor were met at
the outset of the program as verified by the school handbook for the former, and the registrar for
the latter. Student motivation to maintain grades above C and their general perceptions about
AAD were assessed through a student survey. Finally, teachers perceptions about the AAD
process and desire for training were gauged through a teacher survey.
The evaluation was conducted with staff knowledge, but likely had a negligible effect on
program processes as there were many competing demands this fall including a completely new
administration and re-accreditation. The AAD committee that only met on one occasion, student
surveys to five teachers on a single Monday, and teacher surveys open for two weeks at the end
of the trimester were the only reminders that an evaluation was occurring.
Data Sources:
Data was collected from four primary sources: PowerSchool grade reports, staff shared AAD
Google spreadsheet, student surveys and teacher surveys. To assess participant performance and
program use, PowerSchool grade reports for selected students were compared before and after
the October 13 AAD (roughly mid-term) and the November 17 AAD and cross-referenced with
the corresponding AAD Google spreadsheets. Students who did not attend or for whom
attendance was not clear were discarded from the data set. Similarly, the total number of IFHS
students failing classes at the midterm was compared to the number of IFHS students failing at
the end of the trimester and crossreferenced with the student sample. Separate attitudinal
surveys were administered to students and certified staff. Because student access and use of
technology is not evenly widespread, a paper-pencil survey was administered on November 3,
2014 in five advisory classrooms to assess student attitudes and perceptions of the AAD
program. 113 student responses were entered by a student aide into Google forms for analysis.
Teachers were sent a different survey via a Google form link in an email on the same date and
then a reminder one week later. The teacher survey was open for two weeks and 23 teachers from
across disciplines responded; however, 7 were from the Science Department (same as the
evaluator).

Grade Changes with AAD on 10/13

15
10
Grade Change (%)

5
0

10

12

Students 1-47 who DID attend AAD

RESULTS

Before and After Grade Comparison:

Only 47 students of the 100 randomly selected were confirmed participants in the AAD
program the following Monday. 29 students were not correctly assigned by teachers, 13 were
reported as not attending and 11 had no attendance data recorded by their teacher. 72.3% of the
students who attended AAD had some grade improvement by the following week and 48.9%
raised their grade to passing for the AAD assigned class.

Grade Changes in October

Mean
Median
Mode
SD

7.64%
6%
7%
15.8724

Grade Changes in November

Mean
Median
Mode
SD

6.90%
7%
8%
9.06277

Grade Changes with AAD on 11/17

40
30
Grade Change (%) 20
10
0

10

12

Students 1-43 who attended AAD

In November, 43 of the original 100

students were confirmed attending AAD. 16 students were not assigned by the teacher, 6
dropped, 6 were absent and 8 students had unclear attendance. Finally, 23 students no longer had
any poor grades. 76.7% of students raised their grade following the November AAD.
However, 28.8% of students still failed at least one of classes for which they attended
AAD. 8% of the selected 100 failed two or more classes.
School Wide Fs at Midterm Versus End of Trimester:

On October 10, 237 students were at risks of losing credit in one or more classes. On November
24th, 195 students had lost credit in one or more classes. 351 total course credits were lost due to
grade failures or absences in the first trimester.
Student Survey Results: (See Appendix A for survey)
Only 47% of 113 students surveyed had been assigned to AAD this year. 88% of those students
reported attending AAD with their assigned classroom teacher. Students who attended AAD
agreed that AAD helped them make up work and improve their grade, but did not agree that they
learned more. Student were also less likely to report that AAD provided more personal help from
their teacher. Overall, students felt that AAD is helpful to students.
Student Survey Responses (x-axes = 1 strongly disagree 5 strongly agree, y-axes = respondents)
Academic Advisory helped me
learn more about a topic.

Academic advisory helped me

make up work I was missing.

Academic advisory helped me

improve my grade.

I received more personal help

my teacher during academic
advisory.

from

Overall, I think academic

advisory is helpful to students.
(All students polled).

The student survey also looked at motivation for maintaining higher grades. All surveyed
students were asked to rate whether each of the following was a main reason for I try to keep
my grades up because. 88% of students reported that caring about their grades was a major
reason (4 or 5) for keeping their grades up, 72% chose longer Monday lunches and 64% chose
avoiding AAD as major motivators.
In the open comments section, 8 students responded favorably with I like it or Its useful, 3
students made recommendations for improvement Teachers should be prepared to give students
work that they will do to help make their grade better and 4 students responded negatively. (See
Appendix B for student comments)
Teacher Survey Results: (See Appendix C for teacher survey)
Only 23 teachers out of ~60 completed the AAD survey. Teachers surveyed overwhelmingly
reported that AAD does not provide enough time for learning. Many felt that AAD is not valued
by students. More teachers agree that AAD helps students improve their grades or make up
missing work, and teachers were split on whether lack of consistency between teachers hinders

the AAD process or not, and if the process promotes extrinsic over intrinsic motivation. Finally,
teachers were ambivalent about whether faculty valued the AAD program.
Teacher Survey Responses (x-axes = 1 strongly disagree 5 strongly agree, y-axes = respondents)
Academic Advisory helps
students improve their
grades.

Academic Advisory helps

students make up missing
work.

Academic Advisory helps

students learn more.

Academic Advisory does not

provide enough time for
learning.

Academic Advisory reinforces

motivation (reward) over
intrinsic motivation (personal
satisfaction).
Lack of consistency between
teachers hinders the advisory
process

extrinsic

Academic Advisory is not

valued by students.

Academic Advisory is not

valued by staff.

Teachers were also asked about training needs. 80-90% of teachers surveyed reported that they
were comfortable with the assignment process and do not desire further training. Some were
slightly more interested in training on how to help struggling students.
Teacher Survey Responses (x-axes = 1 strongly disagree 5 strongly agree, y-axes = respondents)
I have difficulty entering
student assignments into the
shared google doc.

I have difficulty determining

where students should be
assigned for academic
advisory.

I would benefit from more

training on how to assign
students.

I would benefit from more

training on how to help
struggling students.

DISCUSSION
A summary goal of the AAD program is to reduce the loss of class credits by providing
additional support during the school week to any students with an F or D. Ancillary goals include
motivating students to maintain C or better grades, scheduling time within the week for make-up

work or additional class time for any student, and providing another adult advisor to students.
Overall, the program is positively associated with some grade improvement (6-7% on average)
and perceived as helpful on some level by both teachers and students, though not strongly.
However, the processes to identify and to assign students for this mandatory help are fraught
with issues and both groups indicate that time constraints affect the programs utility.
To evaluate the effectiveness of the AAD program on reducing loss of credit, 100 students with
at least one grade below 70% were randomly selected and tracked for grade progress from
October 10th until the end of the trimester. Specifically, the class for which they should be
assigned AAD was identified, their AAD assignment and attendance were traced and their preAAD and post-AAD grades were compared for two AAD opportunities. For both times, over
72% of students who attended AAD experienced a grade improvement. After the October AAD,
almost half of the students moved to a C or better; but 17% of those were assigned to a different
class in November. However, despite the positive point gains following AAD, nearly one-third
(28.8%) of students who attended AAD still failed the class and lost credit and slightly less than
one in three of those students failed at least one other class.
Similarly, when looking at school wide credit loss, the number of Fs around mid-trimester in
October were compared to end of trimester grades in November. Less than one in five Fs were
recovered between midterm and final for the school. 351 total course credits were lost due to
grade failures or absences in the first trimester and195 IFHS students (~16%) lost one or more
class credits. Both AAD student data and school wide data suggest that AAD attendance is
positively correlated with some grade gains and fewer lost credits, but the impact of the program
is not satisfactory in its current form. A new GAP analysis is recommended with exploration into
other potentially effective strategies and interventions with struggling students.
Program efficiency is definitely in need of review. Twenty-nine of the 100 students in October
and sixteen students in November were not actually assigned to AAD despite their need. Nine of
those students were not assigned either time. This suggests that the processes for identifying and
assigning students are not working. Likewise, attendance data was missing from 6-11 students
each time. In contrast, only 10 or 11 of students correctly assigned with recorded attendance
information missed AAD and a portion of those were actually absent from school. These are
advisory teacher responsibilities, yet surveyed teachers reported high comfort with procedures
and low desire for additional training. Although this seems contradictory, it must be considered
that teachers who are struggling with the shared Google process may also be less likely to
respond to a digital survey. The evaluator did not have access to AAD assignments because no
grade is given, so could not verify whether unassigned students had common advisory teachers.
Conversely, the survey was not mandatory, less than 40% teachers responded, and nonparticipation should not be assumed the result of lower skills. When teachers do not perform
these tasks well, students miss advisory opportunities, do not attend the class they may need the
most help in, or go to an advisory with a teacher not prepared for the students arrival. Some
surveyed teachers felt that the lack of consistency in the program was hindering effectiveness.

10

Ultimately, deficiencies in these processes impact most of the programs goals. Administration
could use their access to determine whether particular teachers are prone to these errors and
investigate the specific causes. Perhaps some teachers need more specific supports and/or
additional training, and apparently all teachers need clarification about common policies.
Likewise, surveyed teachers almost unanimously agree that AAD time is inadequate to meet
student needs and in the few open comments submitted by students and teachers, additional time
is also mentioned (See Appendix D for teacher comments). Only 75 minutes of dedicated time is
provided to the AAD program each month. Again, in its current incarnation, AAD is not being
adequately supported for effective achievement of its goals.
Survey responses from students indicate that students are motivated by both intrinsic and
extrinsic factors, including those built into the AAD process. Most students try to keep their
grades up because they care about their schooling (intrinsic), but also because they desire longer
lunches and wish to avoid academic advisory (extrinsic). A few are also motivated by other
external rewards or benefits inherent to learning. One can infer that the motivation goal is being
met without overly diminishing intrinsic values.
Finally, staff and students seem to be somewhat ambivalent about the program and its rewards.
Neither group is clearly or enthusiastically supportive, nor especially opposed, except for a
handful of students in open response. Student surveys were mandatory and random, whereas
teacher surveys were not. This means student responses are likely more representative of their
group than the teacher sample. Regardless, AAD buy-in is still weak in both groups after three
years, possibly because other programs have infringed on program resources especially time.
At a minimum, proactive strategies to make AAD part of positive school culture for both staff
and students could foster enthusiasm and adoption in a more meaningful way.

11

PROJECT COST
Budget Description:
The following statement reflects services rendered to evaluate the IFHS Academic Advisory
Days program. The evaluator was responsible for conducting and or coordinating all activities of
the evaluation. The registrar provided five grade reports between October 10 and November 24,
2014 and one student aide entered student survey data to the digital database. The daily rate for
the evaluator is $500 per day or $65 per hour, non-negotiable.

IFHS AAD Evaluation Invoice

December 5, 2014

Personnel
Evaluator ($65 per hour or $500
per day)

Vice Principal Meetings (4hrs)

$ 260

Registrar Meeting (1hr)

65

AAD Meeting (1hr)

65

Harvest AAD Google Docs (2 hrs.)

$ 130

Create/conduct Student Surveys (4

hrs.)

$ 260

Create Teacher Surveys (2 hrs.)

$1500

Analyze Data (3 days)

$1500

Write Evaluation (3 days)

$ 130

$ 130

Present findings (2 hrs.)

Registrar ($50 per hour)

Grade Reports (2.5 hrs.)

$ 125

Student aide ($15 per hour)

Student Survey Data Input (3 hrs.)

45

Photocopying, office supplies, etc.

50

TOTAL BUDGET

$4260

Administrative Supplies

12

APPENDICES
APPENDIX A: IFHS ACADEMIC ADVISORY STUDENT SURVEY
* Required
What year are you?*
o

Freshman

Junior

Sophomore

Senior

o
Approximately how often have you been assigned to Academic Advisory this school
year? *
o
never
o

once

2-3 times

usually

always

Academic Advisory Experience:

Skip this section if you have NEVER been assigned to Academic Advisory.

o
o

When assigned, I attended Academic Advisory with that teacher.

never

sometimes

usually

2
always

Rate your level of agreement to the statement,

o
o

Academic Advisory helped me learn more about a topic.

o
o
o
o
o
1
2
3
4

strongly
disagree

o
5

o
5

o
4

o
5

strongly
agree

Academic Advisory helped me make up work I was missing.

o
o
o
o
o
1
2
3
4

strongly
disagree

strongly
agree

Academic Advisory helped me improve my grade.

o
o
o
o
1
2
3

strongly
disagree

strongly
agree

I received more personal help from my teacher during Academic Advisory.

o
o
o
o
o
o
1
2
3
4
5

strongly
disagree

strongly
agree

ALL Students

All students answer the questions in this section.

Overall, I think Academic Advisory is helpful to students. *
o

strongly
disagree

o
1

o
2

o
3

o
4

o
5

I try to keep my grades up because *

strongly
agree

3
o
Rate the following from a major reason (5) to not a reason at all (0) for keeping your
grades up.
o
1

I want a
longer
lunch on
Mondays

I do not
want to
attend
Academi
c
Advisory

o
5

o
2

o
3

o
4

I care
about my
grades

It shows
that I am
learning

I get
other
privilege
s or
rewards

o
Please add any comments, concerns or other information you would like to provide
about Academic Advisory.
o
O

APPENDIX B: IFHS ACADEMIC ADVISORY STUDENT OPEN RESPONSES

Teachers should be prepared to give students work that they will do to help make their
grade better
Should be on Friday
It should only be Tuesday and Thursday, not Monday.
Honestly I really like it. It reminds me what I need to do and my grades are important
Having it on Monday was a good choice
I think academic advisory would help even the kids who have a C or B.
I think it is helpful to students if they actually want to go. Most students don't go, so it
would be helpful to encourage them somehow to want to go.
It's useful
I think academic advisory is very helpful to most students.
It helps for S.A.D/A.A.D too.
I really like how Academic Advisory is currently set up.
I don't really needed it, I have a class that helps everyday
Whenever I go to academic advisory, I don't end up getting a lot done. I think if your
failing a class that's your fault and it's up to you to fix it yourself
I freaking hate academic advisory.
Stop academic advisory
Advisory is no help
I don't like it. We shouldn't have it honestly]
o
o
O

o
o

APPENDIX C: IFHS ACADEMIC ADVISORY TEACHER SURVEY

What subject do you teach?
English

Math

Science

Social Studies

Arts and Music

Professional Technical

Physical Education/ Health

Special Education

Other:

Academic Advisory helps students improve their grades.

o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

Academic Advisory helps students make up missing work.

o

Strongly Agree

o
1

o
2

o
3

o
4

o
5

Academic Advisory helps students learn more.

Strongly
Disagree

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

o
Academic Advisory reinforces extrinsic motivation (reward) over intrinsic
motivation (personal satisfaction).
o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

Academic Advisory does not provide enough time for learning.

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

Academic Advisory is not valued by students.

o
o
o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

7
o

Academic Advisory is not valued by staff.

o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

Lack of consistency between teachers hinders the advisory process.

o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

I have difficulty entering student assignments into the shared google doc.
o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

o
I have difficulty determining where students should be assigned for academic
advisory.
o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

I would benefit from more training on how to assign students.

o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

I would benefit from more training on how to help struggling students.

o

o
o

Strongly
Agree

o
1

o
2

o
3

o
4

o
5

Strongly
Disagree

8
Please add any comments, concerns or other information you would like to provide
about Academic Advisory.
o
o
O

APPENDIX D: IFHS ACADEMIC ADVISORY TEACHER OPEN RESPONSES

Students need more time one's a week is not enough. It is impossible to work with
students only for 25 minutes one's a week.
I think that we should have 2 days for it. It is hard to make significant impact with one
day. I do appreciate having two schedules and not three though. I would deal with three
schedules if we could reach students better
Overall I do think it is a good reminder of student progress when they go to advisory
o