Evaluation Plan

Program Title:

Multi-Tiered Systems of Support

Prepared by:

Nancy Curran

Doug Head

Burt Parker

3/18/2017

1
 Background

Description of the Program

Positive behavior support (PBS) is a general term that refers to the application of positive

behavioral interventions and systems to achieve socially important behavior change. PBS was

developed initially as an alternative to aversive interventions used with students with significant

disabilities who engaged in extreme forms of self-injury and aggression (Durand & Carr, 1985;

Meyer & Evans, 1989). The use of positive behavior interventions has since evolved to

incorporate more students and in fact, entire schools and school districts. The program is

designed to optimize the power of schools to address problem behaviors that challenge the

operational capabilities of classrooms, by rewarding positive behaviors, rather than punishing

negative behaviors.

In 2014 Hawthorne Elementary School (HES) adopted and began implementation of the

Multi-Tiered System of Support (MTSS) in order to address disruptive and challenging

behaviors within the school and classrooms. The program instituted by Hawthorne Elementary

School, and other schools in Clayton County Public Schools, is a Positive Behavior Intervention

System (PBIS). According to the OSEP Center on Positive Behavioral Interventions and

Supports, PBIS, or in this case MTSS, is a broad range of systemic and individualized strategies

for achieving important social and learning outcomes while preventing problem behavior (OSEP

Center on Positive Behavioral Interventions and Supports, 2005).

The school was looking for alternatives to punishment for curbing unwanted behaviors.

By instituting the MTSS program, it was the desire of the school and district to use positive

influencers to teach students to conform to expect behaviors. The behavior matrix is included in

appendix B. It was not a new intervention package or a new theory of behavior, but an

1
application of a behaviorally based systems approach to enhance the capacity of schools,

families, and communities to design effective environments that improve the fit or link between

research validated practices and the environments in which teaching and learning occur.

Attention is focused on creating and sustaining school environments that improve lifestyle results

(personal, health, social, family, work, recreation, etc.) for all children and youth by making

problem behavior less effective, efficient, and relevant and making desired behavior more

functional (Sugai & et al, 2000).

The program is designed to reward students by giving them Hawthorne bucks which

can be used to purchase reward items from an MTSS store. The students receive the Hawthorne

bucks when they have behaved in a manner deemed worthy of reward by the teachers, staff,

guidance counselors, and/or administration of Hawthorne Elementary School. On designated

dates throughout the semester, the students are then allowed to spend the bucks that they have

received in the store for any items that they desire, up to the amount that they possess. In

addition, there are also reward parties for those students who achieve a given level of Hawthorne

bucks earned. The program works on the assumption that students will respond to the positive

reinforcement of desired behaviors and that teachers will respond with more positive interactions

with the students. The logic model, figure 1, provides a visual representation of how the

program theory works.

2
MTSS (Multi-Tiered Systems of Support) Hawthorne Elementary School, Clayton County Public Schools
The principal, Dr. Cynthia James, wants to evaluate the effectiveness of HESs MTSS program which is in its 3 rd year of implementation. The program was chosen because of its
effectiveness at other schools in curbing inappropriate student behavior and negative teacher-student interaction. At HES K-5 students earn behavior bucks for positive
behaviors. These bucks are accumulated over the course of a month and can be spent in the MTSS store. The program assumes that students will respond to positive reinforcement
of desired behaviors and that teachers will respond with more positive interaction with the students.
Outcomes
Inputs Activities Outputs Short Term Intermediate Longer Term
Student Positive Recognition of Office Referrals Students motivated by New strategies There will be a
Behavior positive behavior from Suspension Records MTSS incentives developed by teachers reduction in the
Lesson Plans Teachers, Detention Records Students learn how to from changing data to number of RTIs
Location for MTSS Administrators, Staff, On-Campus Arrests be safe manage behavior established for student
store and Guidance Number of students Students learn how to Students in greatest behavior.
Food and snacks Counselors. receiving counseling be organized need (based on results There will be a
Prizes Reward parties for for behavior Students learn how to from behavior reduction in the
Tracking software students with positive Number of rewards be accountable for tracking software) will number of guidance
Behavior curriculum behavior bucks provided their actions receive small group referrals for behavior
Signs Identify students in Students who are Students learn how to and individual. intervention.
App uploads of greatest need of receiving awards be respectful counseling sessions More positive school
behavior modification behavior intervention How students are Teachers will learn to Office referrals for climate.
program on to teacher Small group sessions spending their acknowledge positive student behavior will Students enter Middle
cell phones for behavioral behavior bucks student behaviors. be reduced. School, High School
Installation of modification Behavior tracking Students will and society at large
software and tie-ins to One-v-one guidance software results participated in reward with self-governing
infinite campus sessions Behavior Matrix functions. behaviors.
portals Development and Zero Point Club Students choose to be Student learning
delivery of behavior safe. improves.
curriculum Students choose to be
Track and monitor organized.
behavior Students choose to be
Posting of signage accountable.
Earning and spending Students choose to be
of behavior bucks respectful.
Teacher training on Students choose to
the use of MTSS attend class more
program. often.
Implementation and Teachers reward
training on the use of desired behaviors.
behavior tracking
software program.

Figure 1. Logic Model. This model shows the program theory and how the inputs, outputs, activities, and outcomes intersect.

3
Evaluation Purpose

The principal of Hawthorne Elementary School, Dr. Cynthia James, requested the

evaluation of the MTSS Program in order to evaluate the effectiveness of the MTSS Program

within the school. The program is currently in its third year of operation, and she desires to

know if it has made an effective impact on behavior patterns. Positive experiences with similar

programs in other schools and districts were the basis for establishing the program at Hawthorne

Elementary, and now that the program has been in place for three years, it is hoped that enough

data will be available to compare the results since implementation to the years recently past that

were void of the program.

The evaluation will be a formative in nature, as the program is not being considered for

discontinuance at this time. It is our goal to provide Dr. James, and the other stakeholders, with

an evaluation of the program that will provide information about how best to revise and/or

modify it for improvement, or to substantiate the effects of the MTSS Program as it currently

functions. The evaluators will not be involved in making decisions on any possible program

changes suggested by the stakeholders.

It is the goal of the evaluators to answer the following questions (figure 2) for the stakeholders:

Evaluation Question 1: To what extent has MTSS implementation impacted student behavior in
a positive way?

It is essential that this question be answered by this evaluation, as it is the guiding factor

for the implementation of the entire program. Without a positive impact on student behavior,

then clearly the program would not be functioning as designed and intended. As indicated on the

Logic Model, RTIs should also be significantly reduced, as will guidance referrals for behavior

intervention, and student on campus arrests.

4
Evaluation Question 2: To what extent has MTSS implementation positively impacted student
attendance?

Student attendance is directly related to positive interactions with faculty and staff as well

as a decrease in behavior requiring punishment from the administration. If attendance has risen

since the implementation of this program, then it could indicate that there have been fewer

suspensions and/or expulsions. Additionally, if teacher/student interactions have improved, it is

more likely that students would desire to attend school on a more regular basis.

Evaluation Question 3: To what extent has MTSS implementation impacted teachers

reinforcement of positive behaviors even in children who have chronic behavior issues?

By tracking the rewards issued by teachers and to the students who are being regarded,

the evaluators should be able to determine if in fact, the teacher/student interactions have

improved. If teachers are rewarding students who, in the past have been considered chronic

behavior problems, it might indicate a more positive level of interactions between teachers and

those students. As indicated on the Logic Model, better interactions between teachers and

students should establish a more positive school climate for both.

Personnel involved in the Evaluation of the MTSS Program:

Each member of the evaluation team has experience working in low-income areas.

Hawthorne is a Title 1 school where at least eighty-five percent of the student population is at or

below poverty level (Hawthorne Elementary School, 2016).

Nancy Curran has experience in communication and understands the importance of

listening to stakeholders and responding in a timely manner. She holds a Masters degree in

Instructional Technology from Kennesaw State University and is in the process of completing

her EdS in Instructional Technology. Nancy brings to the table a diverse background of working

in multiple school systems including Title 1 and private schools during her 11 years in the

5
classroom. She has served on multiple strategic planning committees and understands the

influence that new programs can have at the school level.

Doug Head brings with him the expertise of working in a Title 1 funded school system

and understands the rigorous process of finding funding for programs. He holds a Masters

degree in Middle Grades Education from Mercer University and is in the process of completing

his EdS in Media at the University of West Georgia. He has worked in the field of education for

over fourteen years, but also has twenty years of experience in the professional business world.

Having been a private business owner, he has experience in conducting evaluation of programs

from multiple perspectives. He is currently employed in a rural county and will be able to bring

a different perspective on similar challenges in working with a Title One student body.

Burt Parker holds a Bachelors of Science in early childhood education from the

University of Georgia and two masters degrees in theological studies from Luther Rice

University. Currently, he is completing an educational specialists degree in instructional

technology. He has been a public educator for 8 years and a pastor for nearly 25 years. He has

extensive experience in family counseling and behavior modification. Currently, Mr. Parker is

an Early Intervention Program teacher at Hawthorne Elementary School.

Methodology

During the course of this evaluation, it will be important that proper data collection is

carried out in order to ensure that the program is functioning properly. It will be the

responsibility of the evaluators to ensure that proper data instruments are utilized to limit bias in

the data analysis. Each evaluator will participate in the process of data collection and analysis

and will be able to present information to the stakeholder whenever necessary.

Data and Instrumentation

6
 The data that will be collected during this evaluation will help to provide feedback to

determine whether or not the Multi-Tiered Systems of Support System is one that provides

incentive for students to exhibit positive behaviors. For this reason, the initial data instruments

will focus on the student behaviors that are being seen on a regular basis. The next stage will

focus on teacher behaviors to determine if teachers are implementing the program and to what

extent the teachers have seen changes in student behaviors.

Initial data will be collected from school discipline and attendance records which will be

tracked over the course of implementation. The administration will provide this data to the

evaluators for analysis purposes. This quantitative data will provide trend data which will allow

the evaluators to analyze whether or not students are responding to the program. If the program

is working and the students are incentivized by the positive rewards, then there should be a

decrease in student discipline referrals and an increase in student attendance compared to before

the programs implementation.

Evaluation Question 1: To what extent has MTSS implementation impacted student behavior in a positive
way?

Indicator Data Source Collection Responsible Timing Analysis

Method Party

Decrease in Students Student Assistant Quarterly3 Univariate

student discipline Principals years pre- analysis
discipline referrals implementation
referrals Program to 3 years post
Administrators

Evaluation Question 2: To what extent has MTSS implementation positively impacted student attendance?

Indicator Data Source Collection Responsible Timing Analysis

Method Party

Increase in Students Student Attendance Annual3 years Univariate

student attendance Clerk pre- analysis
attendance rate records implementation
to 3 years post

7
 Program
Administrators

Evaluation Question 3: To what extent has MTSS implementation impacted teachers reinforcement of
positive behaviors even in children who have chronic behavior issues?

Indicator Data Source Collection Responsible Timing Analysis

Method Party

Increase in 25 Teacher School Year 3 of Comparison of

reinforcement of Randomly responses to Administrators program teacher
positive selected HES survey perceptions pre-
behaviors teachers Program and post-
Administrators implementation
to look for trends

Figure 2. Data Collection/Analysis Matrix. This figure describes the evaluation questions, data collection,
and analysis.

Sampling

Questions 1 and 2 will require no special sampling of the students. The quantitative data

collected daily from Hawthornes 1,000 students will supply data needed. Student discipline

referrals will supply sufficient documentation for the impact of MTSS on student behavior.

Daily student attendance data, which is also stored on the district Share Point site, will provide

the data to ascertain the impact of MTSS on student attendance. Data for question 3 will be

gathered by a random sampling of 25 of HESs 88 faculty members, as described in figure 2.

Care will be taken to insure that there is at least one person represented at each grade level K-5

and at least one representative from the Department of Exceptional Students (DES).

Analysis

As delineated in figure 2, the analysis for questions 1 and 2 will be a simple univariate

analysis. The data from the 3 years prior to implementation will be compared with the 3 years

since implementation. For question 3 an analysis of the teacher pre- and post-implementation

attitudes will be compared for patterns and trends discovered regarding teacher perceptions of

8
the success or failure of different components of the MTSS program.

Standards for Evaluation

In evaluating the data, a 15% reduction in student absences for the 3 years of MTSS

implementation over the 3 years prior will be considered successful. A 15% decrease in office

referrals since the implementation of the MTSS program will likewise be considered successful.

For the evaluation of the teacher attitudinal survey contained in appendix A, 85% of respondents

choosing agree or strongly agree will be considered the standard of success in regard to

question 3.

References

Durand, M. V., & Carr, E. G. (1985). Self-injurious behavior: Motivating conditions and

guidelines for treatment. School Psychology Review, 14, 171-176.

Elias, M.J. (2006). The connection between academic and social-emotional learning. In M.J.

Elias & H. Arnold (Eds.), The educators guide to emotional intelligence and academic

achievement (pp. 4-14). Thousand Oaks, CA: Corwin Press.

Hammond, C., Linton, D., Smink, J., Drew, S., National Dropout Prevention Center, C.S., &

Communities in Schools, A.V. (2007). Dropout risk factors and exemplary programs: A

technical report. National Dropout Prevention Center / Network (NDPC/N).

Hawthorne Elementary School. (2016). Comprehensive School Improvement Plan. Retrieved

from: http://131.clayton.k12.ga.us/common/pages/UserFile.aspx?fileId=4869976

Horner, R.H., Sugai, G., Todd, A.W., & Lewis-Palmer, T. (2005). School-wide positive behavior

support: An alternative approach to discipline in the schools. In L.M. Bambara & L. Kern

(Eds.), Individualized supports for students with problem behavior: Designing

positivebehavior plans. (pp. 359-390). New York: Guilford.

9
Meyer, L. H., & Evans, I. M. (1989). Nonaversive intervention for behavior problems: A manual

for home and community. Baltimore: Brookes.

Minnesota Legislature and Department of Education (2016). 2016 Minnesota english language

arts standards and multi-tiered system of supports implementation survey.Wilder

Research. Retrieved from http://education.state.mn.us/MDE/dse/mtss/

OSEP Center on Positive Behavioral Interventions and Supports. (2005). School-wide positive

behavior support: Implementers blueprint and self-assessment. Eugene, OR: Author.

Sugai, G., Horner, R. H., Dunlap, G. Hieneman, M., Lewis, T. J., Nelson, C. M., Scott, T.,

Liaupsin, C., Sailor, W., Turnbull, A. P., Turnbull, H. R., III, Wickham, D. Reuf, M., &

Wilcox, B. (2000). Applying positive behavioral support and functional behavioral

assessment in schools. Journal of Positive Behavioral Interventions, 2, 131-143.

Steinert, D. (2011). The ins and outs of positive behavioral interventions and supports.

Retrieved from

http://centralspace.ucmo.edu/xmlui/bitstream/handle/10768/43/DSteinert_LibraryScience

.pdf?sequence=1

10
 Appendices

Appendix A

Hawthorne Elementary School | Multi-Tiered Support System

(MTSS) Teacher Survey
Teachers,
We value your feedback. Please fill out the following survey and return it to
your grade-level leader. Your feedback is completely anonymous. We
appreciate your participation!

Area of Service Quality rating

Commitment (Is leadership committed to this effort?)

Leadership, faculty, and staff share responsibility for recognizing Choose an item.
positive behaviors of ALL children.
The principal and school leadership are actively committed to a multi- Choose an item.
year implementation of the MTSS program.
The principal and leadership team models and supports the MTSS Choose an item.
school-wide effort.

Infrastructure (To what extent does the school infrastructure support implementation?)

Instructional staff have been trained on how the MTSS framework is Choose an item.
represented in the school (including implications for curricula and
organization).
The school-level leadership team meets regularly to oversee the on- Choose an item.
going implementation of the MTSS program.
School-wide MTSS actions and results are regularly communicated to Choose an item.
multiple stakeholder audiences including all school staff, families,
school board members, and the community

Data-Based Decision Making (To what extent is data used for making programmatic decisions?

Collaboration around student data and instruction are built into school Choose an item.
expectations, schedules, and calendar.

11
 The school has a consistent data system that tracks student behavior Choose an item.
and attendance.
School-level leadership uses data to drive decision making about the Choose an item.
program and to plan appropriate interventions in the program.
School uses a data system to document and access individual student- Choose an item.
level data for all of the years each student has been in the school.
Multiple measures are used to review the overall effectiveness of the Choose an item.
MTSS program and to make programmatic adjustments and
refinements in how the program is carried out.

BEHAVIOR (To what extent has behavior changed as a result of the

program?)

Behavior in our school has been positively impacted by the Choose an item.
implementation of the MTSS program.
Attendance at our school has been positively impacted by the Choose an item.
implementation of the MTSS program.
The MTSS program has had a positive impact on the school. Choose an item.
Individual teachers are actively acknowledging positive student Choose an item.
behaviors.

Modified from the Minnesota English Language Arts Multi-Tiered System of Supports Implementation Survey

12
 Appendix B

Hawthorne Elementary School Student-Behavior Matrix

I can S.O.A.R. Classrooms Hallways/Restrooms Cafeteria Bus/Transportation
Follow classroom Walk on the third block and Walk in the cafeteria Follow the
expectations (rules) at stay to the right Use quiet voices at all transportation
all times Use the restroom quickly times expectations/rules
Keep your hands and quietly Keep your food on your Take your seat once on
and feet to yourself tray the bus and remain
Push in chairs seated
Place trash in trash can Keep all items in your
book bag
Listen carefully to Walk in line with your class Eat with your own class Make sure you have a
instructions during transitions Move quickly through the bus pass
Wear proper uniform Follow the proper routes for lunch line Use your quiet voice
attire dismissal and emergencies Follow the cafeteria while transitioning home
Raise your hand for signals to make sure you Take supplies on and
teacher recognition are using your inside voice off the bus

Bring all of your Take responsibility for your
school supplies actions
Use only your Get permission to leave the
knowledge during tests classroom
Return promptly
Report restroom problems
to teacher/custodian
Gain permission to leave
your seat once seated
Clean around the area
you are eating (table, floor,
etc.).
Get on your own bus
Get on and off at your
own bus stops
Sit in your assigned
seat

Treat others the way Walk quietly so others can Eat only the food on your Treat your bus driver
you want to be treated continue to learn tray kindly, with manners and
Let your teacher (or Use polite language Talk to the cafeteria dignity
any adult) know when Clean up after yourself in monitors if you have a Allow the
you feel upset, angry, the restroom concern or question younger/smaller students
or afraid Allow for the privacy of Treat your nutritional to sit near the front of the
Show kindness and others staff with dignity and bus
make positive respect Keep the bus clean and
friendships orderly

13