Leveraging Technology to Gather, Analyze,

and Use Assessment Data for

Program Improvement

WASC Conference
April 12, 2013

Dr. Kathy Theuer, Associate Dean and Director of Accreditation
Dr. Carla Piper, Assessment Coordinator and Instructional Designer



1
Brandman University
Part of the Chapman University System
Private/Non-Profit
School of Education, 1 of 5 Schools
26 campuses across California, Washington &
Online
Offer Fully Online & Blended/Hybrid Courses
2
Credential Programs
Multiple Subject Teaching Credential
Single Subject Teaching Credential
Special Education Teaching Credential
Administrative Services Credential
Pupil Personnel Services Credentials
School Counseling
School Psychology

3
Degree Programs
Bachelor of Arts in Early Childhood Education
Master of Arts in Teaching (MAT)
Master of Arts in Education (MAE)
Master of Arts in Educational Leadership and
Administration
Master of Arts in Counseling
Ed.S in School Psychology
Master of Arts in Special Education
Doctor of Education in Organizational Leadership
4
Accreditation



5
The Challenge
Gather and store data from multiple sources
Aggregate data by program
Disaggregate data by:
Campus
Delivery model (blended/fully online)
Term/Academic Year
Analyze data and generate reports for multiple
accrediting bodies


6
Creating an Assessment Infrastructure
Platforms
Processes
People
7
The Platforms
Assessment Management System


Paired with our Learning Management System


8
Processes
What data is collected?
Who gathers the data?
When is data collected?
Where is data stored?
Who analyzes data?
Who provides technology training?
Who addresses technology issues?
Who monitors participation in the assessment
process?
9
Assessment Team
Director of Accreditation- Oversees assessment
process, coordinates data collection, analysis and
report writing, collaborates with assessment
team members
Assessment Coordinator- Builds and copies
rubrics each term, generates reports, oversees
calibration of instructors
Accreditation Technical Coordinator- Manages
fieldwork and TPA sites, addresses technical
issues with LiveText
10
Assessment Team
Accreditation Team- Reviews data with
curriculum teams, shares data with
stakeholders and prepares accreditation
reports
Center for Instructional Innovation- Provides
training for students and faculty on the use of
LiveText

11
Data Sources
Course Embedded Signature Assignments
linked to PLOs
Student Disposition Assessments
Fieldwork Evaluations/Field Placement Data
Demonstration of Mastery Portfolios
Teacher Performance Assessments
12
Course Embedded Assessments
Example: Signature Assignment
EDUU 609
Curriculum Comparative Analysis Paper
Student View- Blackboard
Student/Instructor View in LiveText
Generating Reports
Instructor Level
Program Level
13
Student View in Blackboard
14
Student View in LiveText

15
Instructor View in LiveText

16
Instructor/Student View in LiveText
17
Instructor/Student View
18
Instructor Report
19
Aggregated Results
20
21
Disaggregated Results
Inter-Rater Reliability
22
Disposition Assessments

Excerpt from Brandmans School of Education Professional Code of Conduct

The National Council for the Accreditation of Teacher Education (NCATE) defines dispositions as professional attitudes, values and beliefs demonstrated through both verbal and non-
verbal behaviors as educators interact with students, families, colleagues and communities. These positive behaviors support student learning and development. The School of Education
at Brandman University has adopted a set of professional dispositions that we feel are essential for prospective educators to possess and demonstrate throughout the credential/masters
program.

The dispositions adopted by Brandman Universitys School of Education are listed below. The behaviors listed are examples of the kind of conduct expected of students in the School of
Education, but are not intended to be inclusive.

Professional Demeanor & Responsibility - punctual and prepared for classes and field experiences, dresses appropriately in professional
settings, responsible for meeting deadlines and following instructions, accepts responsibility for own actions, capable of dealing with disappointment, flexible, adapts to change,
reacts appropriately under stress, exhibits a positive attitude toward schools, teaching, students and parents

Commitment to Learning for ALL Students - shows sensitivity to all students, respects individual differences, responds to the needs of
students and their families and communities, embraces diversity to improve student learning, demonstrates compassion, patience and fairness

Communication - has command of written language (mechanics, usage, spelling) and oral English language, uses standard English in writing and speaking,
demonstrates responsive listening skills, able to communicate with a variety of audiences

Collaboration - is skilled in expressing self in groups, works effectively with professional colleagues and other adults, able to accept and respect different points of
view

Self Reflection - engages in problem solving and self-evaluation, accepts and acts upon constructive criticism and is receptive to new ideas, uses feedback to make
improvements, strives for personal and professional growth

Ethics - demonstrates truthfulness and honesty, maintains ethical and legal behaviors in interactions with others, maintains confidentiality, respects the intellectual property
of others by giving credit and avoiding plagiarism/cheating
23
Aggregated Results

24
Disaggregated Results
25
Fieldwork Placement Data

- Create a database of partnerships statewide
(over 500 signed partnerships)

- Assign Master Teachers and University
Supervisors

- Provide a shared workspace for student
teacher/master teacher/supervisor

- Evaluate student performance

26



27
Fieldwork
Shared Workspace
)

28
Fieldwork Assessment
29
Demonstration of Mastery
Portfolios
1. Student completes online application form in
LiveText
2. Student builds and submits portfolio in LiveText
3. Technical Coordinator matches student with
with assessors
4. Portfolio is assessed by multiple reviewers
5. Technical Coordinator lets student know if
they have passed
6. Student records are updated accordingly


30
Portfolio Application Form
31
Portfolio Template
32
Portfolio Template
33
Assigning Assessors
34
Scoring Portfolios
35
Teacher Performance Assessments
1. Student completes online application form in
LiveText
2. Student submits TPA
3. Technical Coordinator matches student with
assessors
4. Technical Coordinator lets student know if
they have passed
5. Student records are updated accordingly
36
TPA Application Form
37
Teacher Performance Assessments
38
Assigning Assessors
39
TPA Rubric
40
Sharing Data with
WASC Program Reviewers
41
Benefits of Technology
LiveText has allowed us to:
Gather and store assessment data easily
Establish consistent expectations of
instructors across multiple campuses
Assess inter-rater reliability
Monitor instructor and student participation
in assessment activities


42
Benefits of Technology
Develop a database of fieldwork sites and
placements
Utilize multiple assessors for demonstration of
mastery portfolios and teacher performance
assessments
Conduct blind reviews for portfolios and
teacher performance assessments

43
Challenges of Technology
Training for students & faculty
Getting students registered for LiveText
Monitoring instructor use of LiveText
Revising rubrics
44
Questions???
For more information contact:

Kathy Theuer theuer@brandman.edu
Carla Piper piper@brandman.edu


45