Professional Documents
Culture Documents
Christina (Tina) Christie Associate Professor & Division Head Social Research Methodology Graduate School of Education & Information Studies UCLA 2/24/12
Findings are:
Program & context specific Shared on an ongoing basis
Findings are:
Intended to be broadly applicable or generalizable Shared at the end of the study
Evaluation Defined
Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process. (Scriven, 1991, p. 139) Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs. (Rossi, Lipsey, Freeman, 2004, p. 28) Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. (Patton, 1997, p. 23)
Why Evaluate?
People (stakeholders) naturally make evaluative judgments about programs & policies, often based on limited information and are susceptible to biases Evaluators use a set of tools (research designs, methods) and roadmaps (evaluation theories) that offer stakeholders understanding of and action in relation to programs and policies
5
Tree Roots
accountability- an important motivation for evaluation, a way to improve programs and society systematic social inquiry methodical and justifiable set of procedures for determining accountability epistemology- nature and validity (or limitations) of knowledge- the legitimacy of value claims, the nature of universal claims, and the view that truth (or fact) is what we make it to be
Methods
more accurate to describe these approaches as emphasizing research methodology evaluation is primarily centered on research methodology- knowledge construction models are mostly derivations of the randomized control trail, and are intended to offer results that are generalizable
Valuing
placing value on the subject of the evaluation, the evaluand, is essential to the process initially driven by the work of Michael Scriven (1967) and Robert Stake (1967, 1975), this work firmly establishes the vital role of valuing in evaluation split in two objectivist and subjectivist which distinguishes the two fundamental perspectives informing the valuing process
Use
the pioneering work of Daniel Stufflebeam (initially with Egon Guba) and Joseph Wholey, originally focused on an orientation toward evaluation and decision making an explicit concern for the ways in which evaluation information will be used, and focuses specifically on those who will use the information.
13
What Does it Mean for an Evaluation to Have Impact? (That is, for an Evaluation to be Useful or Used)
14
Plan-Do-Study-Act
PDSA cycle: Plan, Do, Study, Act (Langley et. al., 2009). This methodology is an iterative process in which improvements are developed, tried, studied, tested against evidence, and then refined, over and over, until quality is improved and variability reduced to acceptable limits.
Peer Observation
Formative peer observation assists in the improvement of teaching. Summative peer observation involves the evaluation of teaching effectiveness used for merit, promotion, and/or tenure decisions. Both formative and summative observations can be based on the same observation instruments.
Peer Observation
40% of colleges and universities now use peer classroom observation. Observations offer insight regarding the improvement of teaching. Higher education settings are currently moving toward multiple observation formats. Strengths/ Advantages of Peer Observation Gaining new ideas and perspectives about teaching from colleague(s); Both observer and observe may improve teaching ability;
Peer Observation
Strengths/ Advantages of Peer Observation Gaining new ideas and perspectives about teaching from colleague(s); Both observer and observe may improve teaching ability; Weaknesses/ Disadvantages of Peer Observation Possible bias relating to the observer's own beliefs about teaching; Without a systematic approach including observer training, multiple visits, and use of reliable observation instruments peer observation is not a valid method for summative evaluation.
Peer Observation
Guidelines The observer should arrive at least 10 minutes before class. "Walking into class late is poor practice and inconsiderate" (Seldin, 1999, p. 81). The observer can be briefly introduced to the students, with an equally brief explanation of why the observer is present. Then move on! Observers are not to ask questions or participate in activities during class; such behavior can detract from and invalidate the observations. An effective observation requires an observation instrument designed to accurately and reliably portray the teacher's behavior.