Title i

TITLE

by

Author

An evaluation project report submitted to the faculty of

Brigham Young University

in partial fulfillment of the requirements for the degree of

Doctor of Philosophy

Department of Instructional Psychology and Technology

Brigham Young University

Date
 Title ii

ABSTRACT

Abstract text goes here . . .

 Title iii

TABLE OF CONTENTS
 Title iv

LIST OF FIGURES
 Title v

LIST OF TABLES
 Title vi

Note: Fill in the following sections with information

requested and answering the associated questions in
paragraphs, not in outline form as shown. Change
tense to past for reports, keep tense in future for
proposals. This is using APA with 2 headings. Modify
according to the number of headings needed.
Chapter 1: Introduction

Background and Context

What evaluation has been done on this evaluand already?

Is the evaluand evaluable at present?
Why is an evaluation appropriate now?

Evaluator Background

How did this evaluand come to be of interest to you?

What is your background that is relevant to this evaluation?

Stakeholders

Who asked for the evaluation and why?

Who stands to benefit from the evaluation and how?
Who is served by the evaluand or should be?
Who is likely to use the evaluation results to do something helpful?
Who does not usually have a voice in matters associated with the evaluand but has
a stake in it?

Evaluand

What is the evaluand the stakeholders care about?

What do you know about the evaluand?
What are the evaluand’s objectives?
What is the evaluand’s program logic?
How does the evaluand function or work?
What more do you need to learn to refine the description and definition of the
evaluand so you can focus your evaluation on it?
 Title vii

Stakeholder Issues and Concerns

What issues, concerns or information needs do the stakeholders have regarding

the evaluand?
Do they want summative, formative, or both kinds of information?
Do they want to focus on accountability of someone for the evaluand?
What concerns do they have about the evaluand?
What are they saying they want to know?
What is unsettled among them about the evaluand?
What information are they asking you to gather?

Chapter 2: Literature Review

What does the literature associated with the evaluand say are the key issues?

Key Issue 1

Key Issue 2

Key Issue 3

Chapter 3: Evaluation Design

Evaluation Criteria and Standards

What criteria do stakeholders have for judging the evaluand?

What values do the stakeholders manifest regarding the evaluand?
What do they think the evaluand should be accomplishing (criteria for success)?
What standards do they have or how completely do they hope the evaluand will
meet the criteria?
How will they know when the evaluand is successful to their satisfaction?
 Title viii

Criteria 1

Criteria 2

Criteria 3

Etc.
Standard 1

Standard 2

Standard 3

Etc.

Evaluation Questions

What questions do stakeholders need or want to answer regarding how well the
evaluand meets the criteria?
Based on the previous points, what evaluation questions should be asked?
Based on a rating or ranking of all possible questions raised, which are the highest
priority?
Which questions will this study address and why?

Question 1

Question 2

Question 3

Etc.

Data Collection and Analysis Methods

What processes and activities will be used to collect data to answer the questions
and compare the evaluand to the criteria?
How will each collection procedure be refined to ensure validity, reliability,
credibility, trustworthiness, generalizability, transferability, or whatever other methods
standards apply?
How will qualitative data be summarized and interpreted?
How will quantitative data be summarized and interpreted?
For each evaluation question, answer in paragraphs the details summarized in
Table 1.
 Title ix

Table 1. Summary of evaluation questions and how they are answered

Evaluation Information Using By On what How will How What

Question Needed what whom schedule data be will resource
from what data ? ? analyzed/ results s are
sources? collect interpreted be needed
ion to make reported for all
proced evaluation ? of this?
ures? s?
(How
will
they
be
refined
?

Reporting to Stakeholders

What reporting strategies will be used to get information to stakeholders?

What interim reports will be given to whom and when?
What final reports will be given to whom and when?
How will the reports be organized, around what points?
Will there be oral reports? Written reports? Other formats?

Required Resources

What resources will be used to carry out the entire study?

What resources are needed for each kind of data collected and analyzed?
What time resources are needed and by whom?
What materials, equipment, and other items are needed?
What are the qualifications, characteristics, predispositions, relevant values and
experiences of the evaluation team members?
How will potential conflicts of interest, biases, and so on be addressed?
How will human subjects be treated?
 Title x

Chapter 4: Findings

What are the results or answers to the evaluation questions? In the report, present data
and answers in the sections below. In the proposal, answer these questions since you
can’t present results yet.
How will results be organized?
What displays of results do you anticipate using and why?
Can you anticipate any results already?

Results for Evaluation Question 1

Results for Evaluation Question 2

Results for Evaluation Question 3

Etc.

Chapter 5: Recommendations & Conclusions

What recommendations does this study yield? In the report, recommendations in the
sections below. In the proposal, answer these questions since you can’t present
recommendations yet.
Where will recommendations come from?
Will you be qualified to make recommendations and why?
Can you anticipate any kinds of recommendations already?

Recommendation 1

Recommendation 2

Recommendation 3

Etc.
 Title xi

Limitations

Conclusions

Chapter 6: Meta-evaluation

How did this study hold up against each of the 30 meta-evaluation standards or Guiding
Principles (and others if appropriate)?

This section will contain your meta-evaluation or critique, which, together with the report
in chapters 1-5, constitutes your comprehensive exam for the doctoral degree. You may
find it helpful to use the checklist from Stufflebeam and answer each of 10 questions for
each of the 30 standards at
http://www.wmich.edu/evalctr/checklists/program_metaeval_10point.htm .

But whether you do that or not, in this section explain why you met or did not meet each
standard; or explain why some standards were not relevant, if you feel that is the case.

The standards are listed below. You may fill in your responses there and/or do so in an
attachment using the checklist linked above as a framework for your critique.

Complete the additional questions that follow the standards regarding your overall
critique and comparisons to proposed versus actual schedule and budget.
 Title xii

Utility Standards

The utility standards are intended to ensure that an evaluation will serve the information

needs of intended users.

U1 Stakeholder Identification Persons involved in or affected by the evaluation should

be identified, so that their needs can be addressed.

U2 Evaluator Credibility The persons conducting the evaluation should be both

trustworthy and competent to perform the evaluation, so that the evaluation

findings achieve maximum credibility and acceptance.

U3 Information Scope and Selection Information collected should be broadly selected to

address pertinent questions about the program and be responsive to the needs and

interests of clients and other specified stakeholders

U4 Values Identification The perspectives, procedures, and rationale used to interpret

the findings should be carefully described, so that the bases for value judgments are

clear.

U5 Report Clarity Evaluation reports should clearly describe the program being

evaluated, including its context, and the purposes, procedures, and findings of the

evaluation, so that essential information is provided and easily understood.

U6 Report Timeliness and Dissemination Significant interim findings and evaluation

reports should be disseminated to intended users, so that they can be used in a

timely fashion.
 Title xiii

U7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways

that encourage follow-through by stakeholders, so that the likelihood that the

evaluation will be used is increased.

Feasibility Standards

The feasibility standards are intended to ensure that an evaluation will be realistic,

prudent, diplomatic, and frugal.

F1 Practical Procedures The evaluation procedures should be practical, to keep

disruption to a minimum while needed information is obtained.

F2 Political Viability The evaluation should be planned and conducted with anticipation

of the different positions of various interest groups, so that their cooperation may

be obtained, and so that possible attempts by any of these groups to curtail

evaluation operations or to bias or misapply the results can be averted or

counteracted.

F3 Cost Effectiveness The evaluation should be efficient and produce information of

sufficient value, so that the resources expended can be justified

Propriety Standards

The propriety standards are intended to ensure that an evaluation will be conducted

legally, ethically, and with due regard for the welfare of those involved in the

evaluation, as well as those affected by its results.

 Title xiv

P1 Service Orientation Evaluations should be designed to assist organizations to address

and effectively serve the needs of the full range of targeted participants.

P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be

done, how, by whom, when) should be agreed to in writing, so that these parties are

obligated to adhere to all conditions of the agreement or formally to renegotiate it.

P3 Rights of Human Subjects Evaluations should be designed and conducted to respect

and protect the rights and welfare of human subjects.

P4 Human Interactions Evaluators should respect human dignity and worth in their

interactions with other persons associated with an evaluation, so that participants

are not threatened or harmed.

P5 Complete and Fair Assessment The evaluation should be complete and fair in its

examination and recording of strengths and weaknesses of the program being

evaluated, so that strengths can be built upon and problem areas addressed.

P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full

set of evaluation findings along with pertinent limitations are made accessible to

the persons affected by the evaluation and any others with expressed legal rights to

receive the results.

P7 Conflict of Interest Conflict of interest should be dealt with openly and honestly, so

that it does not compromise the evaluation processes and results.

 Title xv

P8 Fiscal Responsibility The evaluator's allocation and expenditure of resources should

reflect sound accountability procedures and otherwise be prudent and ethically

responsible, so that expenditures are accounted for and appropriate.

Accuracy Standards

The accuracy standards are intended to ensure that an evaluation will reveal and convey

technically adequate information about the features that determine worth or merit

of the program being evaluated.

A1 Program Documentation The program being evaluated should be described and

documented clearly and accurately, so that the program is clearly identified.

A2 Context Analysis The context in which the program exists should be examined in

enough detail, so that its likely influences on the program can be identified.

A3 Described Purposes and Procedures The purposes and procedures of the evaluation

should be monitored and described in enough detail, so that they can be identified

and assessed.

A4 Defensible Information Sources The sources of information used in a program

evaluation should be described in enough detail, so that the adequacy of the

information can be assessed.

A5 Valid Information The information-gathering procedures should be chosen or

developed and then implemented so that they will assure that the interpretation

arrived at is valid for the intended use.

 Title xvi

A6 Reliable Information The information-gathering procedures should be chosen or

developed and then implemented so that they will assure that the information

obtained is sufficiently reliable for the intended use.

A7 Systematic Information The information collected, processed, and reported in an

evaluation should be systematically reviewed, and any errors found should be

corrected.

A8 Analysis of Quantitative Information Quantitative information in an evaluation

should be appropriately and systematically analyzed so that evaluation questions

are effectively answered.

A9 Analysis of Qualitative Information Qualitative information in an evaluation should

be appropriately and systematically analyzed so that evaluation questions are

effectively answered.

A10 Justified Conclusions The conclusions reached in an evaluation should be explicitly

justified, so that stakeholders can assess them.

A11 Impartial Reporting Reporting procedures should guard against distortion caused

by personal feelings and biases of any party to the evaluation, so that evaluation

reports fairly reflect the evaluation findings.

A12 Metaevaluation The evaluation itself should be formatively and summatively

evaluated against these and other pertinent standards, so that its conduct is

appropriately guided and, on completion, stakeholders can closely examine its

strengths and weaknesses.

 Title xvii

Schedule

What schedule will be (was) followed (each part attached to particular questions,
collection procedures, and analysis activities) and how did it compare to what was
planned (when the study is complete)? Use a Gantt Chart or something similar to
plot out when particular activities will be (were) conducted so pre-requisite
activities are complete in time for subsequent activities. In the report, compare
proposed to actual schedule followed.

Budget

What budget will be (was) followed (each part attached to particular questions,
collection procedures, and analysis activities) and how did they compare to what
was planned (when the study is complete)? Clarify for each question what the
budget will be (was) and how it compared to the actual expenditures of time and
money (even if no actual funds were spent).

Overall Critique

Strengths

Weaknesses

References
 Title xviii

Appendix A