Evaluation Research Methods in the Social Sciences

POEC 6352
SOC 6352
Spring 2006

Professor: Dr. Roxanne Ezzet

Office: GR 3.530
Phone: (972) 883-4764
Email: rezzet@utdallas.edu
Office Hours: Wednesday 2:30- 3:30 pm and by appointment

COURSE OBJECTIVES

This course will prepare you to understand and plan research on programs, projects, policies and
processes of the type we study in Social Science. Program evaluation is the applied branch of
evaluation research, a discipline devoted to the study of all aspects of the evaluation of social
programs, processes and policies.

We will cover the essential elements of planning and conducting meaningful evaluations. Particular
emphasis will be placed on practical applications and issues associated with delivering a useful and
high quality product, especially of interest to public-sector and/or non-profit clients. To achieve
this goal, we will explore issues associated with design and organization, measurement, data
collection and analysis, working with schedules, budgets, and time constraints.

There are two specific objectives for this class:

1. To study, discuss, and explore together the different aspects of program evaluation. We will
read about and discuss a specific set of evaluation topics each week. I will present an
overview and synthesis of viewpoints on each topic to guide our discussion.

2. To design a program evaluation for an ongoing social program, process or policy. This
project will give you a chance to apply the theoretical issues we will be studying. We will
discuss a summary of each program evaluation plan in class so that the author of each plan
has an opportunity to benefit from constructive criticism before preparing the final plan, and
so that you can become familiar with an even wider range of general approaches, specific
problems, and creative solutions. Ph.D. students may want to use this assignment as a draft
of their dissertation proposals as appropriate.

1
COURSE FORMAT

I have selected a format for this course which reflects these objectives. The first part of class each
week generally will be devoted to the discussion of various evaluation issues (listed in the section
entitled “Course Schedule”). You will have assigned readings each week from the class texts and
occasionally from supplemental sources. An additional part of many class sessions will involve a
discussion of evaluation projects or parts of projects, including your own. Toward the end of the
course, we will devote several class periods to discussions of the drafts of class members’ program
evaluation plans.

PROGRAM EVALUATION PLAN

Your big project for this class will be to prepare an outcome evaluation plan for an ongoing social
program or process. At the first class meeting, we will discuss the types of programs you might
consider, and each class member will describe several possible programs or kinds of programs
he/she would like to work with. For example, you could design an evaluation of a smoking
cessation program, a job-training program, or a work-based ride sharing program.

Before the second class meeting, you should have made preliminary contact with one or more
programs and be ready to discuss, and then select, the program for which you will design an
evaluation. The program you choose can be of any type, but you should not select a program with
which you have had prior contact. I will provide a letter of introduction to use in contacting
programs.

Social programs typically do not have staff trained in evaluation, so your offer to design an
evaluation will probably be quickly accepted. Be sure to make it clear to the program that you are
only designing an evaluation plan for this course, although I would not be surprised if several of the
programs ask you to carry out the plan, or ask for your help in implementing the evaluation
themselves. It is very possible that masters and doctorate students will conduct research based on a
project begun in this class.

In designing a plan, you will need to visit the project extensively to familiarize yourself with the
program’s goals, organization, operation, personnel, and clients. For the first few weeks of this
class, plan to spend 5 to 8 hours per week on project-related calls, visits, meetings, etc. You will
also be able to work individually with me in developing your evaluation.

Your evaluation will consist of several parts – these are detailed on the next page, in the outline for
your plan.

2
 EVALUATION PLAN

Outline of Chapters

Executive Summary

I. Introduction
II. Literature review and relevant discussion of the evaluation to be studied
III. Program Description

A. General description of the program (e.g., history, size, location, director,

organization, special political considerations, theory/model on which the intervention
is based)

B. Specific goals and objectives of the program; criteria identified to measure the
program’s success in achieving its objectives

IV. Evaluation Objectives (your evaluation will address either the process or the program’s
intended outcomes)

A. Process

B. Outcome

V. Evaluation Design and Measures (your evaluation will address either the process or the
program’s intended outcomes)

A. Process component

B. Outcome component

VI. Proposed Implementation Timetable

VII. Proposed Data Analyses

VIII. Final Report Outline

IX. Evaluation Budget

Notes

Bibliography

Appendices (including your vitae, program materials, drafts of all measures, and other
materials)

3
All students will present their evaluation plans at the end of the semester. For the in-class
presentation of your evaluation plan, prepare a full draft of Chapters 3, 4, and 5 for us to review
before the class meeting. The class discussion, then, can be focused on specific issues rather than
on general discussion. You should bring copies of your draft to class the week before we are
scheduled to discuss it, so that everyone can read your draft before class and be ready for
discussion. We will determine the schedule for class presentations several weeks into the semester.
At the same time, I will give you more details about the format for the class presentations. In brief,
it will involve a short introduction by the student, followed by prepared reactions/comments from
two different classmates, ending with a full class discussion. Each student, then, will be a lead
commentator on two other evaluation plans.

Final, revised evaluation plans will be due by 2:00 pm Tuesday, April 25 to Professor Ezzet’s office
(GR 3.530).

READINGS

Required: The required textbook that we will be using is:

Peter H. Rossi, Mark W. Lipsey and Howard E. Freeman. 2004. Evaluation: A Systematic
Approach. Thousand Oaks, CA: Sage Publications, Inc, 7th edition.

The textbook is available for purchase at the UTD Bookstore and from Off Campus Books (located
at 518 W. Campbell Road in Richardson).

Recommended: The following readings are recommended.

National Science Foundation (2002). The User-Friendly Handbook for Project Evaluation.
Available online in html: http://www.nsf.gov/pubs/2002/nsf02057/start.htm
Or in pdf format: http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf

U.S. General Accounting Office (April 24, 1998). Program Evaluation: Agencies
Challenged by New Demand for Information on Program Results. GGD-98-53.
Available online: http://www.gao.gov/archive/1998/gg98053.pdf

U.S. General Accounting Office (1991). Designing Evaluations. PEMD-10.1.4

Available online: http://www.gao.gov/policy/10_1_4.pdf

Additional Reading:
These resources may be helpful to you as you prepare your evaluation plan.

Aday, L.A. (1996). Designing and Conducting Health Surveys. San Francisco: Jossey-Bass
Publishers.

Aday, L.A. (1998). Evaluating the Healthcare System: Effectiveness, Efficiency and Equity.
Chicago: Health Administration Press.

4
Bloom, M., Fischer, J., & Orme, J.G. (1995). Evaluating practice: guidelines for the accountable
professional, 2 nd ed. Boston: Allyn and Bacon.

Chambers, R. E., Wedel, K. R., & Rodwell, M. K. (1992). Evaluating Social Programs. Needham
Heights, MA: Allyn and Bacon.

Cheetham, J. (1992). Evaluating Social Work Effectiveness. Philadelphia, PA: Open University
Press.

Fetterman, D.M., Kaftarian, S.J., and Wandersman, A. (eds.)(1996). Empowerment Evaluation,

Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: Sage.

Fischer, J., & Corcoran, K. (1994). Measures for clinical practice: a sourcebook (Second Edition).
New York: Free Press.

Jaeger, R. M. (1991). Statistics, A Spectator Sport. Beverly Hills, CA: Sage.

Lamb, S., Greenlick, M.R., and McCarty D. (eds.) (1998). Bridging the Gap between Practice and
Research. Forging Partnerships with Community-Based Drug and Alcohol Treatment. Washington,
DC: National Academy Press.

Madison, A. (1992). Minority Issues in Program Evaluation. San Francisco, CA: Jossey-Bass.

Martin, L. L., & Kettner, P. M. (1996). Measuring the Performance of Human Service Programs.
Thousand Oaks, CA: Sage Publications.

McDowell, I. And Newell, C. (1996). Measuring Health, 2nd edition. New York: Oxford
University Press.

National Institute on Drug Abuse. (1995). How Good is your Drug Abuse Treatment Program?
A Guide to Evaluation (NIH Publication No. 95-3609). Washington, DC: U.S. Government Printing
Office.

Panel on Cost-Effectiveness in Health and Medicine (1996). Cost –Effectiveness in Health and
Medicine. A Report to the U.S. Public Health Service. Washington, DC: U.S. Department of Health
and Human Services, Office of Public Health and Science, Office of Disease Prevention and Health
Promotion.

Patton, M. Q. (1987). How to Use Qualitative Methods in Evaluation. Newbury Park, CA: Sage.

Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Newbury Park, CA: Sage.

Patton, M. Q. (1997). Utilization-Focused Evaluation (3rd Edition). Thousand Oaks,

CA: Sage Publications.

Pecora, P. J., Fraser, M. W. Nelson, K. W., McCroskey, J., and Meean, W. (1995). Evaluating
Family Based Services. New York: Aldine de Gruyter.

5
Posavac, E.J. and Carey, R.G. (1997). Program Evaluation. Methods and Case Studies. 5 edition.
th

Englewood Cliffs, NJ: Prentice Hall.

Rothman, J., & Thomas, E.J (Eds) (1994) Intervention Research: Design and Development for the
Human Services. New York: Haworth Press.

Schalock, R. L. (1995). Outcome-Based Evaluation. New York: Plenum Press.

Smith, P. (Ed.) (1996). Measuring Outcome in the Public Sector. London: Taylor & Francis

Weiss, C. H. (1998) Evaluation: Methods for Studying Programs and Policies Second Edition).
Upper Saddle River, NJ; Prentice Hall.

Wholey, J. S., Hatry, H. P. & Newcome, K. E. (1994). Handbook of Practical Program Evaluation
(First Edition), San Francisco: Jossey-Bass

Windsor, R. C., Baranowski, T., Clark N. and Cutter, G. (1984). Evaluation of Health Promotion
and Education Programs. Palo Alto, CA: Mayfield Publishing Company.

Yates, B. T. (1996). Analyzing Costs, Procedures, Processes, and Outcomes in Human Services.
Thousand Oaks. CA: Sage Publication.

Yates, B. T (1999) Measuring and Improving Cost, Cost-Effectiveness and Cost-Benefit for
Substance Abuse Treatment Programs: A Manual. Bethesda MD: US Dept of Health and
Human Services. National Institutes of Health.

6
COURSE GRADING

Your final grade will be based on the following:

Evaluation plan
Evaluation program description & organization chart 10%
Logic Model 5%
Draft paper and class presentation 20%
Final paper 50%
Attendance & Commentaries on others’ papers 15%
100%

Note: All assignments must be word processed or typed. No handwritten paper topics, outlines,
reading summaries or papers will be accepted.

UNIVERSITY POLICY ON SCHOLASTIC DISHONESTY

Scholastic dishonesty is taken very seriously and will not be tolerated. Cheating on exams,
plagiarism, or any other form of academic dishonesty is unacceptable. The UTD website and
catalog include information on scholastic dishonesty. Please familiarize yourself with the
university's policy. By turning in assignments and exams, you are certifying that the work is your
own and does not plagiarize or otherwise use other works without citing the appropriate reference.
Most students are honest and seek to learn without copying or otherwise violating rules. We trust
that students are obeying the precepts of academic honesty, but that creates an obligation on the part
of students to uphold that trust. If you have any questions about scholastic honesty regulations,
please consult the university's policy or contact the professor. Violations of scholastic honesty are
serious matters, and will be handled in accordance with UTD policies.

DISCLAIMER
This syllabus provides a tentative schedule for the semester and is subject to change. We will try to
stay as close as possible to the scheduled dates in terms of readings, though some changes may
occur. Any changes will be announced in lecture.

7
 COURSE SCHEDULE AND READINGS

Readings listed are required and should be done prior to the class meeting.
----------------------------------------------------------------------------------------------------------------

Week 1 (Wednesday, January 11): Introduction to Program Evaluation

RLF: Chapter 1

Optional: NSF pp 3-13

Week 2 (Wednesday, January 18): The Context of Evaluation

RLF: Chapter 2

DUE: Program ideas to discuss in class

Week 3 (Wednesday, January 25): Evaluation Goals and Objectives

RLF: Chapter 3

Optional: NSF 15-30

DUE: Evaluation program description & organization chart

Week 4 (Wednesday, February 1): Needs Assessment

RLF: Chapter 4

Week 5 (Wednesday, February 8): Program Concept and Theory

RLF: Chapter 5

Week 6 (Wednesday, February 15): Assessing Program Process

RLF: Chapter 6

DUE: Logic Model

Optional: GAO91

Week 7 (Wednesday, February 22): Assessing Program Outcomes

RLF: Chapter 7

8
Week 8 (Wednesday, March 1): Impact Assessment: Field Design

RLF: Chapters 8-9

March 8 – Spring Break (March 6-10) No classes

Week 9 (Wednesday, March 15): Detecting Program Effects

RLF: Chapter 10

Optional: NSF 31-42

Week 10 (Wednesday, March 22): Measuring Efficiency

RLF: Chapter 11

Week 11 (Wednesday, March 29): Evaluation Budgeting; Reporting; Vitae Preparation;

Utilization; Evaluation Research as a Discipline and Profession

RLF: Chapter 12

DUE: Draft Papers for Apr 5 presentations

Week 12 (Wednesday, April 5): Student Presentations

DUE: Draft Papers for Apr 12 presentations

Week 13 (Wednesday, April 12): Student Presentations

DUE: Draft Papers for Apr 19 presentations

Week 14 (Wednesday, April 19): Student Presentations

Final Program Evaluation plan due by 2:00 pm Tuesday, April 25. Please turn in to Professor
Ezzet’s office in GR 3.530.