S290 Syllabus (1/3/2017), Prof.

Eric Taylor

S290—QUANTITATIVE METHODS FOR IMPROVING

CAUSAL INFERENCE IN EDUCATIONAL RESEARCH
SPRING 2016

Instructor Professor Eric Taylor

Gutman Library 469
eric_taylor@gse.harvard.edu
(617) 496-1232

Office Hours Tuesdays 10:00 am – 12:00 noon or Wednesdays 8:00 am – 10:00 am

Sign up at http://taylorofficehours.wikispaces.com

Faculty Assistant Tal Vaval

Gutman Library 466
tal_vaval@gse.harvard.edu
(617) 496-2975

Class Meetings Tuesdays 4:00-7:00 pm

Course Website https://canvas.harvard.edu/courses/25200

COURSE DESCRIPTION AND GOALS

Important research questions in every discipline are usually framed causally. Without a strong and
convincing design, research in education cannot provide the evidence that is needed to answer critical causal
questions. Often research simply leads us to conclude that teacher qualities are related to student learning,
that early intervention is associated with later school success, and that adults in countries with more
stringent compulsory schooling laws have higher labor market earnings. These are statements about
relationships among variables, not about the causal mechanisms that link them. Thus, the ability of
researchers to answer causal questions depends critically on research design.

Research design is also extremely important for policy and practice. Policymakers use research results to
make the case for, or against, particular decisions that may affect the lives of millions of children for many
generations to come. An important part of our job as researchers is to understand the conditions under which
causal conclusions can be drawn, and to communicate the strength of the evidence in support of such
conclusions to policymakers and practitioners.

In this advanced research seminar, we have four major goals. First, I want to ensure that you are aware of,
and understand, the formal application of modern methods of causal inference. Second, I want you to
understand how these methods are used (and misused) in the practice of research. To accomplish these first
two goals, class time will consist primarily of guided discussion of carefully selected quantitative research
studies that apply these methods to important questions in education policy. These studies are all highly
cited and published in major journals, and they investigate important issues such as the effects of class size,

1
S290 Syllabus (1/3/2017), Prof. Eric Taylor

financial aid for college, early childhood programs, and governance structure (public versus charter versus
private schools) on student achievement, as well as on long-run outcomes such as attainment, earnings,
health, crime, and civic participation.

Our third and fourth goals in this seminar relate to your professional development as scholars. Our guided
discussions are designed not only to help you understand new research, but also to put you in the shoes of
the researchers who authored the work in question. We will criticize existing work, but we will also
appreciate its strengths and think carefully about the choices that were made in each study. We will devote
an entire class to the peer review process, where I provide you with initial submissions, editor letters and
referee reports, and final versions of two of my published papers. Finally, the last three weeks of class will
consist of individual presentations of course projects. This will be an opportunity for each of you to present
your own work, but it will also provide a good setting for you to practice the important function of providing
helpful seminar feedback on others’ work.

The last course goal is to help each of you make the transition from being a consumer to a producer of
independent scholarly research. The main work product of S-290 is the course project, and it is discussed
in more detail below. S-290 can stand alone as the termination of a thoughtful program of doctoral study or
be an effective precursor, or follow-up, to other advanced courses in quantitative methods.

EXPECTATIONS AND ASSIGNMENTS

Attendance and participation 20% of grade

Presentation 25% of grade final three class meetings
initial proposal due 1/29
Course project 55% of grade draft due 3/31
final paper due 5/5

The course is letter graded, and a grade of “incomplete” is not allowed.

Preparation for and Participation in Class. I expect each student to prepare for, attend, and actively
participate in class every week. Preparing for class includes three tasks: (i) read the assigned readings, (ii)
answer the discussion questions, and (iii) (re)familiarize yourself with the basics of the week’s focus
methods. I strongly encourage you to form a study group to help you accomplish these tasks. If you need
help finding a group, please let me help.

Each week’s required readings include one or more scholarly articles (available online with your Harvard
credentials), and one or more chapters from the book Methods Matter.

Murnane, Richard J. & John B. Willett. (2011). Methods Matter: Improving Causal Inference in
Educational and Social Science Research. Oxford University Press.

On Friday each week I will post a list of discussion questions for the week’s readings. Our in-class
discussion will focus on, but not necessarily be limited to, your answers to these questions. (I reserve the
right to “cold call” at random for answers to these questions if participation is not well distributed.)

2
S290 Syllabus (1/3/2017), Prof. Eric Taylor

To prepare each week you should also (re)learn the basics of the week’s methods before class. As a general
rule, I will not use class time to lecture on the formal derivation and statistical properties of the methods.
In this course we are primarily concerned with how methods are applied in practice. We will discuss each
method in great detail, but our discussion will be grounded in the assigned research papers. To reiterate,
I expect you to use out-of-class time to make sure you are familiar with the basics of each method. To
support you in that task, I suggest making use of a number of resources. First, the course text Methods
Matter is a clearly-written and well-structured resource to help you learn or relearn these methods. There
are other textbooks covering these methods, including Mastering Metrics and Mostly Harmless
Econometrics both by Joshua Angrist and Jörn-Steffen Pischke, and I would be happy to discuss the
alternatives with you. Second, on the course website I have posted John Willett’s methodology slides which
were developed in past versions of this course. Third, a critical step in your own learning of these methods
is for you to teach the methods to others. There is no time like the present to get started! An excellent use
of group study time is to take turns teaching each other. Fourth, if you (or your study group) encounters a
methodological question you cannot answer, send me an email so we can address the question in class.

Course Project. Your primary assignment for the course is to develop an original research project using the
methods covered in the course. You will submit a brief one-page proposal (due 1/29), a draft (due 3/31),
and a final paper (due 5/5). (You will also present your work in class, as described in the next section.) You
may work on this project alone or in pairs. Indeed, I encourage you to work with a partner.

The primary goal of this assignment is to begin or advance a research project that you will continue working
on after the course and which will eventually result in a published paper for your CV. Please choose a
project (and partner) with that goal in mind. If you are not interested in that goal please come talk to me
now; I can suggest other courses that might be a better fit for your goals. My role in your project is to
provide individualized guidance and feedback along the way. In addition to the scheduled deadlines, I
encourage you to keep me informed on the progress of your project throughout the semester (and beyond).

Your proposal (due 1/29) should be a one page overview describing the proposed project. It should include
research question(s), the data set you will use, and the causal inference methods you plan, at this point, to
use.

Your final paper (due 5/5) should be as close to a complete research paper as possible, with the notable
caveats described in the next paragraph. The typical final paper will include sections for (i) introduction /
motivation, (ii) background / literature review, (iii) data and sample, (iv) methods, (v) results, and (vi)
discussion. (All papers should include all of these topics. But if your paper would be better-off with a
different organization of topics, please come talk with me.) All papers must use one or more of the methods
for causal inference discussed in the course. All papers must be no more than 20 pages in length (double-
spaced, one-inch margins, 12-point serif font).

Some students (partnerships) will be able to produce a complete research paper during the semester,
including results from data analysis. This is more likely for students (partnerships) who come into the course
with a data set in hand. For other students (partnerships) it will not be possible to produce results from data
analysis because the data are not yet available (e.g., you are currently fielding an experiment, or you are
waiting on administrative approval for access to data). If you are in this “data not yet available” situation
your final paper should still include all of the sections described in the previous paragraph. Even without
data, you can write complete sections for (i) introduction / motivation, (ii) background / literature review,

3
S290 Syllabus (1/3/2017), Prof. Eric Taylor

(iii) data and sample, and (iv) methods. Of course, these sections may evolve in future drafts after you have
the data. For the (v) results section you should write an “analysis plan”. First, list all of the statistics and
models you plan to estimate, including plans that are contingent on initial results. Second, make draft tables
and figures which lack results but which do include titles, labeling, descriptive notes, etc. Third, discuss
why you are including each model or statistic, what alternatives you did not choose and the tradeoffs, and
what you expect the results to be. For the (vi) discussion section draft discussion and implications under
different results scenarios (e.g., positive results, null results). The two scenarios in this paragraph are
general descriptions; I encourage you to come talk with me about your project so we can both have clear
expectations about “as close to a complete research paper as possible” means for your project.

Your grade on the course project will depend on how ambitious the project is and how well you carry it
out. (In that regard, the grading is a bit like the scoring of competitive diving, where the difficulty of a dive
affects the total score as well as does the quality of the execution of the dive.) The main objective is to help
you write the best research paper possible, not to get the best grade possible. These two things are not
always in conflict. However, I would rather you undertake an ambitious project with an uncertain outcome
than produce a substandard paper or repackage something you’ve already done to fulfill the requirement.

Presentation. You (and your partner) will present your paper seminar style in class during one of the final
three classes of the semester. The presentation is an opportunity to practice giving a seminar generally, and
specifically an opportunity to gather feedback to incorporate into your final paper. These presentations will
also provide an opportunity to practice participating in a seminar and providing helpful feedback to others.

To help you prepare the presentation, we will devote class time to discussing the components of a good
scholarly talk. I also encourage you to attend other seminars around campus during the semester to see
examples. Finally, I strongly encourage you to practice your presentation in advance. Even better to practice
in front of some of your classmates. Please let me know if I can be helpful in arranging practice time or
space.

COURSE OUTLINE

Class Topics Readings

1/24 The role that “theory” plays in the design of
research, and the role of experimental
research design in making causal inferences in
quantitative research.
Discussion of Chetty et al. (2011).
Preparing to read Howell et al. (2002) and
Krueger & Zhu (2004).
Methods Matter chapters 1-3
Chetty, R., Friedman, J.N., Hilger, N., Saez,
E., Schanzenbach, D.W., Yagan, D. (2011).
How Does your Kindergarten Classroom
Affect your Earnings? Evidence from Project
Star. Quarterly Journal of Economics, 126(4),
1593-1660.

4
S290 Syllabus (1/3/2017), Prof. Eric Taylor
1/31 Recap of experimental design.
Issues with random assignment experiments,
with an application to the New York City
School Voucher Experiment.
Preparing to read Tyler et al. (2000) and
Dynarski (2003).
2/7 Recap of natural experiments and DID
methodology.
Discussion of Tyler et al. (2000) and Dynarski
(2003).
Preparing to read Angrist & Lavy (1999).
2/14 Recap of regression discontinuity designs.
Discussion of Angrist and Lavy (1999).
Preparing to read Ludwig and Miller (2007).
2/21 Recap of estimation and interpretation of RD
designs.
Discussion of Ludwig and Miller (2007).
Introduction to instrumental variables – focus
on inference and estimation.
Preparing to read Currie and Moretti (2003)
and Dee (2004).
5
Methods Matter chapters 4-5
Howell, W.G, Wolf, P. J., Campbell, D. E., &
Peterson, P. E. (2002). School Vouchers And
Academic Performance: Results From Three
Randomized Field Trials. Journal of Policy
Analysis and Management, 21(2), 191-217.
Krueger, A. B. & Zhu, P. (2004). Another
Look At The New York City School Voucher
Experiment. The American Behavioral
Scientist, 47(5), 658-698.
Methods Matter chapter 8
Tyler, J. H., Murnane, R. J., & Willett, J. B.
(2000). Estimating The Labor Market
Signaling Value of the GED. Quarterly
Journal of Economics, 115(2), 431-468.
Dynarski, S. M. (2003). Does Aid Matter?
Measuring The Effect Of Student Aid On
College Attendance And Completion. The
American Economic Review, 93(1), 279-288.
Methods Matter chapter 9, pp. 165-181
Angrist, J. D., & Lavy, V. (1999). Using
Maimonides’ Rule To Estimate The Effect Of
Class Size On Scholastic Achievement.
Quarterly Journal of Economics, 114(2), 533-
575.
Methods Matter chapter 9, pp. 181-202
Ludwig, J. & Miller, D. (2007). Does Head
Start Improve Children's Life Chances?
Evidence from a Regression Discontinuity
Design. Quarterly Journal of Economics,
122(1), 159-208.
S290 Syllabus (1/3/2017), Prof. Eric Taylor
2/28 Recap of instrumental variables estimation
(IV).
Discussion of Dee (2004) and Currie and
Moretti (2003).
Preparing to read Abdulkadiroglu et al.
(2011).
3/7 Recap of interpretation and research design
issues with IV.
Discussion of the Abdulkadiroglu et al. (2009)
paper with a focus on the interpretation of ITT
and LATE parameters, and the scaling issues
in IV/2SLS models.
Introduction to methods for dealing with bias
in observational / non-experimental data,
including fixed effects and propensity score
estimation.
Preparing to read Taylor and Tyler (2012) and
Smith and Todd (2005).
3/21 Recap of methods for dealing with bias in
observational data.
Discussion of Taylor and Tyler (2012).
Discussion of Smith and Todd (2005), with an
emphasis on propensity score matching
methods and the sensitivity of results.
Brief introduction to estimation of teacher
“value-added” models.
Preparing to read Kane and Staiger (2008),
and Chetty, Friedman and Rockoff (2014a,
2014b).
6
Methods Matter chapter 10
Dee, T. S. (2004). Are There Civic Returns to
Education? Journal of Public Economics,
88(9-10), 1697-1720.
Currie, J. & Moretti, E. (2003). Mother’s
Education and the Intergenerational
Transmission of Human Capital: Evidence
from College Openings. Quarterly Journal of
Economics, 118(4), 1495-1532.
Methods Matter chapter 11
Abdulkadiroglu, A., Angrist, J., Dynarski, S.,
Kane, T., & Pathak, P. (2009). Accountability
and Flexibility in Public Schools: Evidence
from Boston’s Charters and Pilots. Quarterly
Journal of Economics, 126(2), 699-748.
Methods Matter chapter 12
Taylor, E. S. & Tyler, J. H. (2012). The Effect
of Evaluation on Teacher Performance.
American Economic Review, 102(7), 3628-
3651.
Smith, J. A., & Todd, P.E. (2005). Does
Matching Overcome Lalonde’s Critique of
Nonexperimental Estimators? Journal of
Econometrics, 125, 305-353.
S290 Syllabus (1/3/2017), Prof. Eric Taylor
3/28 Putting it all together – discussion of causal
inference, methodology, choice of
specification and external validity issues with
the example of teacher “value-added”
estimation.
Discussion of Kane and Staiger (2008), and
Chetty, Friedman and Rockoff (2014a,
2014b).
Preparing to read Deming (2011) and
Deming, Hastings, Kane and Staiger (2014)
and associated referee reports.
4/4 Discussion of the peer review process, using
Taylor and Tyler (2012) and Taylor (2014) as
an illustrations.
Tips on preparing conference presentations.
Tips on preparing a paper for publication.
4/11 Presentations of original work conducted by
4/18 class members during the course. Other class
4/25 members will provide feedback.
OTHER CONSIDERATIONS
Prerequisites. S-290 is designed for HGSE doctoral students who have completed at least S-052: Applied
Data Analysis or its equivalent (please see me if you have not completed S-052), and ideally more advanced
quantitative training in a disciplinary department such as economics, psychology or statistics. Participants
should also be familiar with the basic concepts of microeconomics, because these concepts are helpful in
understanding the required readings.
7
Methods Matter chapter 13
Kane, T.J., & Staiger, D.O. (2008).
Estimating Teacher Impacts on Student
Achievement: An Experimental Evaluation.
NBER Working Paper 14607.
Chetty, R., Friedman, J.N., & Rockoff, J.E.
(2014a). Measuring the Impact of Teachers I:
Evaluating Bias in Teacher Value-Added
Estimates. American Economic Review,
104(9), 2593-2632.
Chetty, R., Friedman, J.N., & Rockoff, J.E.
(2014b). Measuring the Impact of Teachers II:
Teacher Value-Added and Student Outcomes
in Adulthood. American Economic Review,
104(9), 2593-2632.
Read original submissions to journals, the
peer reviews, the correspondence with the
editors of the journals, and the final accepted
version of the papers.
Taylor, E. S. & Tyler, J. H. (2012). The Effect
of Evaluation on Teacher Performance.
American Economic Review, 102(7), 3628-
3651.
Taylor, E. S. (2014). Spending More of the
School Day in Math Class: Evidence from a
Regression Discontinuity in Middle School.
Journal of Public Economics, 117, 162-181.
S290 Syllabus (1/3/2017), Prof. Eric Taylor

Enrollment. Enrollment is limited to 20 students, and priority is given to HGSE doctoral students. Auditing
is not allowed.

Academic Integrity and Plagiarism. Please make sure to read the HGSE policy on plagiarism, presented
in the Student Handbook. Pay particular attention to the notion that “in the preparation of all papers and
other written work submitted…a student must be careful to distinguish between ideas that are his or her
own and those that have been derived from other sources. Information and opinions drawn from all sources
are to be attributed specifically to these sources. It is the student’s responsibility to learn and use the proper
forms of citation.…Students who submit work either not their own or without clear attribution to the
original source, for whatever reason, face sanctions up to and including dismissal and expulsion” (2014,
pp. 54-55, my emphasis).

In addition to citation of published work, it is also traditional in academic papers to document help and
substantial feedback received from others (including the course instructor and classmates) usually in the
form of a footnote at the beginning of the paper.

Academic Accommodations. We encourage students needing accommodations in instruction or evaluation

to notify us early in the semester. If you have a disability or health concern that may have some impact on
your work in this class and for which you may require adjustments or accommodations, please contact
Eileen Berger bergerei@gse.harvard.edu, Access and Disability Services (ADS) administrator in Gutman
124. No accommodations can be given without authorization from ADS, or without advance notice. If you
already have a Faculty Contact Form for this course from ADS, please provide us with that information
privately in our offices so that we can make those adjustments in a timely manner. All inquiries and
discussions about accommodations will remain confidential.