Impact Evaluation of Development Interventions: A Practical Guide

IMPACT EVALUATION OF DEVELOPMENT INTERVENTIONS

A Practical Guide

Howard White

David A. Raitzer

Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO)

© 2017 Asian Development Bank

6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines

Tel +63 2 632 4444; Fax +63 2 636 2444

www.adb.org

Some rights reserved. Published in 2017.

Printed in the Philippines.

ISBN 978-92-9261-058-6 (print), 978-92-9261-059-3 (electronic)

Publication Stock No. TCS179188-2

DOI: http://dx.doi.org/10.22617/TCS179188-2

The views expressed in this publication are those of the authors and do not necessarily reflect the views and policies of the Asian Development Bank (ADB) or its Board of Governors or the governments they represent.

ADB does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. The mention of specific companies or products of manufacturers does not imply that they are endorsed or recommended by ADB in preference to others of a similar nature that are not mentioned.

By making any designation of or reference to a particular territory or geographic area, or by using the term country in this document, ADB does not intend to make any judgments as to the legal or other status of any territory or area.

This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) https://creativecommons.org/licenses/by/3.0/igo/. By using the content of this publication, you agree to be bound by the terms of this license. For attribution, translations, adaptations, and permissions, please read the provisions and terms of use at https://www.adb.org/terms-use#openaccess.

This CC license does not apply to non-ADB copyright materials in this publication. If the material is attributed to another source, please contact the copyright owner or publisher of that source for permission to reproduce it. ADB cannot be held liable for any claims that arise as a result of your use of the material.

Please contact pubsmarketing@adb.org if you have questions or comments with respect to content, or if you wish to obtain copyright permission for your intended use that does not fall within these terms, or for permission to use the ADB logo.

Photos in this publication are the property of ADB.

Notes:

In this publication, $ refers to US dollars.

Corrigenda to ADB publications may be found at http://www.adb.org/publications/corrigenda.

Contents

Tables, Figures, and Boxes

Tables

Figures

Boxes

Foreword

Two important trends are shaping the future of development assistance, particularly in Asia. First, developing countries are ever more able to access finance from a wider variety of sources including private ones. Second, policy makers are increasingly attuned to evidence that can make programs more effective. This means that the value proposition of development agencies, such as the Asian Development Bank (ADB), increasingly depends on the ability to offer knowledge, rather than finance alone. A critical element of this knowledge is derived from evidence on the intended and unintended effects of interventions. Impact evaluation is the main means for empirically testing what actually happens when interventions are implemented.

In parallel to these trends, behavioral economics has increasingly attracted interest within the field of development economics. Prior to the past 2 decades in the field, human behavior was often largely assumed to mechanistically follow neoclassical assumptions. In the period since, economists have increasingly recognized the need to go back and test whether those assumptions hold, by using experimental and quasi-experimental impact evaluation techniques that have been largely pioneered in medicine and other science fields.

This convergence of rising need for impact evaluation evidence among development practitioners and increased interest among academics presents a unique opportunity for intersecting research and practice. Impact evaluation can attract some of the world’s leading economic talent to engage with specific development projects. Such engagement not only leads to rigorous new evidence on what works in development, but also directly enhances project implementation. Leading researchers who have worked across many countries and programs often have insights that can contribute to better intervention design during impact evaluation conceptualization. Impact evaluation also necessitates that project results logics and underpinning assumptions are clarified before they are tested, making projects better designed.

Impact evaluation can help bring the types of positive feedback that have been routinely used in product development in the private sector. Evidence can offer a rationale for continuing or expanding effective projects and programs, regardless of political environment. Impact evaluation can test different ways of tackling a problem, identify what factors condition intended effects, and provide insights on how interventions should be rolled out and combined. Impact evaluation also offers a platform for generating proof of concept for innovations. Ultimately, it can also help build fundamental theories about human behavior and development, shift conventional wisdom, and reorient development toward more effective approaches.

For this to happen, impact evaluation needs to be mainstreamed, so that development practitioners regularly consider what prior impact evidence implies for their activities, as well as how impact evaluation of their interventions can help contribute to that evidence. This book is intended to help in this mainstreaming by serving as an accessible reference for a range of audiences, backed by years of experience in implementing impact evaluation studies. For non-economist audiences, it offers lay descriptions of core concepts, introductions to key methods, and rules of thumb for understanding technical topics, such as power calculation. For more academic audiences, it offers more detailed descriptions of techniques and introductions to STATA commands in technical appendixes. Compared with previous texts, this book introduces a greater range of methodologies, as well as more description of practical considerations.

ADB is currently expanding its impact evaluation coverage, and is devoting increased attention and resources to new impact evaluation studies. To backstop this process, this book is a valuable resource that can help to increase awareness of what impact evaluation offers and how it can be applied. I recommend it as a practical resource for those who have interest in generating or using rigorous evidence on what works in development.

Yasuyuki Sawada

Chief Economist and Director General

Economic Research and Regional Cooperation Department

Acknowledgments

This book reflects the contributions of many individuals within and outside of the Asian Development Bank (ADB). It has been produced under the overall guidance of Edimon Ginting, Director of the Economic Analysis and Operations Support Division and Rana Hasan, Director of the Development Economics and Indicators Division. The volume has also benefited from overall orientation by ADB’s interdepartmental Impact Evaluation Committee. Howard White, independent consultant, and David Raitzer, Economist, Economic Analysis and Operations Support Division, have authored the contents.

Sakiko Tanaka provided valuable insights and contributions to early versions of this book. Background materials were also provided by Scott Rozelle of Stanford University, and some sections drew on materials prepared by independent consultant Nina Blöndal. Additional inputs and contributions were provided by Jasmin Sibal and Marie Anne Cagas. Administrative support has been provided by Lilibeth Poot, Amanda Mamon, Gee Ann Burac, Ricasol Calaluan, Roslyn Perez, and Glennie Castillo. Valuable peer reviews have been provided by Impact Evaluation Committee Members Ari Perdana, Artur Andrysiak, Arturo Martinez, Bernard Woods, Christopher Edmonds, Elisabetta Gentile, Joao Fahrina, K. E. Seetharam, Kiyoshi Taniguchi, Lakshman Nagraj Rao, Lars Johannes, and Maya Vijayaraghavan. Initial conceptualization was under the overall guidance of Cyn-Young Park, former Director of the Economic Analysis and Operations Support Division.

Tuesday Soriano copyedited the manuscript, and Joe Mark Ganaban prepared the layout. This book is produced under Technical Assistance 0012-REG: Developing Impact Evaluation Methodologies, Approaches, and Capacities in Selected Developing Member Countries.

Abbreviations

Chapter 1

Introduction: Impact Evaluation for Evidence-Based Development

Key Messages

• Impact evaluation empirically estimates the effects attributable to a specific intervention and the statistical significance of those effects.

• Deriving reliable knowledge and evidence from development operations depends on impact evaluation.

• Impact evaluation can serve a number of roles. It can determine not only whether an intervention is effective, but it can also compare options for making interventions more effective.

• Evidence from impact evaluation can inform assumptions underpinning economic analysis of specific investments, as well as broader strategies for sectors, regions, and countries.

1.1 Why Does Impact Evaluation Matter?

Development organizations have an ultimate mandate to contribute to development goals. For example, Strategy 2020 of the Asian Development Bank (ADB) reaffirms ADB’s vision of an Asia and Pacific region free of poverty. ADB’s mission is to help developing member countries improve living conditions and the quality of life of their citizens. To this end, billions of dollars of funding are mobilized each year. What have been the impacts of the funded programs?

The answer to this question requires evidence that is produced by counterfactual impact evaluations (IEs). Without IE, it is not possible to ascertain the causal effects of development interventions. In the absence of understanding what effects have occurred as a result of development efforts, it is neither possible to keep accountability about development expenditures, nor to derive meaningful knowledge from development operations to improve development policies.

Impact evaluations are empirical studies that quantify the causal effects of interventions on outcomes of interest. This is far different from traditional process evaluations that are concerned with characterizing how projects were implemented. IEs are based on analysis of what happened with an intervention, compared with an empirically estimated counterfactual scenario of what would have happened in the absence of the intervention. This difference between the observed outcomes and the counterfactual outcomes is the measure of impact, i.e., the difference that can be attributed to the intervention. Effects can be quantified at any level and, contrary to popular perception, do not need to concern only long-term goals or impacts in the jargon of logical frameworks. At the same time, IE is the only method that can provide evidence as to those long-term effects.

IE is unique in that it is data driven and attempts to minimize unverifiable assumptions when attributing effects. A core concept is that identified impacts are assessed not only in magnitude, but also in terms of statistical significance. This approach is not to be confused with impact assessment, which often includes modeling rooted in taking structural and often neoclassical assumptions about behavior as given, and which cannot ascertain statistically significant effects.

Development assistance’s drive toward evidence-based policy and project design and results-based management depends on mainstreaming IE. IE allows for assumptions underpinning the results logic of interventions to be tested and for previously unknown consequences to be revealed.

At the heart of evidence-based policy is the use of research results to inform and supplant assumptions as programs and policies are designed (Sanderson 2002). In turn, this depends on the generation of new evidence on effectiveness, and the incorporation of evidence into program conceptualization. One linkage by which this can be achieved is by informing economic analysis of investments. IE validates and quantifies the magnitude of the effects of an intervention, and these effect magnitudes are critical to understanding project benefits. The impact findings for one intervention can inform the economic analysis for a follow-on project to scale up the investment, or for similar investments elsewhere.

One of the best known examples of evidence-based policy in international development has been the growth of conditional cash transfers (CCTs) in Latin America (Box 1.1). Similarly, an ADB-supported IE of the Food Stamps Program in Mongolia played a part in persuading the government to scale up the program (ADB 2014).

Box 1.1: The Use of Evidence from Impact Evaluations to Inform the Spread of Conditional Cash Transfers in Latin America

The conditional cash transfer (CCT) program, PROGRESSA, was started by the Mexican government in the mid-1990s. The government decided to build a rigorous, randomized evaluation into the program design. The study showed the positive impact of CCT on poverty and access to health and education. These findings meant that the program survived a change in government with just a change in name. A similar story can be told about Colombia’s CCT, Familias en Acion. In Brazil, the President commissioned an impact evaluation of the Bolsa Familia program to be able to address critics of the program, especially those who argued that it discouraged the poor from entering the labor market. The study showed it did not, and Bolsa Familia continued to expand, reaching over 12 million families by 2012.

Source: Behrman (2010).

The IE movement has spread across the world and across sectors (Figure 1.1). A database of over 4,000 development IEs shows this rapid growth, with 500 new studies a year by 2015. Most of these studies are in the social sectors, but there are growing numbers for many other topics, such as rural electrification, water supply, and transportation.

Figure 1.1: Annual Publication of Impact Evaluations

Sources: Cameron, Mishra, and Brown (2016); authors’ estimates from the International Initiative for Impact Evaluation (3ie) impact evaluation repository.

1.2 The Purposes