3IE Case Studies - Conceptual - Framework

Conceptual Framework for 3ies Case Studies Series on Policy Influence of Impact Studies
Sound Expectations. From Impact Evaluations to Policy Change

Conceptual Framework for 3ies Case Studies Series on policy influence of impact studies
Contents
Acknowledgements ............................................................................................................................. 4 About CIPPEC....................................................................................................................................... 4 Abstract ............................................................................................................................................... 5 1- Introduction ................................................................................................................................ 6 2- Policy influence as an objective of research and impact evaluations ......................................... 8 The importance of influence goals ................................................................................................ 10 Levels of impact......................................................................................................................... 14 Dimensions of impact................................................................................................................ 15 Timing and geographical location ............................................................................................. 16 3- Factors and forces ..................................................................................................................... 19 a) Determinants for the demand of research in the policy process ........................................ 19 The context................................................................................................................................ 19 Policy actors and policy communities ....................................................................................... 24 Purposes and commissioners .................................................................................................... 29 b) Determinants for the supply of research in the policy process ............................................ 31 Contextual characteristics ......................................................................................................... 31 Organizational and individual characteristics ........................................................................... 31 Research (impact evaluations) characteristics ......................................................................... 35 c) Determinants for the intersection of demand & supply ....................................................... 40 Policy processes ........................................................................................................................ 41 Bridging communications ......................................................................................................... 42 The interplay of supply, demand & policy influence objectives ............................................... 46 4- Final observations: Measuring the real impact of evaluations ................................................. 49 Bibliography ...................................................................................................................................... 51 Authors .............................................................................................................................................. 54
Tables and graphs

Table 1: Jones, Jones, Steer and Dattas systematization of the conceptualization of evaluation use ................................................................................................................................................... 12 Table 2: Factors, forces and hypotheses regarding the context ....................................................... 23 Table 3: How consensus on the policy base, number of actors and type of information should logically change for different decision regimes ................................................................................ 28 Table 4: Factors, forces and hypotheses regarding policy actors and policy communities .............. 29 Table 5: Factors, forces and hypotheses regarding the commissioning and purposes of IEs........... 30 Table 6: Factors, forces and hypotheses regarding research supply. ............................................... 40 Table 7. Factors, forces and hypotheses regarding the interplay of demand and supply. ............... 46

Graph 1: Types of policy impact objectives: Levels and dimensions ................................................ 13 Graph 2: Lindquists organizational capacity approach in developing, designing and implementing policy ................................................................................................................................................. 26 Graph 3: Supply and Demand forces and factors coming together .................................................. 47 Graph 4. Channels for IEs policy influence ....................................................................................... 48

Acknowledgements
The authors will like to especially thank Harry Jones (Overseas Development Institute, ODI) for his comments and suggestions throughout the production of this framework. Furthermore, the authors will also like to thank Arnaldo Pellini (ODI), Azra Abdul Cade (Centre for Poverty Analysis, CEPA), Enrique Mendizabal (ODI), Charles Owino (African Economic Research Consortium, AERC), Fletcher Tembo (ODI) and Dr. Marie Gaarder (3ie) and her team for their valuable reviews, comments and suggestions. The authors would like to acknowledge Clara Richards (CIPPEC) help during the production of this conceptual framework. This paper is a conceptual framework to be used in the production of six case studies which will be called by the International Initiative for Impact Evaluation (3ie). Its production was led by CIPPEC, with the participation of ODIs Research and Practice in Development (RAPID) Programme, UKs leading development think tank and some members of the Evidence based Policymaking in Development Network (EBPDN), a worldwide community of practice for think tanks, policy research institutes and similar organizations working in international development, to promote more evidence-based, pro-poor development policies.
About CIPPEC
CIPPEC (Centre for the implementation of public policy for equity and growth) is an independent, non partisan and nonprofit organization that works for a fair, democratic and efficient state that point at a better life for its citizens. Accordingly, CIPPEC concentrates its efforts in analyzing and promoting public policies that foster equity and growth in Argentina. Its main challenge is to translate into concrete actions the best ideas that come from the Areas of Social Development, Economic Development, Strengthening of the Institutions and Public Management, through the programs of Education, Health, Social Protection, Fiscal Policy, International Insertion, Justice and Transparency, Local Development, and Government Politics and Management (www.cippec.org).

Abstract
This paper outlines a comprehensive and flexible analytical conceptual framework to be used in the production of six case studies which will be called by the International Initiative for Impact Evaluation (3ie). The cases are expected to identify factors that contribute to or hinder evidence from rigorous impact evaluations (IEs) to influence policy, thereby improving the policy effectiveness. Therefore, this framework is developed in order to be adaptable to the reality of developing countries and is aimed to act as an analytical-methodological tool which should enable researchers, when producing the case studies, to identify the factors that affect and explain IEs policy influence potential. The approach should also enable comparison between cases and regions to draw lessons that are relevant beyond the cases themselves. There are two different (though interconnected) issues that must be approached when discussing IEs policy influence. The first issue has to do with the type of policy influence pursued and, aligned with this, the determination of the accomplishment (or not) of the intended influence. In this paper, we first introduce the discussion regarding the different types of policy influence objectives that impact evaluations usually pursue, which will ultimately help determining whether policy influence was indeed achieved. This discussion is mainly centered in whether an IE has had impact on policy. The second issue is related to the identification of the factors and forces that mediate the policy influence efforts and is focused on why the influence was achieved or not. We have identified and systematized the mediating factors and forces, and we present them in this paper, being these approached from the demand, the supply and considering as well, the intersection between these two. The paper concludes that, ultimately, the fulfillment of policy change based on IEs results is determined by the interplay of the policy influence objectives with the factors that affect the supply and demand of research in and to the policymaking process. The paper is divided in four sections. Beginning with a brief introduction, follows an analysis of the policy influence as an objective of research and, specifically, impact evaluations. A third section identifies the factors and forces that enhance or undermine influence in public policy decision making. The research ends up pointing out the importance of measuring policy influence and enumerates a series of challenges that have to be further assessed.

1- Introduction
To which extent can evidence from evaluations really affect the policymaking process? What are the purposes and potential roles of impact evaluations in the design and implementation of new policies? How can those who commission and produce impact evaluations improve their potential use and value? Which are the factors and forces that determine impact evaluations uptake in policymaking processes? We believe that answering these questions can help to increase existing knowledge on the real benefits of producing impact evaluations. The purpose of this paper is to review academic literature relevant to these questions and develop a conceptual framework that will guide the production of a series of case studies on policy influence of impact studies by 3ie. There is a set of multiple and assumed benefits of evaluating policies. Knowledge about the impact of certain policy programs will provide information to policymakers as to whether the intervention is contributing to the achievement of the stated results. Furthermore, carefully designed and implemented evaluations can ultimately improve peoples welfare and enhance development effectiveness (Gaarder and Briceo, 2010). Consequently, a frequent aim of impact evaluations is to contribute to the development, adoption and amendment of policy. However, and as Weiss (1999) points out, this is quite difficult to achieve. Policy influence of IEs (as well as of other types of research) is not straightforward. The potential advantages of incorporating impact evaluations conclusions and recommendations in the policymaking process are mediated by a number of factors and forces that can make it easier or more difficult to attain. There are two different (though interconnected) issues that must be approached when discussing IEs policy influence. The first issue has to do with the type of policy influence pursued and aligned with this the determination of whether that intended influence was accomplished. In this paper, we first introduce the discussion regarding the different types of policy influence objectives that impact evaluations usually pursue, which will ultimately help determining whether policy influence was indeed achieved. This discussion is mainly centered in whether an IE has had impact on policy or not. The second issue is related to the identification of the factors and forces that mediate the policy influence efforts and is focused on why that influence was achieved or not. Though tightly intertwined these questions pose diverse challenges and factors that should be considered to answer them. We have identified and systematized the mediating factors and forces, and we present them in this paper in an organized way to help conceptualization of the obstacles and opportunities when aiming to influence policymaking with impact evaluations. Under this framework, this paper presents a summarized conceptual review of part of the vast body of literature to shed light on these two main questions regarding IEs potential influence on policy processes. We believe that reviewing both of these issues will be useful for the production of the case studies on policy influence of IEs for two main reasons. Firstly, the acknowledgement

of different types of policy influence objectives will help in the determination of the degree and type of policy influence achieved by IEs in each case study, and ultimately, whether IEs policy influence objective was fulfilled. Consequently, it will enable the authors to answer the following questions: Was policy influence achieved? If so, what kind of policy influence was achieved? Secondly, the identification of the factors and forces will help explaining why policy influence was achieved. In determining the factors that hinder or promote policy influence 3ie, through the commissioning of the case studies, will be able to advance in illustrating some of the causal relations that intercede in the bridge between research (and particularly, impact evaluations) and policy. Following this brief introduction, the second section of the paper presents the discussions regarding policy influence as an objective in research (especially IEs), therefore addressing the first issue stated above. In the third section, we focus on the second issue: the various factors and forces that intermediate between impact evaluations and policymaking. We have divided these factors into three groups1: (a) those linked to the demand of evidence from impact evaluations to incorporate into the decision-making process; (b) those linked to the supply of IEs and (c) those related to the junction between the demand and the supply. Finally, in the fourth section, we present some preliminary considerations that should be acknowledged by 3ie when commissioning the case studies series and when trying to measure the effective influence of impact studies on the policy process. For the concrete conduction of the case studies, another document has been produced: Guidelines for the production of 3ies Case Studies Series on Policy Influence of Impact Studies. These guidelines are based on the present conceptual framework and are delivered separately to 3ie.
We explain the reasons behind this classification in the third section.

2- Policy influence as an objective of research and impact evaluations

Social science research traditional objective has been to better understand reality. However, for the last couple of centuries, a second objective has been incorporated: to influence public policy. This goal has proven to be very difficult to attain (Weiss, 1999). The advice emerging from social sciences on policy issues has been generally ignored, which has in turn led to questioning the efficacy and relevance of its claims. Therefore, a quite important volume of literature on knowledge utilization has grown, which tries to explain the distance between the knowledge producers (social scientists) and knowledge consumers or users (policymakers) communities (Lindquist, 2001). The divide between these two communities, regarding the generation of knowledge and its uptake in policy, has been disputed from various social sciences perspectives, and while it is useful to still use this distinction (Jones, 2009), it is also important to keep in mind that these categories are malleable, flexible and often juxtapose since they are conformed by individuals who can have multiple roles. The mediated relation between research and policy can vary depending on what we understand by both research and policy. In the body of literature which analyses the links between research and policy, both terms are used quite loosely. Before delving more deeply into this field, we need to clearly define some main concepts that will be used in this paper. Research refers to an expanded notion from the rigorous and thorough research methods of social sciences. It may include policy relevant data having been obtained through analyses that dont necessarily follow a rigorous research methodology. In this regard, Lindquist (1989, 1990 as quoted in Lindquist, 2001) suggests that there are distinctions that should be made about the types of information that are being used, depending on their actual value-added contributions. The generation of research, understood loosely, as relevant information for the policy process, can be therefore categorized in three groups: the production of primary data, research (strictly understood) and analysis. Besides these activities, there are other disseminating actions that can be carried out in order to foster a greater proximity between research and policy: publication activities (i.e. production of memos, articles, policy briefs, etc.) and convocation activities (i.e. workshops, conferences, dialogues between researchers and policymakers, etc.) (Lindquist, 2001). The different types of research (i.e. qualitative, quantitative or a combination of both) have different implications in terms of the policy influence that could be achieved2. This definition of the concept of research is not necessarily applicable when considering the policy influence of impact evaluations (IEs). These evaluations assess the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well
2
We address this issue with greater detail in section 3.b.

as ideally the unintended ones (World Bank Poverty Group on Impact Evaluation, 2008). In spite of this broad definition, there are multiple methodological and epistemological debates regarding impact studies, which certainly affect and determine what is understood by IEs. Currently, as argued by Jones et al. (2009), these debates are being dominated by the perception that IEs refer to evaluations that assess the effects of interventions using a counterfactual which tends to be assessed with experimental or quasi-experimental methods. However, there are alternative approaches to impact evaluations, besides the counterfactual model (such as generative causality or the configurational approach3). And while we will not enter this discussion, we believe it is important to keep in mind these debates4, and the fact that what is understood by IEs is not always the same, when discussing the possible uptake in policy process of results. Furthermore, these evaluations are set apart from the rest of social research, since they are indeed characterized by being produced through more rigorous methods. However, it is important to notice that around IEs and with the information produced by them, these other research activities can be held, that do not follow a strict quantitative research method but which can and do affect policy decisions. The strict methodologies with which IEs are conducted mean that there must be certain preconditions prior to the production of impact studies. Impact evaluations can only be done when the project or policys activities impact can be measured, which necessarily requires that the project or policy has been implemented for a while. We will delve more deeply in addressing IEs characteristics that might determine its policy influence potential in section 3.b. Policy encompasses both decisions and processes, including the design, implementation and evaluation of the intervention. Policy is defined as a purposive course of action followed by an actor or set of actors (Anderson, 1975, in Pollard & Court, 2005). This definition, of course, goes beyond documents or legislation, to include agenda setting, policy formulation, decision making, policy implementation and policy evaluation activities. An important observation is that policies are not restricted to governmental courses of action, but also could include those of international organizations, bilateral agencies and civil society organizations (CSOs). There is a vast body of literature regarding policy and policy processes. It should be noted that it is not our intention to review this literature or to repeat it here5. We will only stress that we do not consider policymaking as a linear process, nor as a logical sequence. As tempting as it could be to think of
Generative causality involves identifying underlying processes that lead to change. For example, the use of qualitative methods to assess causality by understanding peoples operative reasons for their actions or behavior change (Bhola in Jones et al, 2009). Another type takes a configurational approach to causality, in which outcomes are seen to follow from the combination of a fruitful combination of attributes (Pawson in Jones et al 2009).
4
Which will also be presented with greater depth in section 3.b.
We take this issue up again in section 3.c. Many authors have widely reviewed this topic. For a review of recent debates, we recommend Paul Sabatiers recent book Theories of the policy process (2007).

policymaking as an orderly sequential process, this tends to be misleading when trying to achieve policy influence.
The importance of influence goals

The broadness of the above defined terms naturally drives to a very general question: How can research emerging from IEs influence public policy? There are multiple ways in which this may (or may not) happen and this heterogeneity should not be underestimated when analyzing the intersection between research and policy. Confusion on what is considered influence can result into contradictory meanings as regards whether a piece of evidence has affected a policy decision or not. Therefore, in this paper we highlight the importance of digging more deeply into the policy influence goals. Defining policy influence objectives will determine whether influence was ultimately achieved and, if so, which type of influence was it. In this sense, it is first fundamental to determine in each evaluation effort the type of impact that might be achieved which is intrinsically linked to the way it is then used. Why is this particular evaluation conducted? By whom has it been commissioned and why? What are the specific aims to be altered? How were the results used? In this direction, we will next explore the different types of use, levels and dimensions into which diverse authors have categorized policy objectives. Our goal is to cover all potential types of objectives in order to highlight the complexity of potential impact of IEs. Weiss (1999) states that expectations for the uses of evaluations are centered in bringing order and rationality to the making of policy:
By finding out the results of government interventions they would: (i) help governments decide whether to continue or terminate particular policy initiatives; (ii) expand and institutionalize successful programmes and policies and cut back unsuccessful ones; and (iii) figure out which programmes to modify and which components of the programme were in need of modification.
Nonetheless, Weiss also highlights that these expectations are hardly ever met. However, she argues that, even though research was not necessarily directly relevant to policy decisions, it could achieve influence in other important ways, through what she called enlightenment. This referred to the percolation of new information, ideas and perspectives into the arenas in which decisions are made. In turn, the incorporation of new insights might lead to attitudinal changes, such as alteration in the language and the perspectives adopted by policymakers (and/or their advisors) on different issues. These changes, though much more difficult to determine, are more cultural, operational and long-termed than a concrete impact on a particular policy (Weiss, 1999). Along these lines, Behrman (2010), in his study of the influence and impact of IFPRIs evolution on the Progresa/Oportunidades Mexican CCT, states that 10

it is rare indeed, and perhaps never happens, that IFPRIs research alone results in any particular policy change. Instead, IFPRIs research is likely to contribute, together with the research of other institutions and individuals, to a changing environment of understanding as to how various policies work, and which work better than others. (p. 1475).
We will take up this issue again and link it with the potential uses of impact evaluations later on. What is important at this stage is to determine the different types of impact that the evaluations aim to achieve, in order to consequently identify the concrete factors and forces that interplay related to the specific policy influence goal. To do so, we will present a review of different typologies regarding the types of policy influence that IEs may achieve, and we will then present a new typology we have created inspired in other authors work. Lindquist (2001) presents a very interesting and complete typology of policy influence objectives considering its capacities, horizons and regimes, and which goes beyond the three traditional uses described above by Weiss. Thus, he argues that policy influence may range from improving the knowledge of certain actors (and therefore expanding policy capacities) to fundamentally redesigning policies (affecting policy regimes): Expanding policy capacities o Improving the knowledge/data of certain actors o Supporting recipients to develop innovative ideas o Improving capabilities to communicate ideas o Developing new talent for research and analysis Broadening policy horizons o Providing opportunities for networking/learning within the jurisdiction or with colleagues elsewhere o Introducing new concepts to frame debates, putting ideas on the agenda, or stimulating debate o Educating researcher and other who take up new positions with broader understanding of issues o Stimulating quiet dialogue among decision-makers Affecting policy regimes o Modification of existing programs or policies o Fundamental re-design of programs or policies
Jones et al (2009) have systematized the various ways in which evaluation use has been conceptualized:
11

Table 1: Jones, Jones, Steer and Dattas systematization of the conceptualization of evaluation use
Author Sandison (2005)
Types of use Instrumental use
Elaboration Involves direct implementation of findings and recommendations to, for example, i) help decide whether to continue or terminate particular policy initiatives; ii) expand and institutionalize successful programmes and policies and cut back unsuccessful ones; and iii) figure out which programmes to modify and which components of the programme were in need of modification Involves evaluations trickling down into the organisation in the form of new ideas and concepts creating debate and dialogue, generating increased clarity and new solutions in the longer run (vanv de Putte, 2001), also providing a catalyst for change Involves learning on the part of the people and management involved in the evaluation Corroborates a decision or understanding that the organization already holds providing an independent reference Where evaluations serve a purely symbolic purpose, representing a desirable organisational quality such as accountability Involves the suppressing, subverting, misrepresenting or distorting of findings for political reasons or personal advantage Is where the evaluation is ignored because users find little or no value in the findings, are not aware, or the context has changed dramatically Underpinned by accountability perspective (summative evaluation, accountability, audits, quality control, cost benefit decisions, decide a programmes future, accreditation/licensing) Underpinned by the developmental perspective (formative evaluation, identify strengths and weaknesses, continuous improvement, quality enhancement, being a learning organisation, manage more effectively, adapt a model locally) Underpinned from the knowledge perspective of academic values (generalisations about effectiveness, extrapolate principles about what works, theory building, synthesise patterns across programmes, scholarly publishing, policymaking) Decision makers have clear goals, seek direct attainment of these goals and have access to relevant information Users base their decisions on a gradual accumulation and synthesis of information Occurs when information or findings are applied directly to change an action or alter a decision Refers to a more intellectual and gradual process in which the decision maker is led to a more adequate appreciation of the problems addressed by the policy or programme This refers to situations where evaluation results are symbolic in that they are carried out simply to comply with administrative directives or to present an image of modernity
Conceptual use
Process use (learning) Legitimising use Ritual use Misuse Non-use Patton (1975) Rendering judgements Facilitating improvemen ts Generating knowledge Marra (2000) Instrumental Enlightenme nt Weiss (1999) Direct Indirect
Symbolic
Source: Jones et al (2009) p. 10.
12

All of these approaches bring in complementary perspectives on the possible use of IEs. A particular evaluation could be used instrumentally (Sandison and Marra), to facilitate improvements (Patton) in a direct way (Weiss), or it could be used to legitimize an understanding (Sandison) by rendering judgments (Patton) in a symbolic way (Weiss). Bringing together these approaches, we established the following main types of impact objectives:
Graph 1: Types of policy impact objectives: Levels and dimensions
1
Source: Authors production inspired on Jones and Villar (2008) and Lindquist (2001).
As can be observed in graph 1, there are different levels and dimensions of policy influence at which IE can aim to. All of them are restricted to a certain time and a particular geography (city, country, region, etc.) and to a specific time frame, which should also be determined. When assessing impact evaluations influence objectives, this should also be taken into account: where and when is the uptake supposed to occur? It is certainly not the same to aim at influencing a social programme at a local level that to seek to influence trade policies at a regional space. This is not to say that an IE cannot have simultaneous influence at different levels and/or dimensions. Most certainly, an IE can have multiple impacts at different dimensions, levels and even geographical locations and timing horizons. Furthermore, an IE can also have negative impacts. However, in order to assess IEs policy influence the type of the intended policy influence objective (determined by its level and dimension) is what should be considered. We explain Graph 1 in the following pages.
13

Levels of impact An IE can have different levels of impact. The notion of levels was inspired by Lindquists (2001) classification of policy influence objectives. The levels range from a particular and specific policy to a broad policy regime. It can aim at influencing a particular project, program or policy. This refers to a concrete public intervention, with a particular objective, defined beneficiary population, budget and set of activities and specific benefits. An example could be a conditional cash transfer program (CCT), such as Comunidades Solidarias Rurales from El Salvador, or a specific component of a Health Program such as the operation of the incentive scheme based on sanitary conditions of the maternal health Plan Nacer in Argentina. Since most of IEs aim at assessing diverse issues in this realm, this level of impact could be the most natural for them. The influence objective of the IE can be targeted either to change a certain aspect of the project, program or policy or to sustain it. For example, IFPRIs evaluation of the Mexican CCT Progresa/Oportunidades has contributed to the short-run and long-run sustainability and expansion of the program (Behrman, 2010, p. 1476). However, impact evaluations can also aim at influencing specific policy areas. In spite of being focused on a particular program or intervention, the goal of an impact evaluation can be to promote this particular type of interventions as the best practice in a specific development field (such as Food Security interventions). In doing so, IEs are in fact intending to broaden policy horizons (Lindquist, 2001), and provide new policy options to decision makers. By highlighting the virtues of a specific intervention, such as a small scale microfinance project in a local level, IEs could aim at presenting it as a policy option at a higher jurisdiction or put it in the public debate agenda as a serious option. Finally, they can also target a whole Policy Regime or System. Sometimes, IEs could aim at presenting recommendations that concern a whole regime. In fact, a specific evaluation on a particular policy can have a wider impact. This was somewhat the case of CCTs impact evaluations and social protection systems. The success of the first CCTs in Mexico and Brazil not only fostered an expansion of CCTs in other countries, but it also altered the social protection system in which they were inserted. CCTs started out as an innovative intervention, with impact evaluation built in as an integral part of the program design from the outset, and nowadays in countries like Brazil, Mexico, Colombia and Argentina, just to name a few, they are in the very core of social protection system and most of all the other programs are engineered around them. The change in the conception6 and institutional architecture of social protection is certainly due to many factors7, but undoubtedly, the success of CCTs is among them (proven through impact evaluations). Impact
6 7
In this way, influence aimed at this level, is similar to Sandisons conceptual use (2005).
Such as these programs high visibility at the national realm (particularly, the endorsement from the Presidential offices) and their high degree of support from the international community (scholars, international organizations and banks).
14

evaluations, not only contributed to evidence the success of CCTs, but they also showed the relevance of the co-responsibilities of these programs in health and education. This, in turn, shifted the focus from the access to health and education services, to their quality, and helped conceiving social protection systems as a combination of various interventions directed to reduce poverty and manage social risks from a multidimensional perspective. As Behrman (2010) points out, there have been substantial spill over effects that were enhanced by OFPRIs involvement. These have occurred both in Mexico and internationally as a result of the evaluation of PROGRESA, on the culture of policy evaluation in general and on CCT programs in particular (p. 1476). This example fits into what Lindquist (2001) describes as affecting policy regimes. There are also some typical combinations of levels and types of impact. For instance, when aiming a direct, instrumental use (as conceptualized by Jones et al, 2009), an IE is generally directed to influence a particular project, program or policy (or a specific component) rather than a policy regime. Therefore, based on its results, operational recommendations are issued (i.e. budget allocation, scale, etc.). As Jones et al. (2009) argue, there are potential problems with this use of evaluations: (1) Clear evidence about whether an intervention is working tend to come with a number of important caveats, qualifications and nuances about what inferences can be drawn and therefore can only be one source of evidence to aide decision making processes and not be able to constitute themselves as a major basis for decision. (2) To be able to compare between different courses of action (diverse interventions or programs) requires IEs to be done of each option. (3) The fact that a pilot or small scale program has been successful does not mean that it should be scaled up, because when doing so a number of variables change. (4) Programs effects on beneficiaries take time as do IEs, and therefore the information and recommendations from the latter may come in too late to influence decision making. Many of these issues are also relevant when an IE aims at influencing a broader policy area or a whole policy regime. Questions of attribution and of assessing other options are especially important when the goal is to have a wider impact, such as the case of social protection systems mentioned before. Therefore, as one moves up the scale, it becomes more difficult to attribute policy responses to individual actors or pieces of research(IFPRI, 2002). Dimensions of impact Besides attempting to influencing narrower or wider policy realms, IEs can try to impact on different dimensions. Following Jones and Villar (2008), we have included five key dimensions of possible policy impact: Framing debates and getting issues on the political agenda: this is about attitudinal change, drawing attention to new issues, affecting the awareness, attitudes or perceptions of key stakeholders.
15

Encouraging discursive commitments from states and other policy actors: affecting language and rhetoric is important, for example promoting recognition of specific groups or endorsements of international declarations. Securing procedural change at domestic or international level: changes in the process through which policy decisions are made for example opening new spaces for policy dialogue. Affecting policy content: while legislative change is not the sum total of policy change, it is an important element. Influencing behaviour change in key actors: policy change requires changes in behaviour and implementation at various levels in order to be meaningful and sustainable.
Many of these dimensions are closely related with what has been conceptualized as the legitimation use and the indirect use of impact evaluations (Jones et al, 2009). The legitimation or symbolic use refers to the use of impact evaluations to primarily justify the actions of a particular organization, project or program, which have already been taken (i.e. to justify the fundraising efforts for a certain intervention). The indirect use, on the other hand, refers to when IEs seek to contribute in a more conceptual way to increasing the understanding of decision makers, create debate and dialogue or make more visible a crucial issue. An interesting approach is to consider the desirability of these policy uses. If an IE is commissioned with a legitimation purpose, then one could suspect a certain bias in its results towards being positive, reinforcing the status quo. Therefore, following Gaarder8, the perceived quality and unbiasedness of the evidence is not only a function of how the IE was conducted, contracted and supervised, but also of the intended policy use underlying its commissioning. Timing and geographical location Besides the level and dimension aimed to be influenced by the results of the IE, timing and geographical location should also be considered. Firstly, when is the impact intended to take place? For instance, are expectations set for changes to take place as soon as the conclusions from the IE are publicly shared? And also, at what stage of the policy cycle is the influence directed? For example, it is different to aim at influencing a new program (policy monitoring stage) than to influence a deeply engrained policy which has been in place for decades9. Secondly, the geographical location should also be defined. If the IE is on a particular project at a national level, but the policy influence goals are set to impact a wider region, IEs results should be
8
In a comment made to this paper. This is also related to Lindquists decision regimes (see table 3), which we explain later.
16

analyzed using other qualitative research methods in order to assess its potential for generalization at a wider geography. Therefore, IEs can aim to have different types of impact and different dimensions. As mentioned before, this is not to say that though aiming at influencing a particular type and dimension, the evaluation cannot also impact on others. For instance, one of the best known applications of impact evaluations, on the Progresa/Oportunidades program (Mexican CCT), looked at assessing its developmental outcomes, focusing on schooling, immunization and child work (Programs components & Policy content). However, its results helped CCT programs visibility in the Latin American Region, thus impacting on a broader type (policy area) and encouraging discursive commitments.
To sum up, IEs may aim at influencing different policy levels in diverse dimensions. In conducting the case study, it is crucial to determine from the outset what type of policy influence objective did the IE have. The policy influence objective may not be explicit. The actors conducting and commissioning the evaluation may also have different expectations regarding IEs policy influence. These underlying notions should be specified and determined, since they ultimately will define whether policy influence was achieved.
The case studies should include:

If so: Did the IE have tacit and/or explicit policy influence objectives? Did all the actors involved in the commission and conduction of the IE share the same objective(s)? What type of use was intended? What level of policy influence objective was sought?: o Project, Program or Policy o Policy Area o Policy System or Regime What policy influence dimension was sought?: o Attitudinal change o Discursive commitments o Procedural change o Policy content o Behavior change
However, the fact that IEs may clearly define from the outset certain policy influence objectives is not enough to guarantee that these will be fulfilled. IEs policy influence is determined by a complex series of factors. In the following section we will systematize and explore these factors
17

that mediate the relationship between the production of an IE and policy influence in the following section.
18

3- Factors and forces

Factors and forces do not only explain ex post how a piece of research may influence or not a policy decision but they should also be considered before establishing policy influence objectives. For example, if our influence objective is to improve the knowledge and debate on a certain issue, factors which will visibly affect this goal are the openness of policymakers to change, the quality and relevance of the produced data, and the reputation of the organization that communicates the evidence, among others. Sometimes, these factors may be identified and assessed from the outset while at other times they may appear as driving forces in later stages. To facilitate the analysis and allow evaluation producers and consumers to more easily assess which set of forces could affect their particular effort, we have categorized recurrent factors and forces analyzed in the existing literature into three groups: a) Those which determine the demand for evidence (based on impact evaluations) in the policy process b) Those which determine the supply of evidence from impact evaluations to the policy process, and c) Those related to the intersection between demand and supply. Needless to say, the boundaries between these three groups are permeable and blurry; therefore there are some issues and factors that affect more than one category simultaneously.
a) Determinants of the demand for research in the policy process

In order to facilitate the analysis of the determinants of the demand for research in the policy process, we have sub-divided them into three groups: contextual variables, variables related to the policy actors and communities and those related to the purposes and commissioners. The context If something is well-known in the developmental studies field, it is that context matters. Indeed, the political, institutional, cultural and structural dimensions of development limit and shape the way in which evidence from impact evaluation might inform the policy process.
a.
b.
What influences policymaking depends on the c. particular moment, policy area and geopolitical level (Crewe and Young, 2002). Overall, what determines whether IEs will be demanded or will be able to impact the policymaking process is greatly set by the entry points allowed by the context. These entry points refer to the places of access within 19
Determinants for the demand of research in the policy process The context Policy actors and policy communities Purposes and commissioners Determinants for the supply of research in the policy process Determinants for the intersection of demand & supply

the political system where information from impact evaluations could be introduced. Clearly, there are other sets of factors (besides the political system itself) which determine and influence these entry points, such as cultural factors contributing to a climate of rationality which could strengthen evidence-based policymaking processes. Weiss (1999) identifies some of these major factors: Democratic system. Democracy tends to be more conducive to the use of research than other political systems, since it is more accessible to inputs from the outside (Weiss 1999, pages 480 - 481). This should be considered especially in developing countries, where democratic systems tend to be more fragile than in developed countries. Competitive political system. Political competition incentivizes more the use of evidence (from evaluations) in the policymaking process. In fierce political competition, impact evaluations (over other types of research) might be in greater demand since they usually provide quantitative data which is considered to have been obtained by rigorous research methods and can be used to put forth or hinder certain political arguments. Again, this should be specially considered in developing countries where political competition tends to be feebler than in developed ones. Decentralization of policy. When the control of policy is not strongly centralized in bodies at the top of the governmental hierarchical structure there are more possibilities of access for the results of evaluation. However, on the other hand, the organisms to which policy is decentralized could also have weaker technical capabilities or formation than the central level and therefore have less interest in IEs. Policymakers capacity to use evidence from IE: The capacity of knowledge users (mostly policymakers, but it could also be brokers or other relevant actors in the policymaking process), affects the way in which evidence from IE can be incorporated in decision making processes. a. Functional specialization of policymakers. When policymakers are specialized experts in the substance of the policy domain, they are more likely to hear about the evaluation findings. However, they are also more prone to rely on their own personal experience than on the studies of evaluators. b. Propensity towards incorporating evaluation explained by educational and professional background. The profession of the policymakers influences their predisposition to accept and incorporate empirical evidence. Moreover, professional prestige of certain professions (such as that of lawyers) can make them less prone to accept advice from other, newer, professions. This, however, could vary for IEs (compared to other social research), since econometrists (who
20

usually conduct the evaluations) tend to be more renowned by other professions than other social scientists. Policymakers belief system. The way in which the policymakers update their beliefs influences the way in which they will receive the evidence from IE. Following Gal and Ruckers (forthcoming, as quoted in Brooks, 2010) research findings, counter-intuitively, when subjects belief systems are confronted with evidence contrary to their beliefs, they become more prone to forcefully advocate in favor of their original beliefs. This suggests that if the evidence from IE is contrary to the policymakers belief system, its effect may be doubly negative, for not only will the policymaker not incorporate its recommendations, but he/she may also argue her previous beliefs more compellingly. What the study does not reveal, however, is whether while arguing more forcefully their original point, people still believe in it in the same way. Perhaps repeated exposure to the new facts from various sources, and/or longer-term studies would reveal that belief systems can be changed. Using Bayesian inference, Mishra (2011) argues that the success of evidence in influencing prior beliefs depends on whether the prior is deterministic/fundamentalist (a tenet), in which case new evidence will have limited or no effect, or whether it is a result of observing a stochastic event (a piece of evidence), in which case new conditional evidence will lead to an update of the prior. Policy Forums. Weiss mentions that Sabatier and Jenkins-Smith (1993) have written about the importance of the existence of a policy forum that can bring together analysts and officials on regular basis. Prestigious, long-termed forums are an ideal arena to set the conditions for the participation of actors who could demand attention to evidence.
Adding to the factors identified by Weiss, one should also include: Existence of an institutionalized system for evaluation. There is not a unique model for institutionalized systems for evaluation; rather, these refer to practices and norms being implemented to guide evaluation efforts at a certain level (country level, provincial or departmental level or local level). Following Gaarder and Briceo (2010), in the successful inception of an institutionalized system for evaluation (building on the cases from Mexico, Chile and Colombia), three factors stand out: the existence of a democratic system with a vocal opposition, the existence of influential monitoring and evaluation (M&E) champions to lead the process, and a clear powerful stakeholder. These authors suggest that the institutionalization of evaluations (in general, including impact evaluations) tend to contribute to the establishment of a more formal, systemic and strategic framework when conducting project or policy evaluations, and therefore help avoid isolated and spontaneous program evaluation efforts. This approach, in turn, makes influence in policy more likely. Along this line, Jones et al. (2009) argue that to promote impact evaluations uptake in the
21

policy process it is important to approach impact evaluations as part of a broader monitoring and evaluation system. The existence of an institutionalized system for evaluation should include a common knowledge management strategy and capacity strengthening mechanisms10, as well as a clear funding strategy to set incentives to help uptake of the evaluations results. Once again, this should be feebler in countries with weaker institutionalization in general. Culture of concrete usage of evaluations. This is different from the existence of an institutionalized system for evaluation. Gaarder and Briceo (2010) suggest that the success of evaluation systems lie in its utilization and that there are two dimensions that indicate it: First, the extent of the evaluation activities in relation to a reference value or universe (coverage). This is usually measured by either the proportion of the budget being evaluated, or the number of programs that have been evaluated in relation to the number of programs in a programmatic classification of the budget or over the multi-year agenda. Secondly, the tracking of the commitments and action plans derived from the evaluation which indicate the usage of its recommendations. Organizational culture and interests. As Jones et al (2009) mention, the culture of organizations, in terms of valuing learning and performance enhancement, also affects the utilization of evaluations findings. In this direction, the utilization of evaluations could be explained by political dynamics and the need for organizations to protect their reputation for funding and from external pressure for change. Following Teller (2008, in Jones et al 2009), they suggest that negative evaluations could be used to make visible their failures and mistakes, and therefore be used politically to withdraw funding or to hinder the agencys legitimacy.
Additionally, Lipsky argues that policy is shaped by those implementing it, who are often constrained by limited resources, continuous negotiation to make sure they are meeting targets and their relationships with their constituencies (as quoted in Crewe and Young, 2002). Furthermore, Lindquist (2001) argues that there are external influences that can affect how policy networks operate. Certain external pressures can force dominant interests to change policy; otherwise, status quo tends to prevail. These pressures can include:
This could include support for a community of practice, training workshops for educated consumers of impact evaluations, supporting the development of national centers of excellence in impact evaluation that could partner with international agencies, peer review of proposed impact evaluation methodologies, and integrating impact evaluations into broader capacity building initiatives on evaluation methods (Jones et at, 2009).
10
22

Changes in government: new governments have an opportunity window and power prerogatives to introduce changes in a broad range of policy issues. They may demand policy advice and therefore increase the possibility for policy influence from research. Changes in the economy: if there are dramatic shifts in the international and domestic economy, there can be implications for a particular policy field and lead to policy change. Changes in technology: the lowering of the costs of transmitting and exchanging information and ideas worldwide can shift and change the accumulated knowledge in a policy field, leading to policy change. Experiences from other countries or regions can be brought in to influence policy in a particular field much more easily. Policy spillovers: a change in a particular policy field can impact in another policy domain, and open an opportunity for policy influence.
It is important to note that most of these contextual factors of demand are not static and depend from the policy sector being considered. For example, in a same country, the Health area could have a firmly institutionalized system for evaluation, and a solid policy forum; whereas the Transport area may not use evaluations at all in its decision making processes and not have a policy forum. To sum up, IEs will have a greater chance to influence policy when being conducted in a context with many entry points (democratic, competitive, decentralized, with policymakers with functional specialization and a background prone to evaluations, and with policy forums), and with an institutionalized system and a culture of usage of evaluations. At the same time, in a context with these many entry points, chances of evaluation influencing policy can be increased by changes in government, economy, technology, and by policy spillovers. We have summarized the aspects that the case studies should consider in the following table, and we have also presented the hypothesis explained above, regarding when IEs policy influence should be easier to attain.
Table 2: Factors, forces and hypotheses regarding the context
Factor / force Political system
Hypotheses IEs evidence in the policy process should be more demanded in a democratic and highly competitive systems IEs evidence in the policy process should be more demanded when policymakers are functionally specialized and when their professions are more prone to incorporate evaluations as valid evidence in the decision making process. IEs evidence in the policy process should be more demanded when: Prestigious, long-termed policy forums exist in the policy area. Policy is decentralized, but technical capacities of the levels of government to which policy is decentralized should be also taken into account
Policymakers
Policy process
23

Factor / force
Hypotheses
IEs evidence in the policy process should be more demanded when: An institutionalized system for evaluation exists, and under which the evaluation effort can be framed. There is a culture of concrete usage of evaluations. Evaluation systems and usage The culture and interests of organizations call for use of IE to protect their reputations and/or funding; or they can use them to meet their targets and/or to strengthen their relationship with their constituencies. IEs evidence in the policy process should be more demanded if there is a window of opportunity provided by: A change in government. Windows of opportunities A change in economy. A change in technology A shift in another policy domain that influences the IEs arena. Source: Authors production.
Policy actors and policy communities Moving from the context to the actors that interplay within it, we can assert that in most policymaking processes there is a multiplicity of diverse actors seeking influence. Political science has explored this field in depth, analyzing the activities of organized interests shaping public policymaking. Currently, it is widely recognized that the actors shaping policymaking processes have been growing in diversity and number, ranging from business, labor and community interests, to as university-based research centers.
a.
b. c.
Determinants for the demand of research in the policy process The context Policy actors and policy communities Purposes and commissioners Determinants for the supply of research in the policy process Determinants for the intersection of demand & supply
specialized types of organizations such
Some schools of thought in political science have traditionally understood the policy process in terms of its policy actors and networks, and their political interests. From their perspective, knowledge is used only to serve political interests in policy processes, used as an ammunition in an adversarial system of decision making, or tactically, as a resource drawn on to bolster decisions or courses of action (or to stall and deflect pressure for action) (Weiss 1977, as explained by Jones, 2009). Therefore, in order to understand the prevailing knowledge used in policy making processes, it is relevant to assess the political interests of policy actors and networks. In this sense, the concept of policy communities, introduced by Pross (1986, in Lindquist, 2001), accounts for the larger milieu in which groups11 attempt to exert influence in a broad policy area. This concept introduces a distinction between those actors located in the sub-government
Not only groups of policymakers, but also encompassing other types of actors, such as lobby groups, business, media, researchers, etc.
11
24

(comprising influential departments in the governments that developed and implemented policies, interest groups exerting strong influence on them and relevant international organizations) and others in the attentive public (including all other actors with an abiding interest in prevailing policy). The sub-government tends to be more supportive of the status quo, whereas the attentive public often criticizes the prevailing status quo, and also tend to be the source of creative new ideas for novel policy approaches (Lindquist, 2001). The existence, shape and nature of these policy communities become essential in the determination of the complexity of policymaking systems, and therefore- how evidence and research can and might inform it. Consequently, considering and understanding how the policy communities are constituted, as well as their functioning, is important when defining policy influence objectives. Being able to understand the interests and motives of the different actors (governmental, non-governmental and international) is fundamental when attempting to foresee how a certain issue or proposal may be received. Mapping the actors in the policy community can be an excellent tool when approaching a policy issue and aiming to influence it12. The policy community actors will be the main consumers of evidence and recommendations drawn from IEs. Lindquist (2001) reviews two different approaches which complement Pross policy community approach, facilitating the understanding of different contexts in which policy research can be supported, conveyed and ultimately- used. Firstly, building on the literature on comparative public policy, he accounts for how public policy and institutional relationships and capacities differ across jurisdictions, by taking in the concept of policy network. This approach seeks to account for the variations in the relative capacities and autonomy of actors, both inside and outside government, and their ability to develop coherent strategies, and mobilize and sustain action. Lindquist summarizes the possibilities in the following graph:
For more specifications on tools to map the actors please refer to Weyrauch and Selvoods (2007) Handbook Weaving Global Networks.
12
25

Graph 2: Lindquists organizational capacity approach in developing, designing and implementing policy
Source: Lindquist (2001)
The policymaking arena will vary depending on the extent of both societal organization and governmental organization. If the degree of societal organization and governmental organization is low, then the policymaking process will be very pluralistic, allowing the interventions of very diverse actors. On the other extreme, if both are well organized, then the policy community will be highly concentrated (or corporative13), not allowing the participation of non organized actors or interest groups. If the society is well organized, but the government is not, then the policymaking arena could be characterized by clientele pluralism, where societal interests can exert their influence in decision making processes. Lastly, if the extent of governmental organization is high, but that of societal organization is low, the state will direct policymaking. Secondly, Lindquist (2001) incorporates the advocacy coalition framework perspective to focus on the clash between belief systems in policy networks and how that may lead to policy change. The advocacy coalition perspective, developed by Sabatier (as quoted in Lindquist, 2001), seeks to comprehend policy communities in terms of beliefs and values. Sabatier argues that in each policy community, different advocacy coalitions can be found. However, in each policy community, there is also one dominant and controlling coalition. This approach complements Pross perspective by stating that actors from the attentive public can also take part in the dominant coalition, when
13
This refers to public corporatism. Business corporations are regarded as part of societal organizations, due to its private goals. However, it must be noted that, for example, in countries with long traditions of public ownerships of business corporation, this divide is blurry.
26

they share basic values and ideals with relevant actors in central institutions controlled by the dominant coalition. It is also useful for understanding the functioning of policy communities in developing countries, where familial and tribal ties tend to be more relevant. Sabatier also distinguishes between the actors stance on a core set of beliefs (which determines their position in a particular advocacy coalition) from their stance on policy situations, which he casts as the secondary aspects of their belief systems. In allowing this distinction, he argues that in terms of core values, advocacy coalitions tend to be conservative, but certain movements can be observed in the secondary aspects. This approach brings important implications on the potential role of researchers influencing the policymaking process. Research findings, according to Sabatier, can assist advocacy coalitions in the production of better arguments. It is crucial to understand that these realms might not exist in certain policy fields. There is not always a right forum for the revision and discussion of research findings. Acknowledging the existence or absence of these spaces is fundamental when attempting to influence policy. If a policy issue is regarded as crucial in developmental terms, sometimes the action that may add more value to its strengthening, besides conducting an impact evaluation, is to promote the creation of such forums. It is also critical to acknowledge, as Lindquist (2001) observations, that policy networks and advocacy coalitions change. This has major implications on the way that the policy influence objective is sought. The actors in a policy field are constantly searching for new evidence and arguments that can translate their beliefs into concrete action. Being able to comprehend how these forces interact, compete and conflict, helps identify possible opportunities to incorporate evidence, arguments and strategies in the policymaking process. In order to do so, it helps to incorporate the different decision modes that may prevail in a policy network. Lindquist argues that there may be four different modes: routine (focus on matching and adapting existing programs and repertoires to emerging conditions, with little debate on its logic and design), incremental (deals with selective issues as they emerge, but does not deal comprehensively with all constituent issues associated with the policy domain), fundamental (infrequent opportunities to re think approaches to policy domains), and emergent (where decision-making goes beyond a questioning of underlying assumptions to a broad new vision). Being able to identify the prevailing decision mode helps to determine the scope and reach of the influence that might be attained:
27

Table 3: How consensus on the policy base, number of actors and type of information should logically change for different decision regimes
Decision regime
Routine
Incremental
Fundamental
Emergent
Status of consensus on the policy base
Intact
Policy base largely intact but marginal issues arise
Core principles of policy base open to scrutiny
Non consensus the field is wide open and open to development of a broad vision Relatively small number of outset
Number of actors involved in decision process
A few actors with responsibility to implement policy by policymakers
A few policymakers with a stake in the marginal issue
All policymakers and actors potentially affected or concerned about a significant change Information on fundamental variables and that probes underlying assumptions. Will also require analysis, data of considerable scope
Type of information sought
Data that can inform existing routines and analysis to determine when to switch to other routines
Analysis on selected issues the method of successive limited comparisons for the issues in hand
Inquiry at a broad level for perspective, but work proceeds on selective issues
Source: Lindquist (2001)
The potential use of evidence from IEs in policymaking will be determined by the prevailing decision mode. If most decisions are incremental or routine, there will be either a built-in bias against the use of research or an interest in data that only deals with incremental issues as they arise. In fundamental or emergent modes, conversely, the demand for evidence will be greater. Furthermore, the decision mode in place also can determine the feasibility of attaining influence in different policy levels (see Graph 1: Types of policy impact objectives: Levels and dimensions1). This is just one example of how the mediating forces and factors between research/IE and policy can affect the IEs policy influence objectives. The stakeholders involved in bridging the IE to the policy process as well as their interaction with other actors in the policy process is most certainly shaped by the nature of the power relations between them. When the IE is produced by a foreigner (either an individual or an organization) to the policy process it aims at influencing, these power relations tend to be asymmetrical and become more relevant. This issue is delved into more deeply in the section of the Organizational and individual characteristics in Determinants for the supply of research in the policy process (see page 33).
28

To sum up, the potential influence of IEs on the policy process is mediated by the existence and shape of the policy community and advocacy coalitions. Its actors, values, beliefs, status of consensus, extent of organization, coordination, capacity and decision mode, determine the way in which the evaluation could be designed, conducted, communicated and, ultimately, its uptake in the policy process. We have summarized the aspects that the case studies should consider, regarding policy actors and policy communities, in the following table, and we have also presented the hypothesis explained above, regarding when IEs policy influence should be easier to attain. With this systematization we aim at facilitating the bridge between this conceptual framework and the conduction of the case studies which will be commissioned by 3ie.
Table 4: Factors, forces and hypotheses regarding policy actors and policy communities
Factor / force
Hypotheses
Ceteris paribus (other things being equal), IEs policy Extent of government and societal information should be more demanded when government and organization society are weakly organized, allowing a pluralistic decision making process. IEs evidence in the policy process should be more demanded if policy communities and advocacy coalitions exist. Moreover, it Existence and characteristics of should be further promoted when: policy communities and advocacy There is no consensus on the policy base regarding the issue. coalitions There are many actors involved in the decision process. Information and evidence provided by IE is sought by the policy community. Source: Authors production.
Purposes and commissioners Why is the evaluation done? To which purpose has it a. Determinants for the demand of been commissioned? Closely related to what we research in the policy process discussed in the second section (policy influence The context Policy actors and policy objectives), understanding why the evaluation is done communities is a very important aspect to bear in mind when Purposes and commissioners assessing its potential policy impact. Bird (2002) b. Determinants for the supply of research identifies three main purposes of impact evaluations: in the policy process c. Determinants for the intersection of improving practice, upward accountability and demand & supply downward accountability (Bird, 2002 in Jones et al, 2009). These different purposes are certainly influenced by who commissions the evaluation and who funds it. In some cases, the main goal is to learn about the concrete practice, in others accountability to donors (upward) might be the priority, while in other cases accountability to 29

beneficiaries (downward) may be prioritized. This influences in the design of the evaluation and how it is developed. For example, when the priority is downward accountability, a participatory approach might be required, allowing the questions to be defined by the beneficiaries, understanding their views of the positive and negative effects of an intervention and taking into account their information needs (Jones et al, 2009). Therefore, it is crucial to consider who has demanded the evaluation. If it has been asked by the implementing agency, it will be more likely to have a voluntary uptake of its recommendations. If it has been required by donors or by external agencies (such as the budget authority, for instance), it is less likely for its recommendations to be voluntarily accepted. Along this line, Jones et al. (2009) mention that it is important to involve multiple stakeholders (including those who demanded the evaluation) in the evaluation process to promote uptake. In any case, it should also be considered whether it has been commissioned for a technical or a political purpose. If it has been required by the implementing staff for technical reasons it is very different to whether it has been requested by the Chief of Cabinet for political reasons. Furthermore, and especially in developing countries, commissioners and donors of IE can also act as brokers for the results of the IE in policymaking processes. In low income and fragile countries, donors are likely to be one of the few actors with the technical capacity to use IE, and therefore advocate for the incorporation of evidence from IE in decision making. All these factors not only determine the way in which the evaluations results might (or not) influence policymaking, but also affect the prior definition of the policy influence objectives of the evaluation itself. Regarding the commissioning and the purposes of the IE which might affect the demand of IEs evidence in the policymaking process, the following aspects should be included in the case studies:
Table 5: Factors, forces and hypotheses regarding the commissioning and purposes of IEs.
Factor / force Commissioning and funding objectives Nature of the commissioner Source: Authors production.
Hypotheses IEs should be more demanded when the commissioner and/or donor has a clear notion of its policy influence objective/s. IE should be more used if it has been commissioned by the implementing agency and when the IE research process has involved multiple stakeholders
To sum up, the demand of impact evaluations from the policy process is determined by contextual factors, the presence or absence of policy communities and advocacy coalitions and the profile of commissioners (and their objectives).
30

b) Determinants for the supply of research in the policy process

In order to comprehend how research may influence policy it is very important to consider when and where research is conducted and by whom. It is normally assumed that research is carried on by academics operating in ivory towers, but there are other organizations with research capacity (as we have stated in section 2), that may also attempt to influence the policymaking process. In the last decades, there has been a hybridization process in knowledge production and currently the supply of research is provided by very diverse actors. In each case, it is crucial to describe how research is produced. Contextual characteristics There are some contextual characteristics that can affect the overall supply of research (and the conduction of impact evaluations) to the policy process. Particularly, this refers to the existence of a thriving social science community (or communities) that can be on the sending side of the research-policy communication (Weiss, 1999).
a. b.
c.
Another pre-condition for the supply side to exist has to do with the availability of a sizeable group of social scientists who are interested in conducting policy-oriented research. In the following sections we will argue about the characteristics that the researchers (and the institutions they belong to) should have to promote a greater influence of the impact evaluation. Furthermore, the existence of public funds or funds from international organizations or banks to conduct research (and impact evaluations in particular), can generate an incentive structure which could promote its production. In countries where these funds are available, the supply side tends to be stronger and better organized, enabling, therefore, a greater capacity in conducting impact evaluations. The source of the funds (whether impact evaluations are financed by governments or international organizations/ banks) can pre-determine their focus, in terms of fields, projects and methodology. Organizational and individual characteristics As we have mentioned before, currently, b. knowledge is produced by very diverse actors. c. Particularly, impact evaluations can be conducted by organizations or by individuals. The organizations can be either private or public (and therefore, the evaluation internal or external), and within private organizations, we
d.
Determinants for the demand of research in the policy process Determinants for the supply of research in the policy process Contextual characteristics Organizational and individual characteristics Research (impact evaluations) characteristics Determinants for the intersection of demand & supply
Determinants for the demand of research in the policy process Determinants for the supply of research in the policy process Contextual characteristics Organizational and individual characteristics Research (impact evaluations) characteristics Determinants for the intersection of demand & supply
31

can find both for profit organizations and non for profit. When the evaluation is conducted by a single individual or a by a non-organized group of individuals, their personal capacities (technical, political and social) become relevant: Technical capacities: The researchers educational background and professional credentials can be crucial to determine the technical capacity. Political and social capacities: The researchers links to the policymaking arena become crucial, therefore, his or hers networking capacity, legitimacy and prior contacts should be considered when assessing the evaluations impact. To assess these capacities it is relevant to determine whether the researcher is a national or a foreigner of the country where the policy is being evaluated. Communication skills: If the researcher has good communication skills it is much more plausible that he or she will be able to both conduct the IE and convey the results and recommendations more convincingly, therefore enabling a greater possibility for policy impact.
Individuals can conduct impact evaluations, even when it has been commissioned to an organization. Often, organizations seem to not conduct the evaluations themselves, but hire an external impact evaluation specialist. Considering the organizations, there are several characteristics that can either contribute or hinder the possibility of informing the policymaking process with evidence or lessons from the evaluation. Following Braun et al. (2006), the institutional factors that can determine the policy influence capacity of research produced by an organization are14: Organizational governance. The institutions leadership structure and organizational diagram may influence how the evaluation is conducted and then communicated, therefore impacting in how it can (or cannot) be incorporated in the political process. The organizational governance structure is obviously shaped by its origins (who where the champions in the inception of the institution, their leverage, their contacts and their beliefs). The knowledge management processes within the institution should also be taken into account, since important know-how transferences can take place within the organization. Along these lines, Jones et al (2009) mention that there is a need for a broader, strategic framework for impact evaluation production. In this sense, it becomes relevant whether
14
The study conducted by Braun, Chudnovsky, Ducot and Weyrauch (2006) focused on the characteristics of Policy Research Institutions. However, many of these characteristics can be generalized to also include governmental agencies that conduct impact evaluations.
32

the organization has a strategic planning process where the evaluation can be framed. In turn, this also relates to a more general framework that can be provided by the existence of an institutionalized evaluation system at the country level, which would allow the evaluation to be a part of an intervention addressing a wider policy-level. Budget lines / Sources of funding. How the organization is funded, and whether it is from diversified, secured and discretionary funds or not determines part of its policy influence capacity. If the agency conducting the evaluation has its own financial resources, it is likely to have greater enforcement capacity than if it is underfunded. It is also important to consider the source of funding, whether it is external or internal. If it is external, then the agency will probably have greater independence but less policy influence potential, situation which would be reverted in the case of it having internal funding. This should be considered especially for nongovernmental organizations conducting impact evaluations. However, it should be acknowledged that many governmental agencies that produce policy impact evaluations are funded externally (by international banks, for instance) and that this also affects their influence potential. Jones et al (2009) also add that it is critical to regard funding as a variable which shapes evaluation practice and the incentive structures it creates. They mention that it is vital to promote process changes such as wider consultations, better sequenced and integrated lesson learning processes and closer engagement between the implementation and evaluation staff (Jones et al, 2009), encouraged through funding incentives. The organizations affiliation is another critical factor, related with the funding of the organization, which has to be considered. Semi-governmental organizations that operate as think tanks may have a greater influence capacity than other, non-affiliated, more independent organizations. The same could be said (but probably to a lesser extent, at least in countries with presidential regimes) for think tanks or research institutions affiliated with political parties. Even when this may be more applicable to evaluation in general, and less so to independent impact evaluations as defined by 3ie, the organizational context in which the independent IE is being conducted may certainly affect the way it is conducted. Human resources. The technical capacity of the organization to conduct IEs, the value and participation of its human resources in the policymaking process, and their motivation to impact decisions, can be huge determinants of how the evaluations lessons are produced, communicated and perceived. In this sense, it is relevant to assess whether the organization has invested in evaluative capacity building to promote knowledge sharing and learning within the institution. Institutional mechanisms for research management. How research is selected, conducted and characterized according to institutional procedures may increase or obstruct its potential to influence policy. Furthermore, if research quality control mechanisms (such as
33

in-house discussions, experts opinions, peer reviews and feedback from seminars and workshops) are in place, there is a greater chance that the research product will be considered legitimate, relevant and of quality, and therefore augment its possibilities to influence policymaking. Communications. The existence of institutional communication policies and products, the role of communication in the evaluation plan, the staffs communicational skills, as well as the presence of networks or ongoing relationships with policymakers and other stakeholders can also facilitate a greater degree of policy influence.
As mentioned earlier, IEs can be conducted either by governmental or by non governmental agencies. Whether the evaluation is done internally (intra-governmentally) or externally (from an outside institution) determines several characteristics that may influence the degree of impact that the evaluation will have in the policy process. For example, it can influence the perception of the evaluations independence. As Gaarder and Briceo (2010) argue, institutions positioned outside the government assumed to enjoy a higher degree of independence. Independence is one of the major issues in determining the evaluations legitimacy. An additional aspect that should be taken into account for intra-governmental evaluations is the location of the institution that is conducting the evaluation. The location of the agency within budget or planning authorities provides the strongest powers to the system to enforce adoption of recommendations. This can be also achieved through regulations that make evaluations compulsory for inclusion by budget or planning authorities (Gaarder and Briceo, 2010). An example of an evaluation agency which is close to the enforcement power is DIPRES in Chile, which is located within the Ministry of Finance, the budget authority, with a dedicated budget line to finance the approved evaluation plan. On a different basis, is the example of SINERGIA in Colombia, in which the evaluation agency is neither institutionalized nor has the backing of a law to give teeth to its evaluation oversight mandate. Despite this, it addresses the difficulty of enforcement by an alternative strategy: generating a tradition of utilization as a managerial tool rather than a control tool. If the implementing agencies are involved in identifying the issues to be addressed by the evaluations, and consulted in the design, implementation and analysis phases, a sense of ownership of the evaluation efforts may ensue and will increase the likelihood of utilization and voluntary adoption of recommendations by the program managers. Their effort is focused on promoting a culture of utilization of evaluation as a project management tool. (Gaarder and Briceo, 2009) Besides considering all these characteristics, it is also important to acknowledge that institutions are not static: they evolve. Therefore, the internal changes that organizations may suffer should also be taken into account when assessing their potential roles in producing and brokering evidence for policymaking.
34

Another important consideration refers to the capacity to produce knowledge depending on the geographical setting of the organization and/or of the individual conducting the evaluation. It is important to acknowledge that IE can be done by international consultants/organizations and, in those cases, it is crucial to consider how (and if) the IEs results are appropriated in the local context. Lately, there has been a growing hegemony in the production of knowledge, especially impact evaluations, from the US. However, there does appear to be a growing movement among impact evaluation experts to undertake these in partnership with developing country researchers and/or government agencies [].that have legitimacy in the eyes of the government, so that the findings will be more likely to be adopted (Jones et al, 2009). It must be noted, nevertheless, that these partnerships are far from being the ideal, since the negotiation leverage of the alleged partners is highly unequal and tends to, more often than not, benefit the northern researchers in the division of both responsibilities and funding. To sum up, it is very important to identify who is conducting the impact evaluation and its basic features. The fact that IEs can be conducted by individual experts or by institutionalized organization (and the characteristics that one or the other may have) affect the evaluations potential uptake in the policy process.
Research (impact evaluations) characteristics Besides contextual and b. individual/organizational characteristics, the c. researchs qualities can also influence its potential for policy impact.
Types of research methodologies Many of the aspects mentioned before regarding the potential policy influence refer to generic research. Notwithstanding, it is important to notice that different types of research may attain diverse policy influence potentials. For example, qualitative and quantitative research results bear important d. differences when aiming at influencing policy. Qualitative research is usually considered more subjective than quantitative research, even though the methods used to conduct them can be equally rigorous. Quantitative research, on the other hand, tends to be perceived as more accurate, objective and representative when fed into the policymaking process. Impact evaluations tend to be considered in the latter group, and even when reference is frequently made to mixing qualitative and quantitative methods, qualitative research is not often
Determinants for the demand of research in the policy process Determinants for the supply of research in the policy process Contextual characteristics Organizational and individual characteristics Research (impact evaluations) characteristics: Types of research methodologies Impact evaluations as a knowledge product Quality and independence Determinants for the intersection of demand & supply
35

considered as being as rigorous as the evidence produced through quantitative methods (Jones et al, 2009). Jones et al (2009), in their review on literature on concepts of impact evaluation and their various methodologies, state that debates about impact involve looking at the effects of development interventions on their surroundings. These effects do not relate to the interventions outputs but on its outcomes and impacts, being more closely related to the broadest developmental goals, such as the contribution it makes to reducing poverty. Jones et al (2009) also argue that despite the existence of different perceptions of how impact should be evaluated, there is one current approach that has dominated donor discourse on IE: nowadays, in many aid agencies IE refers to an evaluation that assesses the effects of an intervention using a counterfactual which tends to be assessed with experimental or quasiexperimental methods. Experimental methods involve comparing what actually happened with what would have happened if the project had not been implemented, by using a randomly assigned and well defined control group to compare the beneficiaries targeted by the intervention to those who did not. Therefore, the differences in outcomes between these two groups (the actual beneficiaries and the control group) can be attributed to the intervention. Experimental IE methods therefore focus on the assessment of the effects of a particular project, program or policy on different measured (and therefore quantifiable) outcomes in a target group of beneficiaries, by counterfactually comparing them to what happened to the control group in the absence of the intervention. Quasi experimental methods are similar to experimental methods, but differ in the assignation of the program to the beneficiaries, which is not done randomly but using statistical methods (i.e. propensity score matching) to simulate a control group. In spite of the current propensity to use counterfactual models as gold standards in donor agencies, there are other alternative approaches to IE. As Jones et al (2009) argue, The counterfactual is just one among many types of causality, for which there are various alternatives (recognised in the natural and social sciences): generative causality involves identifying underlying processes that lead to change (one method of this type uses qualitative methods to assess causality by understanding peoples operative reasons for their actions or behaviour change (Bhola, 2000))15; another approach takes a configurational approach to causality, in which outcomes are seen to follow from the combination of a fruitful combination of attributes (Pawson, 2002). Many argue that is also possible to assess the counter-factual using non-experimental theory-driven methods, such as process tracing, which examines causation as part of a theory
Indeed, proponents of this approach argue that it is the only way to understand causality in the social sciences: this is because the reasons people have for their behavior are not reducible to physical models of causation exemplified by the counter-factual (Jones, 2006).
15
36

focusing on a sequence of causal steps (Jones et al, 2009, p 4). However, most of these alternative approaches are dismissed by proponents of experimental IEs. Whites (2009) reflections on this debate, argue that there are a number of misunderstandings. Firstly (and most important), there are different definitions of IE in the debate, meaning diverse things. Therefore, the co-existence of different conceptions of IE makes the discussion on the appropriate methodology somewhat absurd. White states that:
The tradition in evaluation has been that impact refers to the final level of the causal chain (or log frame), with impact differing from outcomes as the former refers to long-term effects. For example, the DAC definition of impact is positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Any evaluation which refers to impact (or often outcome) indicators is thus, by definition, an impact evaluation () But this definition is not shared by many working on impact evaluation, for example in the World Bank. Impact is defined as the difference in the indicator of interest (Y) with the intervention (Y1) and without the intervention (Y0). That is, impact = Y1 Y0 (e.g. Ravallion, 2008). An impact evaluation is a study which tackles the issue of attribution by identifying the counterfactual value of Y (Y0) in a rigorous manner () These are completely different definitions of impact. They may overlap if Y is an outcome indicator. But I now believe that drawing attention to the overlap (which I have done many times), or worse still, treating the two definitions as if they are somehow the same, confuses the real issue, which is the fundamental difference between the two definitions. Since this is a purely semantic matter, neither side is right or wrong. The definitions are just different. No debate about methodology will be of any use unless we first agree which definition is being used.
Secondly, a further area of confusion relates to counterfactuals and control groups. The use of comparison groups does not always lead to the construction of a counterfactual, and this is not clear in the debate. Thirdly, there is also confusion in the debate regarding contribution and attribution. White argues that the calls for addressing contribution rather than attribution are also definitional, and they mistake claims of attribution to mean sole attribution. The debates regarding the definition, meanings and methods of IEs are not settled. Nonetheless, the current domination of the counterfactual approach in this debate causes that most of IEs that are practically conducted follow this type. For example, The International Initiative for Impact Evaluation (3ie), though embracing a range of evaluation methods and approaches, states in its Principles that High-quality impact evaluations measure the net change in outcomes amongst a particular group, or groups, of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question(s) being investigated and to the specific context (International Initiative for Impact Evaluation, 2011), therefore requiring a counterfactual approach. This is reinforced when clarified that this organization supports IEs that follow methodological standards for addressing the attribution challenge, this
37

is to say that they refer to IEs that establish cause and effect between programmatic activities and specified outcomes capable of addressing: a) confounding factors; b) selection bias; c) spillover effects; d) contamination of control groups; and e) impact heterogeneity by intervention, beneficiary type and context. Therefore, in this paper, we have focused on this kind of IE. This fact has implications in terms of policy influence. Firstly, counterfactual IEs can be regarded as being more rigorous that other types of research and IE, and therefore influence its perceived legitimacy. Despite this being the case in most policy debates, and as Jones et al (2009) argue, it should be supplemented with other knowledge to understand how the results can be interpreted outside that specific context (p. 6). Secondly, counterfactual IEs have a set of requirements that should be fulfilled in order for IE to be feasible, regarding the timeframe, outcomes, dimensions, and other variables16. Impact evaluations as a knowledge product When considering policy influence, the main relevant characteristic of research is whether it is oriented (or not) to influencing policymaking. This orientation would mean a clear focus on the type and usage of the research products. In practical terms, this could be translated as to be constituted as a good knowledge product, in terms of policy influence. Following Malik (2002) the characteristics of a good knowledge product are: Designed for a specific audience Relevant to decision making needs, especially for country office staff Available when the window of opportunity for decision making arises (i.e. timely) Easily and quickly understood Based on sound methodological principles Areas of uncertainty and their significance clearly identified Accompanied by full acknowledgement of data or information sources Provides information on both tangible and intangible products and processes of development Available at minimal cost in terms of time, money and administrative costs
Adding to this list, another important factor in an IE that aims at influencing policy is to give real time results to stakeholders, and also, to do so through specific and policy oriented recommendations. The evaluations results should be explained in such a manner that a policymaker can easily understand its policy implications17.
16
Following Jones et al (2009), there are six requierements, which are explained in page 5 of their paper. They should be brief, clear, and consider its technical, political and fiscal feasibility.
17
38

Quality and independence An essential pre-condition for policy influence has to do with the perceived quality of the research. It is important to note that it is really the perception of quality and not the actual quality of the research product what is at stake. Much of the perception of quality has to do with the legitimacy of the researchers and/or the institution that is conducting the evaluation. But some visible mechanisms to ensure quality could facilitate to the construction of this image of quality. Impact evaluations should be carried on with the relevant skills, sound methods, adequate resources and transparency, in order to be of quality. This can be sought establishing control mechanisms, external verification or audits and alliances with internal control or auditing office. The quality of the evaluation also depends on the technical capacity among program staff and researchers in evaluation methodologies. It is important to note that the quality of impact evaluations is not limited to its credibility and thoroughness, but also includes its relevance and accessibility (Jones et al, 2009). In developmental social research, institutions (or researchers) who ensure they build legitimacy chains to their informants may be less likely to be ignored (Crewe and Young, 2002). These authors argue that pro-poor research implies adequate consultation with poorer people and this may have to take priority over doing an exhaustive literature review. Whereas this might be valid for traditional social science research, it may not be applicable directly to IEs. However, it may be useful to validate the way in which the conclusions and results from IEs are presented with key informants with legitimate chains of information and local involvement. In a similar note, Rogers (2008) mentions the use of multi-stakeholders dialogues in data collection, hypothesis testing and intervention, in order to allow greater participation and recognize the differences that may arise. This methodology can be a framework to tackle difficult and complex problems and foster a solution-oriented network for the later usage of the evaluation. In order to increase its policy influence potential, IEs need to be both independent and relevant. Independence is achieved when it is carried out by entities and persons free of the control of those responsible for the design and implementation of the development intervention (OECD, 2002 in Gaarder and Briceo, 2010). However, this is not to say that complete independence is desirable. For one, researchers need to access program data in order to develop the evaluation. Furthermore, the influence of the recommendations from the impact evaluation tends to be greater if the researcher and the staff implementing the program and designing future interventions arent isolated. Therefore, to be relevant, researchers need to consider the operating context and access operational information (Gaarder and Briceo, 2010). Arriving to a balance of both (independence and enforcement) is one of the challenges faced by evaluators. It is important to acknowledge that there might be a trade-off between quality and likelihood of policy influence. Following Gaarder this could be due to two reasons. Firstly, rigorous studies take longer than non-rigorous ones (on average), and there are less researchers available for doing the
39

former, which impacts on the possibility of the IEs results being fed into the policy process in a timely fashion. Secondly, independence (which, as mentioned, is a contributing factor to quality), may hinder the adoption of the recommendations as agencies may be less open to recommendations for outsiders.
To sum up, the supply of research (IEs) into the policymaking process will also depend on the researchs characteristics. We have summarized all aspects that the case studies should consider regarding the supply of research in the following table, and we have also presented the hypothesis explained above.
Table 6: Factors, forces and hypotheses regarding research supply.
Group Context
Factor / force
Hypotheses
IEs with policy influence objectives should be in greater supply Social science community when there is a thriving national social science community and when there is a sizeable group of social scientists. Funds IEs should be in greater supply to the policymaking process if there are international or public funds available to finance them IEs with policy influence objectives should be in greater supply when they are conducted by individuals with high technical, political and social capacities, and with developed communicational skills. IEs with policy influence objectives should be in greater supply if they are conducted by organizations: With a strategic framework of impact evaluation production oriented to feed in policy processes; with research management mechanisms, communications strategy and a highly technically capable staff and that participate in policymaking process. That are part of the governmental structure, or have affiliations to the government or to political parties.
Researchers Individual
Organizational
Research characteristics
An evaluation which is from the start designed to influence policy as a knowledge product, which is of quality, relevant and independent, will have a greater policy influence potential.
Source: Authors production.
c) Determinants for the intersection of demand & supply

We have considered external and internal forces that are at play in determining whether evidence from research (and from IEs, in particular) influence policy change. But what finally determines whether evidence and recommendations emerging from evaluations are adopted and yield to policy change is the way the demand and supply aspects finally interact. This is the space where
40

intermediaries or brokers (i.e. think tanks) and other influential actors such as media might play a significant role. Policy processes It is first important to analyze the nature of the process in which policy and research may (or may not) come together.
a. b. c. Determinants for the demand of research in the policy process Determinants for the supply of research in the policy process Determinants for the intersection of demand & supply Policy processes Bridging communications The interplay of supply, demand & policy influence objectives
It is the policy process which provides the context for the intersection of demand and supply. Therefore, it is necessary to understand the nature of this process, for which different models have been developed. Following Crewe and Youngs (2002) literature review: policymaking models have been a major issue for discussion in political science. The rational (either linear or incremental) model has been long under attack, and has given away for political context models (such as M. Grindles (in Crewe and Young, 2002) interactive model) and policy change models (as the offered by Keeley and Scoones, in Crewe and Young, 2002). This last approach suggests that the policymaking process is structured by a complex interplay between political interests, competing discourses and the agency of multiple actors. Policy processes arent logical nor do they follow the stages traditionally described in the policy cycle. John Kingdon (1984, in Crewe and Young, 2002) argues that there is always an element of chance and randomness in these processes. His garbage can model of decision-making identifies three streams of activity that attempt to move alternatives higher on the agenda: the problem stream (the recognition of certain social issues as problems), the policy stream (ideas of what may be the best solution for a certain problem), and the political stream (which brings in both new problems and new potential solutions18). The result of the moving of one issue up in the policy agenda is produced by the interaction and confluence of at least two of the streams. Additionally, we should also consider the importance of power theories within these complex policy processes, since it is still power interests that determine knowledge processes, from generation to uptake. As the politics and legitimization paradigm of thought to frame the link between knowledge and policy, knowledge used in the policy process often will reflect and sustain existing power structures (Jones, 2009). To do so, evidence from research will be used to contest, negotiate, legitimize and marginalize in political debates, but always strategically aligned with political interests. It is also the distribution of power within the policy communities which determines the number of entry points or opportunities for knowledge uptake. In this regard, it
18
The political stream affects both potential problems and solutions, since the failure or success of the problematization of an issue and/or of the quest for a solution can hinder or strengthen particular interests.
41

is crucial to acknowledge the places and spaces/sectors in which policy making processes can be framed. This refers to the level in which policy is being made (local, subnational, national, regional or international), which determines the participating actors, the prevailing discourse, the degree of participatory approaches, etc. It also refers to the particular sector in which policy is being produced. Each sector might have a differing operating dynamic (Gaventa, 2007 as explained by Jones, 2009). Therefore, it is in these complex-natured policy processes, that evidence and lessons from impact evaluations are poured. It is important to remember the notion of policy networks, policy communities and advocacy coalitions when trying to conceive the nature of the policy process. In this regard, one may note that the boundaries between knowledge producers and knowledge consumers are increasingly permeable (Lindquist, 2001) or that the alleged divide between the production and the use of knowledge is not such (Jones, 2009). On the one hand, researchers and policy analysts tend to be decoupled from specific institutions and truly committed policy focused scholars tend to be constantly in and out the public sector, therefore being on the one side of the counter on a particular day and on the other side the next day. Furthermore, as Lindquist argues, there is a third community emerging, between the academia and the public sector. This third community is composed by organizations committed to producing policy-relevant data, research or analysis, though not always being fully committed to social science research rigor (i.e. Think tanks, consultants, policy shops, legislative committees, etc.). These organizations tend to act as knowledge brokers when fostering evidence based policymaking and can be a great channel to communicate the evaluations results. Furthermore, this process takes place in a context where the different players have imperfect information. For example, policymakers and evaluators have different capacities and knowledge to determine whether an evaluation is rigorous or not, and whether its results and recommendations are unbiased. In this sense knowledge brokers can have a role in correcting the asymmetries of information. To sum up, the case studies should take into consideration the main characteristics of the policy process, determine the main entry points (places, spaces/sectors, levels), identify the central political interests that are being contended and detect and characterize the main knowledge brokers. Bridging communications An effective intersection between demand and supply is based on good communications which is the key process that determines how research reaches the policy field. We understand communications as the processes of generation, circulation
b. c. d.
Determinants for the demand of research in the policy process Determinants for the supply of research in the policy process Determinants for the intersection of demand & supply Policy processes Bridging communications The interplay of supply, demand & policy influence objectives
42

and reception of intentional and unintentional signals that create perceptions and expectations among different actors. In fact, even though the boundaries between research producers and research consumers are blurring, these are still different roles that generate diverse expectations and interests around research and how it might be used. These differences that naturally emerge from the supply and demand factors explained in the previous sections frequently explain why and how an impact evaluation has had a positive reception or not. For instance, if evaluation is an institutionalized practice in a governmental agency, it is more likely that policymakers will commission an IE themselves and decide when and how its results will be communicated and used. In spite of all these differences, we should note that all knowledge consumers, producers and brokers operate and communicate among themselves influenced by certain prevailing discourses. Discourse encompasses the concepts and ideas relevant for policy and the interactive processes of communication and policy formulation that serve to generate and disseminate these ideas (Schmidt and Radaelli 2004, in Jones 2009). These discourses are based on a set of assumptions that are taken for granted (engrained in policy narratives, cognitive paradigms, frames and categories) by the diverse actors and thus heavily influence what policymakers are prone to listen to and how they understand it and use it when rethinking a policy issue. Additionally to detecting and understanding the policy discourses that will affect the interaction from its inception, there is a set of factors that facilitate a good interaction between knowledge producers, consumers and brokers. Firstly, it is important to highlight that communications among these diverse groups of actors should begin as early as possible. In fact, best practice examples demonstrate that a central factor facilitating uptake of IEs is stakeholder involvement. This involvement must be brought in at the early stages of the IE process, include the support of high-profile champions and attract political agents interested in learning or using instruments to demonstrate effectiveness (Jones 2009). The reporting of the results of IEs is as relevant as its conduction. As Crewe and Young (2002) argue, the sources and conveyors of information may be as influential as the content. There is intrinsic importance in the way messages are conveyed. The communication of the evaluation is, of course, easier if the evaluation itself has been conducted with a policy influence goal from the start. A key aspect in this regard is to determine who is in charge of reporting the results of impact evaluation. Is the researcher (or the research institution or agency) who conducted the evaluation or is it the organization in charge of the activities under evaluation? The objectivity and independence of the communication can be compromised if it is the latter due to its proximity and responsibility in the implementation of the activities being evaluated (Gaarder and Briceo, 2010), but the (filtered) messages that are communicated could have a greater influence. 43

Another aspect is who participates in the definition of the communicable outputs. If the agency responsible for the implementation of the program is involved in identifying the key messages, and consulted thereof, it is more likely that they will be willing to adopt the recommendations. However, this comes at the cost of a probable compromise of the independence of the reporting. A key issue in the definition of the who communicates is to ensure that a communications expert works with the research team in order to ensure good communications products (Yaron and Shaxson 2008). In the same vein, knowledge producers should try to identify consumers preferences, expectations and interests from the very beginning. Policymakers are primordial knowledge consumers since most impact evaluations aim at changing and improving existing policies. Within the complex context of the policymaking process, and following Weiss (1999), policymakers might pay attention to research for the following reasons: To get a better sense of the issues Because they distrust the information that is fed to them by other sources To provide legitimacy for political action To find evidence or theory that supports their position To be considered modern, up to date and well informed
Considering the links between the institutions and the individuals in the policy and the research sides is crucial, since they shape the influence they may have on each other.
Furthermore, it is important to note that there are different possible audiences for the recommendations being issued from the results of impact evaluations. Also audiences may vary according to who uses the results of the IE since not always are the knowledge producers the (only) ones to produce a set of recommendations based on these results. Moreover, evaluation experts are not necessarily the best placed to turn their findings into actionable recommendations due to their role and competitive advantages. Other stakeholders, for example research units at governmental departments (with inner information about what is technically and politically feasible) or think tanks are frequently well positioned to translate results into practical steps for policymakers.
In this direction, the presence of high-profile issue champions of the program in question is key. One possible audience is constituted by agencies from the Executive, such as the Planning and Budget authorities. Other, very different, public is the implementing agency. Whereas the planning and budget authorities will use the information to improve the efficiency and effectiveness of resource allocation, the implementing agency will be more interested in revising implementation processes, fine-tuning the design, changing and improving managerial practices and responding to its constituencies with concrete information (Gaarder and Briceo, 2010). However, policymakers are not the only stakeholders in the game of using impact evaluations for policy processes. As we have mentioned earlier, policy is constructed by and filtered through 44

policy communities conformed by advocacy coalitions with pre-conceived ideas and values and where knowledge brokers (think tanks, media, etc) may play a significant role. It is critical to determine how information flows, is received, digested and acted upon by these policy communities and advocacy coalitions when planning a communication strategy. Also, regarding the format and timing of communications, the lessons and recommendations drawn from the impact evaluation could be transmitted either orally or in writing (or both). Following Weiss (1999), written reports are important to provide a full account of methods, findings, caveats and interpretations, while oral briefing are usually more effective in engaging policy actors interests and communicating key ideas. This is due to the degree of specificity and detail that can be included in a written document is not matched by an oral briefing. Furthermore, it is also important to consider the timely scope of the communication. It is certainly different to distribute the paper with the evaluations results once it is finished, than to engage in the policy arena debating the issue which has been evaluated. In addition to these two typical communication channels, the opportunities provided by new technologies should also be acknowledged. Therefore, we should also consider new channels through which the evaluations results can be transmitted, such as blogs, videos, social networks, etc. A body of literature on bridging research and policy suggests that keeping research conclusions messages simple may help it influencing policy processes. However, simple messages are more vulnerable to being manipulated by policymakers and overlook the complexity of proposed solutions. Therefore, this tension between the benefits of simplification and the disadvantages of oversimplification of research conclusion should be taken into account. Briefly, the policy process is a largely complex and chaotic process within which an impact evaluation may be interpreted and used in very diverse ways by different audiences. When knowledge producers, consumers and brokers participate and communicate from the outset, and are aware of the common and opposite needs, interests and expectations, it is more likely that they will find the adequate channels, timing and formats to communicate and discuss the value and potential use of the impact evaluation. This is naturally highly influenced by the contextual factors described above. For instance, if there is an institutionalized culture of evaluation, there will probably exist a set of established communications channels between producers, consumers and brokers. We have summarized the aspects that the case studies should consider regarding the intersection of demand and supply of research in the following table, and we have also presented the hypothesis explained above.
45

Table 7. Factors, forces and hypotheses regarding the interplay of demand and supply.
Factor / force Policy process
Communications Source: Authors production.
Hypotheses The intersection between IEs demand and supply should be better articulated when the nature of the policy process is considered, entry points identified, the main political interests detected and the knowledge brokers identified. The intersection between IEs demand and supply should be better articulated when prevailing discourses, timing, conveyors of key messages, main audiences characteristics, channels and formats are considered.
The interplay of supply, demand & policy influence objectives Through this paper we have analyzed the a. Determinants for the demand of research in the policy potential impact of IEs on public policies process from two main perspectives. b. Determinants for the supply of research in the policy
process
First, we considered the relevance of c. Determinants for the intersection of demand & supply defining (ex ante) or identifying (ex post) Policy processes research policy influence objectives. This Bridging communications definition is key to determine whether an IE The interplay of supply, demand & policy has had or not impact, and if it has indeed, influence objectives to describe what type of influence can be observed, including diverse levels and dimensions of influence. Second, we centered our analysis in the factors and forces that may explain why a certain IE has influenced (or not) a specific policy making process. These intermediating factors belong both to the demand and supply of research, and to the way these finally come together. As we have mentioned, it is the concrete and particular interplay of all these variables what determines, ultimately, the potential policy influence of impact evaluations. The interaction between these variables is multidirectional. The existence and determination of policy influence objectives when conducting an impact evaluation are influenced by, and also influence, the characteristics of supply and demand of research. For example, the existence of an institutionalized system of evaluation (demand, contextual variable) might promote the production of better impact evaluations (therefore, putting pressure on the technical capacities of the individuals or of the organization conducting the evaluations and on the evaluation characteristics itself, all supply variables). However, if the evaluation is not conducted and communicated with a concrete and viable policy influence objective (for example, to promote operational improvements of a specific program) it will be hard to finally fulfill its potential policy impact. 46

Graph 3: Supply and Demand forces and factors coming together
Source: Authors production.
Consequently, the policy influence objectives of research should be determined and later analyzed with a thorough assessment of the supply and demand variables. For example, if there is a thriving policy community with a fundamental decision regime, and if the evaluation is conducted by a legitimate organization or an individual, with technical capacity, and a clear focus on knowledge products, policy influence objectives can be more ambitious than if there is not a dynamic policy community (or if there is, but it operates under a routine decision mode) and if the individual researcher conducting the evaluation is a junior evaluator with no political, social or communicational capital. In the later the policy influence objective should be much more limited. But this is not a one-way determination. If from the start the evaluation is set out with clear policy influence objectives (for instance, if it aims at a procedural change in a particular project), the evaluation can be commissioned to an organization with the capacity to help attaining this objective and, on the other hand, the initiative may aim at altering certain contextual variables (i.e. promote the creation of policy forums on the projects subject, or strengthen the policymakers awareness on the issue through training sessions) It is the interplay of research supply, research demand and policy influence objectives which ultimately constitute themselves as channels and determine the policy change that an impact evaluation has attained or may attain.
47

Graph 4. Channels for IEs policy influence
48

4- Final observations: Measuring the real impact of evaluations

In this paper, we have presented a summarized conceptual review of literature to shed light on IEs potential influence on policy, in order to guide the production of 3ies case studies series on policy influence of impact studies. In this sense, we have discussed the importance of setting policy impact objectives, and the different types of impact that can be sought and -ultimatelyattained. In the third section we have discussed the factors that mediate the relation between IEs and policy change. This should contribute to better explain how and/or why an impact evaluation may achieve (or not) policy influence (and the very diverse ways in which this may take place). Sound expectations, based on a clear understanding of demand and supply factors and their interplay, should help to establish viable policy influence objectives, and, in due course, foster IEs impact on policy change. Yet, the effective measurement of policy influence of IEs is another large challenge, which deserves special attention and further development. The literature has for long recognized the deep complexity of the task to assess the impact and role of a certain piece of research on public policy because of the nature of research and its related activities and because as already highlighted in this paper- policymaking is a very dynamic process with a multiplicity of actors and relationships (Lindquist, 2001). This means that there are both conceptual and technical challenges in measuring the impact of evaluations in policy change, most of which have to do with the attribution problem19. Kunal Sen (2010) identifies three main challenges facing the measurement of the rate of return to research:

The attribution problem -did the research influence policy? The identification problem -did the policy intervention or reform lead to the desired/observed outcome? The measurement problem -can benefits of the outcomes be quantified?
He argues that these three problems tend to have different relevance depending on the policy sector and that if they are satisfactorily addressed, the return rates of research will be more accurate. For instance, he concludes that in agricultural and health research, there are more credible proxy rates of return; whereas in climate change and governance research there are no credible ways to calculate them. Taking into account these restrictions, some considerations for the strategic measurement of the impact of evaluations can be as follow:
19
For a deeper insight regarding the attribution problem and its possible solutions, refer to John Maynes Contribution Analysis: An Approach to Exploring Cause and Effect.
49

First, establish precisely what the contributions and intentions of the projects are (before attempting to assess policy influence) Account for the larger institutional environment in which research proceeds (and the multiplicity of actors) Identify a sufficiently long time frame with which to understand the influence of research Recognize that values and the ongoing struggle over ideas and policy matters greatly in the commissioning, interpretation and use of research. Determine who and how the impact measurement will be carried out, including methodology, policy influence indicators, sources of information and format for reporting.
Even though it is a challenging endeavor, measuring the policy impact of an evaluation could be regarded as a window of opportunity to generate new knowledge and thus improve future interventions and refine expectations. In fact, sound expectations work better when there is also openness and willingness to learn.
50

Bibliography
Bamberger, Michael, Institutionalizing impact evaluation within the Framework of a monitoring and evaluation system, World Banks Independent Evaluation Group, Washington DC, 2009. Behrman, Jere R., The International Food Policy Research Institute (IFPRI) and the Mexican PROGRESA Anti-Poverty and Human Resource Investment Conditional Cash, World Development, Vol. 38, No. 10, pp. 1473-1485, 2010. Braun, Miguel; Chudnovsky Mariana; Ducot, Nicols y Weyrauch, Vanesa, Lejos de Thinktanklandia: los institutos de investigacin de polticas en los pases en desarrollo. CIPPEC, Buenos Aires, 2006. Brooks, David, Social Science Palooza, New York Times, December 6, 2010. Carden, Fred, Knowledge to Policy: Making the Most of Development Research, IDRC, New Delhi, 2009. Christiansen, Karin and Hovland, Ingie, The PRSP Initiative: Multilateral Policy Change and the Role of Research, ODI, London, 2003. Crewe, Emma and Young, John, Bridging Research and Policy: Context, Evidence and Links, ODI, London, 2002. DFID, Political economy analysis, A DFID practice paper, 2009. Downie, Anna, From access to action, Knowledge Services, Institute of Development Studies at the University of Sussex, Brighton, 2008. Gaarder, Marie and Briceo, Bertha, Institutionalization Of Government Evaluation: Balancing Trade-Offs, Journal of Development Effectiveness, 2010. Gaarder, Marie and Briceo, Bertha, Institutionalizing Evaluation: Review of International Experience, DFID, New Delhi, 2009. Gaarder, Marie, La importancia de la Evaluacin de Impacto de los PTC, Presentacin, Guatemala, 2008. Hovland, Ingie, Making a difference: M&E of policy research, ODI, London, 2007. Hovland, Ingie, Tools for Policy Impact: A Handbook for Researchers, ODI, London, 2004. International Initiative for Impact Evaluation, Principles for Impact Evaluation, consulted on www.3ieimpact.org on January 11th, 2011. IFPRI (International Food Policy Research Institute), Impact Evaluation: Assessing the Impact of Policy-Oriented Social Science Research, Washington DC, 2002. Jones, Harry, Policy making as a discourse, A Joint IKM Emergent ODI London, 2009.
51

Jones, Nicola and Villar, E. Situating children in international development policy: challenges involved in successful evidence-informed policy influencing, Evidence and Policy, vol4, no.1, 2008. Jones, Nicola; Jones, Harry; Steer, Liesbet; Datta, Ajoy, Improving impact evaluation production and use, ODI, London, 2009. Kingdon, John Alternatives and Public Policies, Harpens Collins, New York, 1984. Lardone, Martn and Roggero, Marcos, Estudio sobre monitoreo y evaluacin de la incidencia de la investigacin en polticas pblicas de Institutos de Investigacin de Polticas (IIPs) de la regin, CIPPEC/GDN, Buenos Aires, 2010. Lindquist, Evert, Discerning policy influence: Framework for a Strategic Evaluation of IDRC Supported Research School of public administration, University of Victoria, Victoria, 2001. MacKay, Keith, Cmo crear un sistema de SyE que contribuya a un buen gobierno, Banco Mundial, Grupo de evaluacin independiente, Washington DC, 2007. MacKay, Keith, Desarrollo de la capacidad de evaluacin, institucionalizacin de los sistemas de seguimiento y evaluacin para mejorar la gestin del sector pblico Banco Mundial, 2006. Malik, Khalid, Handbook on Monitoring and Evaluating for Results, UNDP, New York, 2002. Mayne, John, Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16, May 2008. Mendizabal, Enrique and Sample, Kristen, Dime a quin escuchas Think Tanks y partidos politicos en AmricaLatina, IDEA/OSI, 2009. Pollard, Amy and Court, Julius, How Civil Society Organisations Use Evidence to Influence Policy Processes, ODI, London, 2005. Pross, Paul, Group Politics and Public Policy Oxford University Press, Toronto, 1986. Rogers, Patricia, Using Programme Theory to Evaluate Complicated and Complex Aspects of interventions, RMIT University, Melbourne, 2008. Sandison, Peta, The utilization of evaluation, review of Humanitarian Action, ALNAP, 2005. Sen, Kunal, Literature Review on Rates of Return on Research , DFID, 2010. Weiss, The interface between evaluation and public policy, Harvard University, 2008. Weiss, Carol Hirschon, Nothing as practical as good theory Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families in New Approaches to Evaluating Community Initiatives, The Aspen Institute, Washington DC, 2005. Weyrauch, Vanesa and Ins Selvood, Weaving Global Networks: Handbook for policy influence, CIPPEC, Buenos Aires, 2007.
52

Weyrauch, Vanesa; Browne, Francisca; DAgostino, Julia; Richards, Clara, Tierra frtil. La gestin del conocimiento sobre incidencia en polticas pblicas se asoma en Amrica Latina, CIPPEC/GDN, Buenos Aires, 2010. White, Howard, Some Reflections on Current Debates in Impact Evaluation, International Initiative for Impact Evaluation Working Paper 1, April 2009. White, Howard, Theory-Based Impact Evaluation: Principles and Practice, International Initiative for Impact Evaluation Working Paper 3, June 2009. Yaron, Gil and Shaxson, Louise, Good practice in evidence informed policy: an initial review for DFID, 2008.
53

Authors
Vanesa Weyrauch Director of CIPPECs Civil Society Program. PhD candidate in Communications Sciences at Universidad Austral, focusing on civil society organizations communications to impact public policy. Postgraduate studies in Administration and Management, Harvard University; graduated in Communications Sciences from Universidad Austral. She has been consultant and analyst of organizational communications for several companies and non-profit organizations such as Trinity Communications, Siemens, and Fundacin Cambio Democrtico. Her research areas include the link between research and policy, and the relationship between State and civil society. With the support of ODI, she has created and directed the network EBPDN Latin America which gathers more than 200 CSOs and think tanks interested in influencing policy through research. Author of Weaving global networks (handbook on global and regional policy influence) and Learners, doers and teachers. Handbook for the monitoring, evaluation and knowledge management of the influence in public policies, among other publications. Besides, she has provided training and technical assistance on policy influence for policy research institutes and CSOs at the national and international levels. Gala Daz Langou. Coordinator of CIPPECs Social Protection Program. M.A. in Development Management and Policy, Georgetown University and Universidad de San Martn (thesis pending). Holds a BA in International Studies from the Universidad Torcuato Di Tella. Postgraduate studies in International Integration, Development and Public Policies from FLACSO University. She has been a Rotary Club Scholar. She has worked as a consultant and analyst for international organizations, private institutions and CSOs.
54

3IE Case Studies - Conceptual - Framework

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3IE Case Studies - Conceptual - Framework

Uploaded by

Copyright:

Available Formats

Conceptual Framework for 3ies Case Studies Series on Policy Influence of Impact Studies

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Tables and graphs

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

We explain the reasons behind this classification in the third section.

Sound Expectations. From Impact Evaluations to Policy Change

2- Policy influence as an objective of research and impact evaluations

We address this issue with greater detail in section 3.b.

Sound Expectations. From Impact Evaluations to Policy Change

Which will also be presented with greater depth in section 3.b.

Sound Expectations. From Impact Evaluations to Policy Change

The importance of influence goals

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Author Sandison (2005)

Types of use Instrumental use

Source: Jones et al (2009) p. 10.

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

The case studies should include:

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

3- Factors and forces

a) Determinants of the demand for research in the policy process

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Factor / force Political system

Sound Expectations. From Impact Evaluations to Policy Change

specialized types of organizations such

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Source: Lindquist (2001)

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Status of consensus on the policy base

Policy base largely intact but marginal issues arise

Core principles of policy base open to scrutiny

Number of actors involved in decision process

A few actors with responsibility to implement policy by policymakers

A few policymakers with a stake in the marginal issue

Type of information sought

Source: Lindquist (2001)

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

b) Determinants for the supply of research in the policy process

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change

Sound Expectations. From Impact Evaluations to Policy Change