2

1. Introduction In recent years there has been an increasing demand for aid agencies to demonstrate the results of their work. Despite this growing emphasis on performance and results from donors, policy makers and the general public, there exists little evidence on the real impact of aid on the lives of the intended recipients (Hoffman et al 2004; Darcy, 2005). In the development sector this lack of evidence on impact has been referred to as an evaluation gap, which exists because there are too few incentives to carry out impact studies, and too many obstacles including technical, bureaucratic and political challenges (Center for Global Development, 2006: 3). Similarly, it has been argued that the inability of humanitarian organizations to measure the impact of their work can be attributed to a number of methodological and organizational issues, as well as the challenging operating context in which humanitarian assistance is typically carried out (Hoffman et al 2004; Watson, 2008). Drawing upon the current literature on impact, as well as experiences from the Feinstein International Centers work on impact measurement, this paper will present and discuss some of the key challenges to impact measurement in the humanitarian and development sector. The paper also aims to demonstrate how at least some of the more salient methodological challenges can and have been addressed using participatory approaches to impact assessment.

Challenges to Impact Measurement

The difficulties in measuring the impact of humanitarian assistance interventions include a set of practical, methodological, organizational and institutional factors. Some of the key practical and methodological constraints to impact measurement have been summarized as problems of causality and attribution, ethical dilemmas around using control groups, a lack of basic data, the short term nature of interventions, limited capacity and resources for assessment, and a security environment that often limits access and space for assessment and analysis (Hoffman et al, 2004). Other methodological issues exist around the appropriate indicators, tools, methods, and the timing for impact assessments (Watson, 2008). Furthermore the ambiguity surrounding the definition of humanitarian assistance, and unclear intervention objectives represent another challenging dimension to impact measurement (Hoffman et al, 2004; Watson, 2008). Some of these practical constraints to impact measurement can arguably be linked directly to organizational and institutional (or sector wide) issues. For example, the lack of capacity and resources for impact assessment has much to do with a donor and agency culture that does not prioritize impact measurement, and therefore provides few incentives for impact assessment and learning (Center for Global Development, 2006; Watson, 2008). Some commentators reason that the compulsion to respond quickly to emergencies has created an organizational culture that values action over analysis (Hoffman et al, 2004: 6). Additionally, there is increasing evidence to suggest that aid organizations have become averse to risk (Kent, 2004; Hewett, 2006) and this has implications on organizational learning, of which impact measurement is one component (Watson, 2008). The following section will focus on donor and agency culture, and the

associated aversion to risk within the humanitarian sector as a lens with which to assess and unpack some of the key organizational constraints to impact assessment.

Organizational constraints to impact measurement and learning

In his speech on NGO risk management Andrew Hewett, the CEO of Oxfam, Australia presents numerous examples from the Tsunami Evaluation Coalition report which point to risk averse behavior by international NGOs. He argues that after the Asian Tsunami, this behavior manifested itself in less innovative responses to the disaster, in that aid agencies resorted to tried and trusted methods as a first and last resort (Hewett, 2006). He attributes this risk-averse behavior to a prioritization of the institutional imperative over the development imperative and suggests that this type of risk aversion stifles innovation (ibid). It would appear as though these assertions on risk-aversion amongst aid agencies also applies to, and his implications on impact measurement and learning. In his comments on organizational risk aversion, Randolph Kent (2004: 7) suggests that New types of information pose hazards and present potentially costly diversions. They threaten to disrupt or dismantle two essential pillars of organizational behavior - consistency and predictability without which those within and outside the organization would not know how to operate. Consistent with this, the information or results from an impact assessment are unpredictable and yet they have the potential to challenge organizational behavior and disrupt institutional operating procedures and norms. Or as one analyst proposes, action oriented organizational
3

learning will have implications on: the guiding ideas (or paradigms) upon which organizational practice is founded, and the types of organizational culture, structures, incentives and procedures which dominate (Pasteur, 2004: 6). It could be argued that from an ethical perspective aid agencies cannot afford to take risks, and to do so would be unjustifiable and irresponsible. This might explain the tendency in the humanitarian sector to favour conservative approaches over innovative ones (Hoffman et al, 2004: 13). However, the nature and emphasis of risk aversion in question appears to focus overwhelmingly on mitigating risks to the organization, not to the recipients of aid interventions. For example, during a training workshop under a Bill and Melinda Gates funded impact assessment initiative, a similar set of issues surfaced around the risks involved in impact measurement and learning. It became apparent that a major disincentive to measuring impact was the risk of discovering failure. One participant representing a large international aid agency stated that headquarters does not want to hear about failure, we only report success to our donors (Feinstein Center, 2006). Another participant offered a similar assessment on the disincentives to learning; we work in a competitive funding environment, where it is more important to show success than to learn (ibid). In his discussion paper on organizational learning in the humanitarian sector, Koenraad Van Brabant (1997, 3) echoes similar sentiments proposing that there is a tendency for agencies to hype up their performance in the face of growing competition for funding and that such an environment encourages defensive attitudes and an unwillingness to question assumptions, beliefs and attitudes. Reviews by the Active Learning Network for Accountability and Performance (ALNAP, 2002 and 2003) support this view by suggesting that the Incentive structures in agencies promote defensive behavior and a culture of blame (cited by Hoffman et al, 2004). Similarly, under the Gates Impact Assessment initiative, one workshop participant
4

maintained that there is no incentive to report on impact (Feinstein Center, 2006). Another participant whilst being questioned about their agencies ambitious project proposal said that HQ (headquarters) told us to do whatever it takes to get the funding (ibid). These statements support the premise that there is a common perception that funding for humanitarian assistance is strongly linked to success, and conversely that NGOs will be penalized in terms of future funding if they report on failure; the fear that failure will result in reduced funding is itself a powerful disincentive (Smillie and Minear, 2004: 223). This goes a long way toward explaining the defensive attitudes and motivations for exaggerating success in the humanitarian sector. It would also seem to imply that the competitive funding environment provides a strong disincentive to report on project impact as impact assessment results are unpredictable and may turn out to be negative, or at least less positive than hoped for. In such an environment, this unpredictability represents an unacceptable risk, as one impact workshop participant admitted Its better not to measure impact, if we were to discover we were having little or no impact, its better not to know about it (Feinstein Center, 2006).

As surprising as this head in the sand mentality may seem, it probably has less to do with a lack of accountability or desire to do better, than it has to do with the dysfunctional way in which donor funds are disbursed and administered. Contrary to accusations that they are not accountable, international NGOs are heavily scrutinized by their donors, the general public, and the media, and some argue that the problem is not so much a lack of accountability, but too much accountability, to the wrong people, for the wrong things (Smillie and Minear, 2004: 203). The demand for accountability and transparency in the sector, in tandem with a donor and media intolerance for failure encourages agencies to downplay failures in order to preserve
5

funding (ibid). If this is the case, not only does it represent a disincentive to impact assessment, but the possibility of negative results may potentially pose a considerable risk to the organization.

Evidence of this misplaced sense of accountability can be seen in conventional humanitarian reporting systems which emphasize upward financial accountability, and largely ignore the perceptions of aid recipients. These reporting mechanisms almost exclusively measure the process of project implementation and service delivery as opposed to the real impact on the lives of project participants (Hoffman et al, 2004; Shoham, 2004; Catley et al, 2008; Watson, 2008). While there are certainly practical and methodological reasons for this - its easier to quantify and measure process than impact, this focus on process defines a system of upward accountability. This kind of reporting system also reinforces the power dynamics of aid which emphasize the fact that resources flow from the top down, and beneficiaries exert little or no influence on the way assistance is provided (ALNAP, 2003 as cited by Hoffman et al, 2004), or indeed evaluated. Excluding aid recipients from project evaluations, and focusing on measuring project implementation as opposed to impact, ensures that success is defined and measured by donors and the implementing organization rather than project participants or communities. In recent years there has been an increasing demand for humanitarian agencies to adopt results based management reporting systems. However, these largely focus on measuring output as opposed to real impact (Hoffman et al, 2004). The benchmarks or results indicators used are almost exclusively defined by donors and agencies with little or no consultation with project participants, and usually consist of either process or proxy indicators of impact. This emphasis on external process and proxy impact indicators as opposed to community or client based
6

indicators is a less risky prospect for implementing agencies, as the results are sure to be more predictable. In an environment where failure is used to penalize as opposed to learn, this kind of reporting is a safer option for NGOs. As long as agencies can deliver on the inputs and services outlined in their proposals, favorable results can more or less be predicted. In other words results on performance can largely be managed and controlled, even in contexts which are typically associated with unpredictability and complexity. The emphasis on these types of indicators suggests that being evaluated by anyone elses criteria, for example those of project participants, poses too much of a risk for both donors and NGOs alike.

Some commentators argue that in many cases humanitarian action has become a substitute for political and military inaction, and although the demand for this type of reporting and accountability is presumably meant to satisfy tax payers that aid money is being well spent, in reality it is more about donor control and threat (Smillie and Minear, 2004). If accurate, this might explain why donors in general do not demand agencies to report on impact, nor do they fund impact assessments. An alternative explanation might be the interpretation that donor governments are by definition rule-based bureaucracies that are incapable of processing complexity; as such there is a demand that information be summarized and pared down to the basic essentials that administrators can absorb (Duffield, 2001). The basic quantification and over simplification of results that this type of reporting yields tells us very little about real impact, and only creates the illusion of a replicable and predictable environment and a mechanical understanding of the impact of aid (ibid).

Not to suggest that financial accountability and the monitoring of project implementation is not important, it is, but it doesnt always tell us much about impact and so is rarely useful for informing policy and practice. There are of course methodological constraints and challenges involved in collecting more complex information in a measurable format which will be discussed in section five.

Some analysts argue that the current approach to evaluation is limited in imagination, scope and learning potential (Smillie and Minear, 2004: 224).By deliberately avoiding the lessons that might be learned from failure, this approach effectively drives real lessons underground (ibid). In order for the humanitarian sector to more effectively measure the impact of their work, it requires a culture shift away from risk aversity and the focus on quantitative outputs, to an emphasis on learning (Watson, 2008: 38). Without risk there is no learning, as the concept of learning implies that you will make mistakes, and any learning approach requires some level of risk. One recommendation that has been proposed is that that the current approach to evaluation should be replaced with one that puts learning at center stage (Smillie and Minear, 2004:224). Such an approach if done well would likely satisfy donor accountability requirements in the process, but as a by-product as opposed to being the only product (ibid).

The arguments presented in this section aim to illustrate that the current approach to monitoring evaluation and donor reporting provides no incentive for impact measurement and learning. Indeed, the existing incentive structures attached to donor funding appear to penalize failure, thus representing a major disincentive to measuring impact, as impact by definition implies laying open the mistakes and uncertainties inherent in development work (Roche, 1999: 3).
8

These incentive structures encourage and promote a risk averse donor and agency culture which only serves to further discourage any acknowledgment of failure and by implication discourages learning. There are certainly a number of practical and methodological challenges to measuring impact and some of these have nothing to do with the organizational and institutional issues mentioned in this section. These practical and methodological constraints are often used as an excuse and justification for not measuring impact. The following section will take a look at some of the most salient practical and methodological challenges associated with measuring impact in a humanitarian context.

Methodological and Practical Constraints to Assessing Impact

4.1 Time, timing and resource issues Most if not all-humanitarian interventions take place under challenging conditions often in remote areas characterized by insecurity, limited infrastructure and a lack of basic services. Simple access to affected populations can be problematic and data collection activities such as needs and impact assessment are understandably perceived to be a luxury that may put agency staff at increased risk. Security and access therefore represent two potential challenges to impact assessment in many humanitarian contexts.

Time and timing represent two more obstacles to humanitarian impact assessment (Watson, 2008). As one impact workshop participant summarized; the short term nature of projects does not lend itself to measuring impact (Feinstein Center, 2006). In situations of chronic suffering
9

there is pressure to respond immediately and the priority is to deliver assistance not to assess it. Typically humanitarian project timeframes are short, and this has a number of implications on impact measurement (ALNAP, 2002 and 2003 as Cited by Hoffman et al, 2004). Firstly, this time constraint reinforces the prioritization on implementing the hardware project components such as the delivery of food, shelter, medicine, and water over the software components such as monitoring and evaluation, and other data collection activities. Certainly an ethical argument can be made to justify this prioritization which explains why time is rarely allocated for data collection. This lack of data can pose a methodological problem for downstream impact assessments, and the lack of any meaningful baseline data is often cited as a major challenge to measuring impact (Hoffman et al, 2004; Watson, 2008). There are also ethical considerations surrounding the cost benefits associated with peoples participation in an assessment. Some commentators speculate that in a humanitarian crisis, the time spent participating in an impact assessment may actually have a negative impact on livelihoods production (Mayoux and Chambers, 2005; Burns and Suji, 2007).

Short term project cycles may also imply that a project doesnt have any significant impact within the project timeframe (Oakley et al, 1998). As humanitarian assistance has evolved to include longer term livelihoods support and recovery projects, impact might be delayed for up to at least one agricultural season, by which time the project is most likely to have ended - theres no point assessing impact if you know you are not going to find any. The appropriate time to do an impact assessment is often determined by factors such as seasonal population movement and the timing of agricultural, cultural and religious activities (Catley et al, 2008: Watson, 2008).

10

These factors have implications on access, and add an extra set of practical challenges to impact assessment.

Although distilled down to the practical issues of time and timing there is some correlation between some of these challenges and the organizational and institutional constraints to impact assessment mentioned. Many of the challenges around timing fall within the broader debate on what defines humanitarian action and the restrictions imposed on humanitarian funding. Time constraints also serve to reinforce an action as opposed to analysis oriented organizational culture (Hoffman et al, 2004). The issue of short project timeframes undermines the incentive to carry out an impact study, as the learning benefits are unlikely to be realized by the project implementers (Center for Global Development, 2006).

The prioritization of the hardware as opposed to software project activities is not limited to time, but also applies to the allocation of other resources. The overarching perception appears to be that where funding is limited it seems wasteful to spend valuable resources on collecting data. This perception characterizes an industry that is averse to learning and what on the surface may appear to be a practical decision to maximize impact is in reality a collective act of faith or assumption that humanitarian response automatically translates into impact (Darcy, 2005).

4.2 Issues around defining impact and identifying indicators

Within the humanitarian sector there is a lack of clarity regarding the definition of impact. This can be challenging when it comes to identifying indicators with which to measure impact
11

against. As mentioned conventional monitoring and evaluation systems focus on measuring process or output as opposed to real impact (Shoham, 2004; Hoffman et al, 2004; Catley et al, 2008; Watson, 2008).

During an impact assessment workshop that included representatives from seven different projects, participants were asked to list all the indicators they were using for monitoring and evaluation purposes. They were then asked to classify each indicator as a process indicator, a proxy impact indicator, or a real impact indicator. Out of a sample of 129 indicators listed, 63 were classified as process indicators, 36 as proxy impact indicators, and 30 as real impact indicators, and most of those identified as real impact indicators were in fact only proxy indicators (Feinstein Center, 2006). During the same workshop, some participants actually equated impact assessment with an audit type evaluation (ibid). This example suggests that the overemphasis on measuring process or even proxy indicators of impact has blurred peoples understanding of impact within the sector. Most definitions of impact in the humanitarian and development literature involve the concepts of cause and effect, or change and attribution, with change and effect representing impact either positive or negative (Roche, 1999; Hoffman et al, 2004; Watson, 2008; Catley et al, 2008). While this might provide a good starting point, how does one measure change and what exactly do we mean by change?

One of the problems with measuring impact or change in a humanitarian context is the challenge of trying to measure the counterfactual, or what would have happened without a particular intervention (Center for Global Development, 2006). Where cause and effect is already established such as the impact of a particular vaccine on mortality or disease prevalence, then the
12

number of vaccines administered could provide a useful proxy indicator of impact. However, other than in some health, nutrition, and water interventions in which indicators of project performance should relate to international standards, for many interventions there are no gold standards for measuring project impact. For example, the Sphere handbook has no clear standards for food security or livelihoods interventions (Catley et al, 2008: 6).

Suggested standardized indicators for impact measurement in a humanitarian context include crude mortality rates and nutritional status of children under five (Hoffman et al, 2004). But again there are no standard methods for mortality surveys and it is extremely challenging to collect reliable mortality data in an emergency context (ibid). Other commentators suggest that while mortality data might be useful in assessing program level impact, it has limited value in assessing the impact of individual projects (Checchi and Roberts, 2005). Similarly, there are quality control issues around nutrition data in emergencies as well as issues around attribution (Young and Jaspers, 2006). Furthermore, nutrition and mortality data might be useful for looking at the impact of a specific health intervention for example, but they do not tell us about the impact of interventions designed to improve or protect livelihoods assets or mitigate the employment of various coping mechanisms (Shoham, 2004; Watson, 2008; Catley et al, 2008). Consistent with this, project indicators are typically tied to project objectives, but many projects aim to deliver benefits against qualitative indicators of impact such as protection, dignity, hope, status, trust, confidence, well being or social cohesion (Hoffman et al, 2004; Catley et al, 2008). Seeing as these types of indictors cannot be quantified, a common perception is that it is difficult to collect rigorous, representative data on impact using these indicators. This is often used to

13

justify why conventional monitoring and evaluation systems focus on measuring the process of project implementation (ibid).

4.3 Difficulties in assessing attribution

One of the most challenging methodological constraints to impact measurement involves the question of attribution. In any community where a project is implemented changes will take place over time regardless of the project. Other changes or effects might take place as a direct result of the project and these can be attributed to the project (Catley et al, 2008). In epidemiological research the relationship between intervention and effect relates to the concepts of association and causation, and the effect of an intervention is measured by statistical comparison of a treatment or intervention population with a control group (Abebe et al, 2008). The use of controls in a humanitarian context raises a number of ethical concerns, as it implies the deliberate exclusion of the control population from humanitarian interventions (Catley et al, 2008, Abebe et al 2008). Even in a stable development context the identification of a reliable control group raises a number of methodological issues involving the independent variables associated with targeting inclusion and exclusion (Mayoux and Chambers, 2007). Aside from these considerations, there are also a number of practical issues in terms of logistics and costs which makes the use of controls in a humanitarian context challenging. The following table illustrates some of the practical and ethical concerns with control groups identified by project participants during an impact assessment workshop.

14

Table 4.1 Practical and ethical concerns with using control groups Identification of a control population with similar characteristics Willingness of comparative group to participate openly and honestly if incentives are given for participation, is this really a true control group? How can you be sure that the control group will not receive assistance from another source? Potential security risk to NGO staff for excluding the control group from the project If a control group is selected from non project participants in the same community, how can you be sure that they do not indirectly benefit from the project? Increased cost and time of including a control group, staff, vehicles Exclusion of one community goes against the humanitarian imperative and the principles of participation Its unethical to use human placebos in a humanitarian context good research protocol should not take precedence over the provision of assistance The use of a control group is disrespectful of peoples time Raising expectations of the control group will the information be reliable? The extra resources required to include a control group should be used to assist the excluded community Using controls could potentially create tensions or even fuel conflict between recipient and non-recipient communities Source: Catley et al, 2008

An alternative way to assess attribution that avoids the ethical implications is to measure a relevant variable or indicator before and after an intervention and assume that a significant change after the intervention is evidence of impact (Abebe et al, 2008). This can be problematic as the key assumption ignores external factors contributing to change. Also, as mentioned where there is no meaningful baseline data as is often the case in a humanitarian context, there is no reference point or benchmark against which to measure impact.

Given all the organizational, practical and methodological constraints it is understandable that impact assessment is rarely done in the humanitarian sector. There are also clear overlaps between some of the organizational and practical and methodological challenges. The key methodological issues involve the challenges of identifying appropriate indicators, the lack of
15

baseline data, and the ethical concerns around assessing attribution. The next section will explain how to some extent these methodological issues can and have been addressed using participatory approaches to impact assessment at the project level.

5. Applying participatory methods to the madness

The use of standardized participatory methods to generate statistically representative data or participatory numbers has been extensively applied in the development sector to answer a variety of questions including the measurement of project impact (Mayoux, and Chambers, 2005; Chambers, 2007). Arguably, the operational context and longer-term nature of development projects makes them better suited to participatory approaches. However, many of the methodological issues and challenges around impact assessment apply to both contexts and have been long debated in the development literature. Drawing upon experiences from the application of participatory research in the development sector, since the mid nineties, a number of organizations have been developing and adapting participatory approaches to measuring the impact of projects in a number of humanitarian contexts including Niger (Burns and Suji, 2007; Burns et al, 2008) Zimbabwe (Burns and Suji, 2007) southern Somalia (Hopkins, 2002; 2004), Southern Sudan (Catley, 1999; Hopkins and Short, 2001) and Somaliland (ActionAid, 1994). These examples demonstrate that even in emergency contexts characterized by political instability, insecurity, and protracted crisis, participatory approaches can be applied to generate meaningful evidence of project impact, and offer certain advantages over more conventional approaches to impact measurement.

16

5.1 Identifying appropriate indicators

There are a number of key elements to Participatory Impact Assessment (PIA) that make it well suited to measuring impact in the challenging contexts typically associated with humanitarian interventions. The approach is flexible and can be adapted to different types of projects and environments and has a number of distinct advantages over more conventional monitoring and evaluation and research approaches. For example, the PIA approach emphasizes the use of community or client based indicators of impact which not only shifts the focus from assessing process to real impact, but it does it from the perspective of aid recipients as opposed to that of external stakeholders. This shift in emphasis acknowledges that the experience and ideas of aid recipients are not irrelevant and that they are capable of defining their own indicators of change (Catley, 1998; Irvine et al, 2004). This goes some way towards addressing the demand from organizations such as ALNAP and the Humanitarian Accountability Partnership (HAP-I) for greater downward accountability within the humanitarian sector. The use of participatory methods also allows for retrospective baselines to be used, which is useful in overcoming issues around the lack of baseline data. The approach can also be useful in overcoming some the ethical concerns around attribution as participatory ranking and scoring methods can be used to measure the relative importance of both project and non project factors against any assessed impact. One of the key innovations of PIA is the standardization and systematic application of participatory methods with different informants to collect numerical data which can then be analyzed using more conventional statistical tests.

17

The traditional focus on the use of process indicators to measure results only really tells us about project performance from the perspective of the people who identified those indicators. Typically, process indicators are identified by project managers, monitoring and evaluation specialists or consultants. This process rarely involves aid recipients, who may or may not be interested in the actual number of medical kits delivered or the number of training workshops that took place. These types of indicators rarely tell us much about real impact or what changes a project brought to the lives of the participants.

As discussed, proxy impact indicators can be useful where cause and effect are clearly established, but this is not always the case. For example, a good proxy impact indicator for an animal health project might be an increase in the number of household livestock assets (herd size), but this indicator does not tell us how people benefited from having more livestock. It may be that an increase in livestock assets actually created tension over water and grazing resources with a neighboring community enhancing the potential for conflict. Participatory Impact Assessment tries to go beyond looking at process and proxy indicators by asking the fundamental question, what difference did the project have on peoples lives? One way of answering this question is by simply asking project participants what benefits or changes they experienced if any, as a result of the project. For example, an animal health project in southern Sudan resulted in improved animal health, a reduction in livestock mortality and an associated increase in herd size. In order to understand the impact of this on peoples lives, project participants were asked what benefits they derived from their livestock. Ten key benefits were identified, including milk and meat, the ability to pay dowries and income from livestock sales. These were then prioritized by assessing the relative importance of each benefit using proportional piling (Catley, 1999).
18

These community identified benefits provide useful impact indicators which can tell us far more about project impact than an assessment that limits itself to looking at just an increase in herd size. The identification of community impact indicators can even go a step further and look at utilization. For, example if milk and income are derived from livestock then you would ask project participants how these were utilized. Milk utilization such as consumption, charity, or sales could then be used as impact indicators (Burns et al, 2008). Alternatively project derived income might be spent on a variety of goods and services, these goods and services can also be used as impact indicators.

Another advantage of identifying community benefits or impact indicators is that the process can reveal priorities that may not have otherwise been identified or even considered in the project objectives. For example, during an impact assessment of a food security project in Zimbabwe, project participants suggested that one of the key project benefits was an increased ability to cover education related expenses from project derived income, whereas the project objectives and monitoring and evaluation indicators focused mainly on the anticipated food security outcomes (Burns and Suji, 2007).

5.2 Measuring change

One of the challenges with these types of indicators is that they are difficult to quantify in exact terms, and so how do you go about measuring them? One way of doing this is by comparing the relative importance of these indicators against other indictors or variables. Using the example of milk and income derived from project livestock, if you were to determine that the milk was
19

consumed within the household, this could be measured by assessing the relative contribution of this milk to the household food basket, which in turn gives you an understanding of the relative impact of the milk on household food security. Similarly the income derived from project related livestock sales might be measured as a contribution to overall household income in comparison to all other income sources. These kinds of measurements can be assessed using simple participatory tools such as proportional piling. Typically, this method would involve a project participant being asked to distribute a given number of counters amongst a set of indicators (such as income sources) with the greatest number of counters representing the most important indicator and the least number of counters representing the least important. Visual aids representing each of the different indicators might be used to facilitate the exercise.

Although indicators such as income derived from livestock sales can be quantified, qualitative indicators such as improved knowledge, status, dignity, participation or social cohesion are less tangible. The qualitative nature of these types of indicators is often used to justify the focus on more easily quantifiable process indicators associated with conventional monitoring and evaluation systems. Nevertheless, participatory exercises such as ranking and scoring can be used to assign a numerical value to qualitative indicators allowing them to be measured, at least in relative terms. This concept of participatory numbers has been extensively applied in the development sector (Chambers, 2007). One example of this was the use of radar diagrams to assess changing levels of participation throughout a development project. This was done by asking project participants to score their level of participation in five different project components such as needs assessment, resource mobilization, and management for each year of the project. Each component was assigned an individual spoke of the radar diagram, with each
20

spoke calibrated into five bars, each bar representing levels of participation from zero to five. The results showed an increase in the levels of participation over time (Rifkin et al, 1988; Catley, 1999). This demonstrates how even an intangible indicator such as participation can be measured in numerical terms.

5.2.1 Retrospective and nominal baselines

As discussed, the time constraints associated with humanitarian projects means that there is rarely any time to collect any baseline data or baseline impact indicators. Again, participatory exercises such as before and after scoring have been used to develop retrospective baselines. Although, these may be subject to recall bias and other types of non-sampling error they do provide a useable baseline against which to measure impact or change (Abebe et al, 2008). This involves participants doing a scoring exercise to represent a given situation before a project started and then repeating the exercise to represent the current situation (after the project). This method has been used in a variety of humanitarian contexts to assess changes in cattle disease during an animal health project (Catley, 1999); changes in crop production during an agricultural support intervention (Burns and Suji, 2007); and changes in household food and income sources for different food security projects (Burns and Suji, 2007, Burns et al, 2008). It has also been used to assess changes in household expenditure as a way capturing investments in productive assets and livelihoods diversification over time (ibid). A similar method was used to compare milk production in healthy and unhealthy cattle during an animal health project in southern Sudan. This was done by using a hundred counters to represent milk production in healthy cattle. Participants were then asked to make separate piles to represent the difference in production for
21

cattle infected with eight common diseases (Catley, 1999). Similarly, actual changes in food production and income were assessed using a nominal baseline of ten counters. Participants were then asked to either add counters to, or take counters away from the original pile of ten to demonstrate positive or negative changes over time (Burns and Suji, 2007).

If applied systematically, with carefully selected samples, these methods are capable of generating conventional statistic measures of association. One of the key innovations of PIA is the standardization and repetition of the methods to generate a set of reliable results. A common concern with participatory methods is that they are often only repeated a limited number of times, and as such they do not produce convincing or representative evidence (Catley et al, 2008). The process of assigning a value to indicators allows the data to be presented numerically and analyzed using various types of statistical tests. If the participatory methods are standardized and repeated systematically they can generate meaningful and representative data on project impact (Ibid).

5.3 Assessing attribution

Although these methods can capture change against relevant impact indicators by using retrospective or nominal baselines, they do make the assumption that any significant change is evidence of project impact (Abebe et al, 2008). However, when this data is triangulated with other sources of information, for example a projects process monitoring data, this can help to establish and validate cause and effect providing compelling evidence of impact. There are also some specific methods that can generate evidence of attribution without having to resort to using
22

controls. On a hierarchy of evidence the information generated from these methods may not be as scientifically rigorous as that provided by a good control group (Catley et al, 2008). However, the advantage is that they do provide an alternative means of attribution that doesnt involve the ethical and practical constraints associated with the use of controls in a humanitarian setting (ibid). Essentially these methods involve assessing the relative importance of project and nonproject factors in contributing to any identified impact or change (ibid). This typically involves the identification of all factors that might have contributed to any assessed impact, which can be done by simply asking project participants what these factors are, and recording them. Participants might then be asked to rank or score these in order of importance (Ibid).

Alternatively, a matrix scoring exercise can be done to assess the relative importance of project and non-project factors against a whole set of different indicators of impact. In an example from Ethiopia, seven types of drought interventions were compared against seven different impact indicators by asking participants to distribute thirty counters amongst the intervention indicators in order of preference. This exercise was then repeated for each of the impact indicators and an overall preference score was calculated from the seven sets of results (Abebe et al, 2008).

Another type of attribution method has been used as a follow up to a set of before and after impact scoring exercises. Where the results from these exercises showed any change, participants were asked to explain these changes, and the responses were summarized into categories and tallied. The highest number of responses was then assumed to represent the most important factor with a decline in importance being attached to all the other factors corresponding to the number of responses (Burns et al, 2008). These types of attribution methods work well where
23

retrospective baselines such as those derived from before and after scoring exercises are used, as the explanations or factors contributing to change can be identified and scored by the same respondents that assessed the change in question. With more conventional baselines, the practical issue of identifying the same respondents for the final assessment would be more challenging, particularly in areas associated with high levels of population movement or pastoralist communities.

5.3.1 Using simple controls

In some cases it is also possible to use simple controls to assess attribution without the practical or ethical concerns typically associated with this method. An example from a community animal health project compares camel disease in livestock treated by community animal health workers as opposed to untreated camels (Admassu et al, 2005). Similarly, an impact assessment of a supplementary livestock feed intervention compared mortality rates between fed and unfed cattle during a recent drought in Ethiopia (Bekele, 2008). Another example from Zimbabwe compared the duration of household food security between project and non project participants attending the same focus group discussion (Burns and Suji, 2007). Nevertheless, caution should be applied with this last control, as with any control involving people in a complex environment there are a number of independent variables at play. Also as discussed by Mayoux and Chambers (2005) with reference to micro-finance interventions, the targeting criteria for project participation can by definition make non-project participants an unsuitable control group. Similarly in the example from Zimbabwe, the targeting criteria implied that the socio economic profile of the two groups made them incomparable. Having said this, the intervention group in this particular context was
24

technically more vulnerable and the results indicated that this group experienced a longer period of food security, suggesting that the project did have a considerable impact on household food security. Some analysts caution that participatory assessments can raise expectations (ibid), and in this particular case study there may well have been an exaggeration bias motivated by expectation of additional assistance in the project area (Burns and Suji, 2007). The use of simple controls will largely depend on the type of intervention and the opportunities that arise. Where these possibilities exist they can provide an extra level of attribution which can be useful and interesting even if the methods are not always scientifically that robust.

6. Demystifying impact measurement The previous section describes how some of the key methodological constraints to impact measurement, such as the identification of appropriate indicators, the lack of baseline data, issues around attribution, and concerns over soft participatory approaches can and have been overcome using participatory impact assessment. Some of the methods described have been applied to impact measurement for over fifteen years now, yet still very little is known about the impact of humanitarian assistance projects. Where participatory impact assessment has been applied in a systematic and rigorous way, not only has it provided compelling evidence of project impact, the results have also been used to improve programming, and inform policy and best practice (Hopkins and Short, 2002; Behnke et al, 2008; Catley et al, 2009). The methodology and results from participatory impact assessments have been published in academic journals (Hopkins and Short, 2002; Nalitolela and Allport, 2002; Abebe et al, 2008) and disseminated on various relief and development websites such as relief-web and ELDIS. If an approach to measuring impact exists, that can seemingly overcome the methodological constraints discussed,
25

and is rigorous enough to pass the scrutiny of academics and policy makers, then why is impact assessment practice so rarely applied to humanitarian assistance projects? One reason might be that the institutional and organizational constraints and disincentives discussed represent a far greater challenge to impact measurement than do the methodological constraints. If donors dont require or fund impact assessment, and humanitarian organizations dont provide any incentive or space for learning and analysis, then it doesnt really matter what methodologies exist or how good the tools are. If there is no demand, then agencies simply will not carry out impact assessments. The Center for Global Development (2006) presents a scenario whereby in ten years from now the development sector is still bemoaning the lack of evidence of what really works. Similarly, unless the funding and learning incentives change within the humanitarian sector, in ten years time we will still be asking why humanitarian organizations dont measure the impact of their work. If however, donors increasingly demand and fund impact assessments then agencies will have little choice but to start assessing impact. However, the motivation behind any impact assessment should be soundly based in learning objectives. If impact evaluations are directly linked to future funding for agencies this will undermine their value, and the only real function they will serve will be to penalize agencies for failures, thus validating the argument that the purpose of evaluations is largely about control and threat (Smillie and Minear, 2004) or donor policing (Mayoux and Chambers, 2005: 273). If these kinds of disincentives exist, then aid organizations are liable to respond by massaging or presenting the results of their impact assessments to demonstrate success rather than failure. Although this could be partly overcome by independent impact evaluations, external evaluators will not have the same kind of access to
26

project participants, and the internal learning benefits will be largely lost during the process. Furthermore the objectivity of external evaluators is highly questionable in an industry where an evaluators livelihood depends on consultancies with the very organizations they are evaluating. Although some degree of objectivity and technical oversight might be achieved by partnering independent research institutions with implementing agencies, in order for them to have any real value in terms of learning benefits, impact assessment needs to be institutionalized within organizations. 6.1 Donor demand Although there is certainly some truth to the argument that donors do not demand or fund impact assessments, in recent years this has started to change. In Ethiopia, USAID has been funding impact assessments of drought related interventions under the Pastoralist Livelihoods Initiative, and is currently supporting a longitudinal impact study of a productive safety net support program. The Bill & Melinda Gates foundation has also been funding impact assessments of livelihoods interventions under the Sub Saharan Africa Famine Relief Effort, and supporting the development of impact assessment methodologies under the same initiative as well as under the Emergency Capacity Building project. If this trend continues, its more than likely that other donors will follow suit, in which case the incentive structures associated with impact measurement will undoubtedly change. 6.2 Opportunities and costs The argument that impact assessment is too costly also needs to be put into perspective. There is no doubt that in terms of scale and need, humanitarian assistance is underfunded and resources
27

are limited. However, the same arguments about costs and timing that are applied to impact assessment are also used to justify the lack of needs assessment data; and the disparity between humanitarian appeals and actual funding suggests that the assessment of need is often exaggerated creating skepticism from donor agencies (Maxwell and Burns, 2008). If this is the case then theres actually very little reference point for the cost benefits of good needs assessment. It logically follows that the cost in financial terms of over-estimating needs or the cost in terms of human suffering of under-estimating needs justifies the investment in good needs assessment. Similarly with impact assessment, if a project has no impact and we never find out about it, and then implement the same project in another community, and this too has no impact, the potential human and financial cost of not measuring impact might be considerable. Or, as the Center for Global Development (2006: 20) puts it ignorance is more expensive than impact assessment. A recent impact assessment of a multi-million dollar livestock vaccination project bears this out. The results of the assessment showed no difference in mortality rates between vaccinated and un-vaccinated livestock. This was attributed to weaknesses in the design and implementation of the project, the use of inappropriate vaccines, low coverage, incorrect timing, and problems with dosing and vaccine storage (Abebe et al, 2008). This example clearly illustrates the costs of not doing an impact assessment. However, there are also missed learning opportunities which factor into the overall calculation on the cost benefits of measuring impact. The results from the impact assessments carried out under the USAID funded Pastoralist Livelihoods Initiative in Ethiopia were ultimately used to develop national guidelines for livestock relief interventions in pastoralist areas. The overall project budget amounted to $ US 29 Million, of which between 2-3% was estimated to have been spent on impact assessment or, on discovering if the project had been effective in improving pastoralist welfare (Behnke et
28

al, 2008). An independent evaluation of the project concluded that this amount was disproportionate in comparison to what was spent on implementation, and suggested that these impact assessments were the most innovative and successful aspect of the program, recommending that consideration be given to using similar types of impact assessments as a supplement to USAIDs standard internal evaluation procedures (ibid). The report also suggested that the impact assessments are likely to be one of the lasting contributions (of the program) to pastoralist livestock development policy and practices in Ethiopia and more generally in semiarid Africa (Behnke et al, 2008: 44). This example not only demonstrates the missed opportunities of not doing impact assessments, but also illustrates some of the incentives that impact assessment can provide for donors, governments and policy makers. Not to suggest that every single humanitarian intervention requires or demands an impact assessment, but the potential costs and missed opportunities of not doing impact assessment can no longer be ignored. There are for sure time and timing issues that present a significant challenge to impact measurement, but arguably because impact assessment has not been given the priority it deserves. Essentially issues around time and timing arguably come down to resources, but the resources allocated to impact assessment are insignificant in comparison to the cost of poorly designed and poorly implemented programs. One might expect that the incentives in terms of the potential savings and learning benefits should engender the demand for such resources.

29

7. Measuring impact, from theory to practice The examples discussed in this paper demonstrate that good impact assessments can be carried out in a wide variety of humanitarian contexts. Although most of the case studies involve food security, livelihoods or livestock projects which some would argue fall into the grey area between emergency humanitarian relief and longer term reconstruction and development, they took place in remote contexts characterized by political instability, conflict, and chronic vulnerability. Many were also funded under the umbrella of humanitarian assistance as defined in its broader application. Nonetheless, similar tools were used by the World Food Programme as part of their routine monitoring to measure the impact and utilization of food aid during the Bhar el Ghazal famine in Sudan in 1998 (Maxwell and Burns, 2008). Although this was not done systematically, it does suggest that the impact of the most visible form of humanitarian assistance (Walker and Maxwell, 2008) can be measured using the methods described. The use of participatory methods to generate numerically representative data on qualitative indicators such as participation, social cohesion, dignity, status, hope and well being suggests that the application of these methods might also be adapted to assess other types of humanitarian programming, and that new innovations may enable agencies to measure the less tangible but oftentimes more important aspects of project impact. Ultimately, if impact assessment can be institutionalized within aid organizations, the costs will eventually come down. At present impact assessment capacities are fairly limited and this is also used as a justification not to measure impact. Arguably though, if the demand exists, capacities will be developed. In regards to this, most aid agencies have field staff with considerable assessment skills and experience, and the tools and methods discussed in this paper are by no
30

means beyond the reach of most humanitarian practitioners. Discussions on causation, association, attribution and effect might be interesting for academics, but are not very useful when it comes to convincing practitioners or project participants on the value of impact assessment. Therefore, if impact assessments are to become a common feature of humanitarian assistance projects, there is a need to demystify impact measurement and provide aid practitioners with simple accessible impact measurement tools. The discussed examples illustrate that these tools exist, and demonstrate that the key methodological challenges can, and have largely been overcome. If these tools and the required skills to use them can be transferred to aid agencies then there will be one less excuse for not measuring impact. However, Linda Mayoux and Robert Chambers (2005: 275) caution that although the good use of participatory tools does not require high levels of formal education it does require training experience and importantly appropriate behavior and attitude. Although there will always be methodological imperfections and compromises involved in measuring impact in a humanitarian context, this should not be used as an excuse not to measure impact. The humanitarian sector needs more quality impact assessments, not perfect ones. If this happens, and aid agencies start conducting impact assessments and experimenting with different tools, then methodological improvements and innovations are sure to follow. More importantly we can learn a considerable amount about what does, and doesnt work as part of the process.

31

References Abebe, D., Catley, A., Admassu, B. and Bekele, G. (2009). Using Participatory Impact Assessment (PIA) to inform policy: Lessons from Ethiopia. In: Scoones, I., Thomson, J. and Chambers, R. (eds.), Farmer First Revisited. Practical Action Publishing, in press Abebe, D., Cullis, A., Catley, A., Aklilu, Y., Mekonnen, G. and Ghebrechirstos, Y. (2008). Livelihoods impact and benefit-cost estimation of a commercial de-stocking relief intervention in Moyale district, southern Ethiopia. Disasters 32/2 June 2008 (Online Early) ActionAid (1994) ACTIONAID Somaliland Programme Review/Evaluation. ActionAid, London Admassu, B., Nega, S., Haile, T., Abera, B., Hussein, A. and Catley, A. (2005). Impact assessment of a community-based animal health project in Dollo Ado and Dollo Bay districts, southern Ethiopia. Tropical Animal Health and Production 37/1, 33-48 Behnke, R., Kerven, C., Teshome, A. (2008). Evaluation of USAID Pastoral Development Projects in Ethiopia. Odessa Centre Ltd. and USAID Ethiopia, Addis Ababa. Bekele, G. and Abera, T. (2008). Livelihoods-based Drought Response in Ethiopia: Impact Assessment of Livestock Feed Supplementation. Feinstein International Center, Tufts University and Save the Children US, Addis Ababa Burns, J. and Suji, O. (2007). Impact Assessment of the Chical Integrated Recovery Action Project, Niger. Feinstein International Center, Medford, MA https://wikis.uit.tufts.edu/confluence/display/FIC/Impact+Assessment+of+the+Chical+Integrated+Rec overy+Action+Project%2C+Niger Burns, J., Suji, O. and Reynolds, A. (2008). Impact Assessment of the Pastoralist Survival and Recovery Project,, Dakoro, Niger. Feinstein International Center, Medford, MA https://wikis.uit.tufts.edu/confluence/display/FIC/Impact+Assessment+of+the+Pastoralist+Survival+a nd+Recovery+Project+Dakoro%2C+Niger Burns, J and Suji, O (2007), Impact Assessment of the Gokwe Integrated Recovery Action Project, Zimbabwe. Feinstein International Center, Medford, MA https://wikis.uit.tufts.edu/confluence/display/FIC/Impact+Assessment+of+the+Gokwe+Integrated+Re covery+Action+Project Burns, J and Suji, O (2007), Impact Assessment of the Zimbabwe Dams, and Gardens Project. Feinstein International Center, Medford https://wikis.uit.tufts.edu/confluence/download/attachments/14553652/Burns-Impact+Assessment+of+the+Zimbabwe+Dams+and+Gardens+Project.pdf?version=1 Catley, A., Burns, J., Abebe, C,. Suji, O. (2008) Participatory Impact Assessment, A Guide for Practitioners. Feinstein International Center, Medford, MA

32

Catley, A. (1999). Monitoring and Impact Assessment of Community-based Animal Health Projects in Southern Sudan: Towards participatory approaches and methods. A report for Vtrinaires sans frontires Belgium and Vtrinaires sans frontires Switzerland. Vetwork UK, Musselburgh. http://www.participatoryepidemiology.info/Southern%20Sudan%20Impact%20Assessment.pdf
Chambers R. (2007) Who Counts? The Quiet Revolution of Participation and Numbers Working Paper No. 296, Brighton: Institute of Development Studies

Checchi, F. and L. Roberts (2005) Interpreting and Using Mortality Data in Humanitarian Emergencies: A Primer for Non-Epidemiologists Humanitarian Practice Network Paper 52, London: Overseas Development Institute Darcy, J (2005) Acts of Faith? Thoughts on the effectiveness of humanitarian action: A discussion paper prepared for the Social Science Research Council seminar series The Transformation of Humanitarian Action, New York 12th April 2005. Duffield, M. (2001) Global Governance and the New Wars: The merging of development and security, London: Zed Books Feinstein International Center, (2006) Tufts University, Participatory Impact Assessment Training Workshop, Aide Memoir, Addis Ababa, September 2006. Global Center for Development (2006) When Will We Ever Learn? Improving Lives through Impact Evaluation Report of the Evaluation Gap Working Group, Washington, D.C.: Center for Global Development Hewett, A. (2006) Being Risky and Trustworthy: Dilemmas Facing a Non-Government Development Agency Speech by Andrew Hewett, Executive Director, Oxfam Australia, to the Deakin University Community Development in a Global Risk Society Conference, Melbourne, 20 April 2006 Hofmann, C-A, L. Roberts, J. Shoham and P. Harvey (2004) Measuring the Impact of Humanitarian Aid: A Review of Current Practice Humanitarian Policy Group Research Report No. 17, London: Overseas Development Institute Hopkins, C. (2004). Participatory Impact Assessment and Evaluation of the Pastoral Assistance Program (PAP III), Gedo Region, Somalia. VSF Suisse, Nairobi. Hopkins, C. (2003) Community participatory evaluation of the OLS livestock programme, a community based animal health project implemented by VSF Belgium, Bahr el Ghazal, South Sudan. VSF Belgium Nairobi. Hopkins, C. (2002) Participatory Impact Assessment: Emergency Veterinary Relief Program, Gedo Region, Somalia VSF Suisse Hopkins, C. and A. Short (2002) Participatory Impact Assessment in Ethiopia: Linking policy reform to field experiences PLA Notes 45: 23-28. London: Overseas Development Institute

33

Irvine, R., Chambers., R and Eyben, R (2004) Learning From poor Peoples Experience; Immersions Brighton: Institute of Development Studies, University of Sussex Kent, R. (2004) Humanitarian Futures: practical policy perspectives Humanitarian Policy Group Network paper No. 46, London: Overseas Development Institute Mayoux, L, and Chambers, R. (2005) Policy Arena; Reversing the Paradigm: Quantification, Participatory Methods and Pro-Poor Impact Assessment: Journal of International Development; 17, 271-298: John Wiley & Sons, Ltd 2005 Maxwell, D. and Burns, J. (2008) Targeting in Complex Emergencies: South Sudan Country Case Study, Feinstein International Center, Medford, MA 2008 Nalitolela, S. and R. Allport (2002) A Participatory Approach to Assessing the Impact of a Community-based Animal Health Project with Maasai Communities in Tanzania PLA Notes 45: 1722. London: Overseas Development Institute Pasteur, K. (2004) Learning for Development: A literature review. Lessons for Change in Policy and Organisations N6. Brighton:Institute of Development Studies. Rifkin, S.B., Muller, F. and Bichmann, W. (1988). Primary healthcare: on measuring participation. Social Science and Medicine 26(9), 931-940 Roberts, L. (2004) Assessing the Impact of Humanitarian Assistance: A Review of Methods and Practices in the Health Sector HPG Background Paper, London: Overseas Development Institute Roche, C. (1999) Impact Assessment for Development Agencies: Learning to Value Change Oxford: OXFAM, Novib Shoham, J. (2004) Assessing the Impact of Humanitarian Assistance: A Review of Methods in the Food and Nutrition Sector Background Paper for HPG Research Report No 17, London: Overseas Development Institute Smillie, I and Minear, L. (2004) The Charity of Nations: Humanitarian Action in a Calculating World: Kumarian Press, Bloomfield, CT. Van Brabant, K. (1997) Organisational and Institutional Learning in the Humanitarian Sector: Opening the Dialogue A Discussion Paper for the Active Learning Network on Accountability and Performance in Humanitarian Assistance. London: Overseas Development Institute Walker, P. And Maxwell, D (2008) Global Institutions; Shaping the Humanitarian World: Routledge, December, 2008 Watson, C. (2008). Impact Assessment of Humanitarian Response: A Review of the Literature; Feinstein International Center, Medford. Young, H. and S. Jaspars (2006) The Meaning and Measurement of Acute Malnutrition in Emergencies: A Primer for Decision Makers Humanitarian Network Paper 56, London: Overseas Development Institute
34