Natalie Chun Impact Evaluation Workshop 11 July 2012

Project: $20.1 million loan. Establish/strengthen market linkages between farmers and buyers for high value crops (HVCs). Cooperatives/groups submit grant proposal of business plan related to HVCs. Grants reviewed and selected by committee. Infrastructure built. Target Population: Small and medium sized farmers that are part of savings cooperatives (or farmer groups), plus excluded populations Location: Mid-West and Far-West Nepal Implementation Period: 2012-2017
Raising Incomes for Small and Medium Farmers in Nepal
2

Implementation Criteria/Plan Still Unclear

Selection criteria for treated (grant) population not defined Unable to obtain agreement on how implementation will occur in different districts or if randomization of cooperatives, districts/sub-districts is possible.

Large variations in amount of money that could be granted ranging from $1500$250,000.

Raising Incomes for Small and Medium Farmers in Nepal

Deciding what to measure

Budgetary Constraints for Evaluation

~$150K currently allocated

Intention to treat effects where compare areas that have the opportunity to vie for grant applications to areas that do not have the opportunity. Average treatment effects on treated where compare people in groups who actually receive the grant to similar groups who did not have the opportunity to receive the grant. Localized average treatment effects using (fuzzy) regression discontinuity design. Where grant applicants just above the approval threshold are compared to grant applicants just below the approval threshold.

Raising Incomes for Small and Medium Farmers in Nepal

Survey only 4 districts, but more intensively (rather than10 districts).

2 districts in Far West and 2 districts in Mid-west (one terai and one hill district for each region) Sample size: 2400 farmers (20 per cooperative/group = 30 groups per district). Randomly select VDCs, obtain census of farmers within VDCs, randomly select farmer groups from census. Ensure selection of VDCs is roughly representative of land holdings within the 4 districts, crop production, incomes.

Raising Incomes for Small and Medium Farmers in Nepal

Survey questionnaires:

Timing of Surveys
Baseline Mid-term Endline (?)

Household Cooperative or farmer group Village

Try and obtain process measures Expected costs (baseline only)

Capture quality of treatment Timing of treatment

Survey firm: ~$70K for baseline, analysis, report. Impact evaluation manager: ~$15K for baseline questionnaire design, sampling design, oversight
Raising Incomes for Small and Medium Farmers in Nepal

Obtain Agreement on Approach to Implementation

First best: (i) phase-in approach to districts/subdistricts (ii) random assignment of grant applicants who obtain grant first versus second (iii) survey applicant population and targeted population. Should be able to identify ITT, ATE, ATT. Worse case: (i) selection criteria defined and no phasein or random assignment (ii) Random sampling of targeted population (iii) More extensive questionnaire to capture selection criteria of treatment (iv) Find some instrument to identify LATE or hope to be able to construct match off of sample population OR use propensity score matching to obtain matches between treatment and control based on baseline data.

Raising Incomes for Small and Medium Farmers in Nepal

First best:
(i) Phase-in approach to districts/sub-districts (ii) All applications occur at once and random assignment of grant applicants who obtain grant first versus second (iii) survey applicant population and targeted population. Should be able to identify ITT, ATE, ATT. Risk: May not be able to obtain adequate number of grant applications in initial phase to have over subscription.

Raising Incomes for Small and Medium Farmers in Nepal

Worse case:
(i) selection criteria defined and no phase-in or random assignment (ii) Random sampling of targeted population (iii) Develop more extensive questionnaire to capture selection criteria of treatment (iv) Find some instrument to identify LATE or hope to be able to construct match off of sample population OR use propensity score matching to obtain matches between treatment and control based on baseline data.

Raising Incomes for Small and Medium Farmers in Nepal