Professional Documents
Culture Documents
Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006
Overview
What is a baseline? Why you should care The phases of conducting a baseline
The
What is a baseline?
Fixing the time at the base a benchmark from which you measure progress Snapshot of indicators at a time Instrument used to:
Test
To identify whether there were any benefits for the investments made
Were
objectives met? What factors explain the result? How can the program be improved?
Compare alternative models to get the biggest bang for your buck To inform next generation projects Evidence-based policy making demonstration effect for government
Design Implementation actual survey Data entry and analysis Report writing
P1: Check-list
Clear objectives (what is the problem?) Clear idea of how you will achieve the objectives (causal chain or hypotheses) Clear and measurable indicators
Clear
Design survey instrument keep it simple and related to the objectives and hypotheses you want to test Link surveys to GIS use consistent units Select controls/counterfactuals to attribute change (causality) Timing of baseline
Before project (what if project never materializes)? 2 years into the project (intervention has begun)? Other factors (seasonality)
Sampling strategy
Population Size
10 50 100 500 1,000 3,000 50,000 1,00,000
Sample Size
10 44 80 217 278 341 381 385
Over sample for attrition Design the database system for data entry Translate the questionnaire
The objectives and importance of the study How the sections are linked and what the questions mean
Ask yourself, can you answer these questions? Are they relevant to the outcomes? Will they be understood? Field test: To test both how the surveyor administers the instrument AND how the respondent understands the question.
Phase 2: Implementation
Were a city person who didnt speak the local language very well Had to travel to several villages and spend hours asking people questions that have no relevance to you. Were paid a small sum per questionnaire Not monitored by supervisors This is not your full time job
P3: Check-list
Data Entry
Make the data entry system as fool proof as possible - has unique identifiers to link both household, village and GIS data Ensure database allows for merging of data Do not change/erase data on questionnaires Raw data should always be input as is, changes can then be made in the database software (programatically) with documentation
Name variables corresponding to the question and section in the questionnaire include a dictionary Code descriptive answers (to facilitate analysis) All fields should be filled (NA or NR) Units should be uniform by district Totals calculated by formula not from summary column
Consistency checks check for missing entries, wrong entries, sample statistics, patterns (queries should be inbuilt) Validity checks similar questions in different places on the questionnaire (RCH example)
No analysis!
Over generalizing the results Mis-reporting statistics Using % when the numbers are small Attributing causality when it is not demonstrated
Simple, Clear and Relevant State limitations (attribution, causality) Major findings should be upfront Focus on quality rather than quantity Technical details in an appendix Should always
include the questionnaire in the appendix ask for electronic copy of data Request copies of filled out surveys
Essential if you change consultants at midterm or want to conduct internal analysis to compare modes of delivery (data lost example).
Phase 1: Design
objectives and hypotheses know what you want to test Identify a person in your unit who will manage this process Write a good TOR, remember the baseline determines the quality of your panel You can add questions as project evolves but cannot change questionnaire threat to internal validity Identify consultants
Clear
Procurement focus on quality not the cheapest bid if you throw peanuts youll attract monkeys Ideally you should have a black-list of organizations
Phase 2: Implementation
Organize
an impact evaluation workshop if necessary Randomly verify questionnaires to reduce the likelihood of false responses (no filling it in a bar) Pay reasonable wages to surveyors (if possible) Show the client and firm that you care
entry (2 separate organizations and verify. Payment based on quality of data entry) Select 15 questionnaires at random and check data entry person in your unit managing Check data quality (consistency and validity checks) Hold an IE workshop to build data analysis capacity (if necessary)
on an outline beforehand Dedicate a chapter on indicators you are tracking Focus on quality not quantity Think Big Picture
5.
6. 7. 8. 9. 10. 11.
Background: Project objectives and components Survey design: Consult a sampling expert!!! Survey instruments Guidance on survey implementation Data processing and analysis Staffing Duration and time schedule Submission of reports and datasets Support to the firm Budget & Payment Schedule Annexes: Draft questionnaires, Results Framework
Recommend the methodology for sampling Calculate the optimal sample size Develop the sample frame and select the sample The final sample and details of the statistical methodology used to select the sample need to be cleared by the project Construct the sample weights and provide documentation on the methodology used to construct the weights
Design or refinement and adaptation of the data collection instruments Specify levels of data collection Length of questionnaires Prepare all support documentation including coding guides, interviewer and supervisor manuals and the data entry manual Translation and back-translation Skip patterns, coding open ended questions
Implementation plan Selection and training of field workers: specify minimum duration of training Pilot testing should be explicitly specified in ToR Responsibility for all field operations, including logistical arrangements for data collection and obtaining household consent lies with Consultants. Ask for field-work progress reports (biweekly/monthly)
Staffing
Sampling expert/statistician Technical specialists as relevant Economist Sociologist. Core survey staff: the survey manager, the field manager, the data manager Enumerators, supervisors and data entry staff
Consult the expertssurvey specialist and sampling specialist and develop the ToR in consultation. Selection committee should include a survey expert and social scientists in addition to technical experts. You can never over-supervise!!! Hire third-party supervision consultant if needed. Question the data and the findings.