Questionnaire Testing and Interview Techniques: Survey Cycle

Second Workshop for Improving Disability Statistics and Measurement
Bangkok, 27-29 September 2004
Questionnaire Testing and Interview

Techniques
Survey Cycle
Client Liaison
Evaluation Planning
Question and
instrument
Survey development and
Dissemination testing
Development
Survey Sample Design

Analysis
Estimation Data Collection
Processing
1
Minimising measurement error

zRemember
ƒTotal survey error is the sum of sampling error
and non-sampling error

ƒSampling error is minimised through increasing
sample size
ƒNon-sampling error is bias and is minimised
through attention to survey procedures, such as,

development, questionnaire, sample design,
interviewing and processing
zAim of testing questionnaires is to minimise
measurement error
Sources on non-sample error
2
Questionnaire Testing
zTwo broad categories used
ƒPre-Testing
ƒField Testing
Pre-test properties
zOccurs:
ƒBefore field tests
ƒSmall, targeted sample
ƒControlled environment
ƒQualitative analysis
ƒFocusses on:
the interviewers administration of the questions,

and
the respondents ability to answer questions
without error
3
Aims of pre-testing
zIdentify sources of potential non-sample
error:
the effect the interviewer has on question
responses;
error the respondent makes due to the
questionnaire;
errors due to the method of delivery or errors
caused by sequencing; and
errors caused by the interaction of these
factors.
Benefits of pre-testing
zReduce survey development time
zReduce survey costs
zImprove the quality of data collected
zIdeally all survey development programs

should incorporate some degree of pre-
testing
zPre-testing can minimise non-sampling error

but cannot eliminate all errors in survey data
4
Types of pre-testing techniques

zLiterature reviews
ƒdatabase and library searches
ƒother statistical agencies and organisations
zExpert reviews
zFocus groups
zCognitive testing
zInterviewer debriefing
zObservational interviews
zBehavior coding
Sampling for pre-tests

zParticipants recruited purposively using
selection criteria specific to the questions to
be tested
z"Questions that fail in the laboratory will

certainly fail in the field but questions that
work in the laboratory may not always work
in the field"
5
Expert reviews: overview

zGroup of "experts" review questionnaire to
identify potential problems and gather an
understanding of the respondent's task
zList potential problems and classify them to
specific points where difficulties may occur
zWho should conduct an expert review?
ƒAt the ABS, experts means survey design
experts, not subject matter experts, but can
include interviewers
Focus group: overview

zModerator led, informal small group
discussion
zModerator remains neutral at all times
zSeveral groups may be necessary to
represent different population subgroups
zUsed to explore respondent understanding
of, and reaction to terms, concepts and
issues
6
Cognitive interviews: overview

zCognitive interviews are essentially:
ƒ Asking respondents questions about the
survey questions
zCognitive interviews identify problems in

ƒquestion comprehension
ƒmemory recall
ƒselecting responses
ƒinterpretation of reference periods
ƒreactions to sensitive questions
Cognitive interviews
zConducted one to one with an interviewer in
a standardized environment, usually taped
zCognitive interviews can use

ƒprobing questions
ƒthink alouds
ƒparaphrasing
ƒvignettes
ƒcard sorting tasks
7
zRetrospective probes
ƒInterviewer administers the survey in totality and
then asks specific probes about particular questions.

ƒUsed to provide information about the survey as a
whole eg, flow and context effects and mode

effects.
zConcurrent probes
ƒRespondents are either asked to verbalise their
thought processes as they answer or they are asked

additional questions straight after a survey
questions.
ƒUsed to understand detailed response processes.
zThink-alouds
ƒThe interviewer asks the respondent to describe
their thought process aloud.

ƒUsed in either concurrent or retrospective
probing techniques.
EG: "Please tell me what you are thinking
about as you answer the next question."
8
zParaphrasing
ƒThe respondent is asked the question and then is
immediately asked to repeat the question back

in their own words (before they have a chance
to answer the question itself)

zVignettes
ƒParticipant responds to a question from the
point of view of a hypothetical situation.

ƒThis allows interviewers to explore response
processes, especially information about storage

and retrieval of information, regardless of
whether the respondent has experienced the
situation being described, or not.
9

zCard sorting tasks
ƒProvide interviewers with information about
how respondents think about categories, group

information or define particular concepts.
ƒHighlights the ways in which respondents
define concepts (in terms of inclusions and

exclusions).
Is cognitive interviewing the

best method?
zLeast reliable method for indicating problems
with a questionnaire (Presser & Blair, 1994)
BUT
zThe most detailed pre-testing method for

ensuring respondents are answering questions
as intended
zTechnique that can identify the most detailed
problems with comprehension, retrieval and
judgment processes of the respondent.
10
Analyzing the data

zWhat attributes of cognitive interviews
make results difficult to generalise?
ƒrespondents are self selecting
ƒpurposive sampling
ƒsmall sample size
ƒnon standardised interview creates between
interviewer variation
ƒlaboratory environment is different to normal
interview environment
Observational interviews
zA trained interviewer observes the survey
process
ƒ eg: form completion, responses during an interview
zCan follow-up with in-depth interview
zIdentifies where respondents seem to stumble
or have trouble
zIdentifies problems in wording, question order,
presentation or layout
zUsually used in testing Self Administered
Questionnaires such as Census forms
11
Behavior coding
zSystematic coding of respondent and
interviewer behaviors during the interview
zEvaluates the question-asking process and
the administration of the survey by finding
problems in question wording or
understanding
zUses a third party "coder" and a behavior
coding sheet
zCan be done concurrently or retrospectively
Selecting a pre-testing strategy

zNeed right combination of techniques to
achieve the objectives of testing, within
available resources
zFactors to consider
ƒCost
ƒSkilled resources
ƒTimeliness of results
ƒStage of development process
ƒAims of test
ƒTypes of errors to be identified
12
Selected attributes of pre-

testing techniques
Technique Resources Timeliness of Stage of
results development
process
Cognitive High Fast Middle
Interviews
Focus Groups Medium Fast Start/early
Interviewer Medium Fast Start/early
Debriefing
Expert Review Low Fast Early
Literature Low Moderate Start
Review
Behaviour High Slow Later
Coding
Observational High Moderate Middle
Interviews
Comparison of pre-testing techniques

by sources of error able to be examined
Sources of
error
Focus Interviewer Cognitive
Groups Debriefing Interviews
Interviewer Consistency detailed
Sensitivity detailed
Respondent Comprehens detailed detailed
ion
Recall detailed detailed
Judgement detailed detailed
Response detailed detailed
Burden detailed limited detailed
Sensitivity detailed some detailed
13
Field Testing
zTwo broad types
ƒPilot Tests
ƒDress Rehearsals
zEvaluation
ƒconducted after each test
ƒconducted after final survey
ƒbased on qualitative and quantitative data
ƒused as a basis for next development cycle
Field Testing (continued)

zSmall operational tests conducted in the
same mode as the final survey
zABS uses:
ƒSample size of about 300 households
ƒSample usually selected randomly
ƒ8 to 12 interviewers involved in tests
14
Pilot Testing
zPilot testing can provide information on:
ƒidentification of populations
ƒunderstanding of questions (broad)
ƒquestion sequencing and overall survey flow
ƒinterviewer instructions
ƒinterviewer training manuals and procedures
ƒrespondent reaction to questions
ƒediting procedures
ƒprocessing systems (if pilot test data processed)
Dress Rehearsal
zDress rehearsal is the final test and can
provide information on:
ƒall of the items covered in the pilot test
ƒinterview times (important for survey cost)
ƒfield procedure
15
Changes after tests

zChanges to instruments and procedures can
be made after pilot tests and before the dress
rehearsal
zThe dress rehearsal should be a test of all

elements of the survey in the fashion they
will be conducted in the final survey. There
is only very limited scope to change things
after the DR.
Interviewer debriefing
zDebriefings are:
ƒSubjective and anecdotal
zSurvey interviewers provide feedback on a

previously used survey instrument or draft
questionnaire
zInterviewers are given questionnaires or
participate in focus group style interviews
zAsked to report on any problems they had
administering the questionnaire, using the
assisting documentation, or any particular
reactions or problems respondents had.
16
Evaluation
zPost survey review

ƒReview all aspects of the survey
ƒUse interviewer feedback on issues relating to the
questions, response categories, procedures etc.

ƒSurvey processing staff feedback on issues of
processing data items, validation etc.

ƒKey user feedback on whether survey data met
their information needs. Identify:

Gaps
Deficiencies in collected items
zData quality analyses
ƒNon-response evaluation
Testing reliability/validity of
questions
zOne way to evaluate the validity and reliability of
survey questions is to conduct a follow-up study of
the same respondents and compare survey results
with "gold standard" statistics.
zThis method has been used extensively in testing
Australian census questions on disability to:
ƒ Compare different census test questions
ƒ Conclude that general disability census measure was
not identifying the same population as the survey

measure
zResults very promising for the need for assistance
questions being tested for 2006
17
Gold standard vs test measure

statistic
z Compare the results of the gold standard and
the test measures (as per the following table)
Test measure
True status + - Total
(Has a disability) (Does not have a
disability)
D a b a+b
(Has a
disability)
not D c d c+d
(Does not
have a
disability)
Total a+c b+d a+b+c+d
Sensitivity and Specificity

zSensitivity is the probability of correctly
identifying someone with a disability (a/(a+b))
zSpecificity is the probability of correctly
identifying someone that does not have a
disability (d/(c+d))
zFalse positive rate is the probability of
identifying someone as having a disability
when they do not (c/(c+d))
zFalse negative rate is the probability of
identifying someone with a disability as not
having a disability (b/(a+b))
18
Interviewer training
zABS disability survey training sessions were
three days for all interviewers
zInterviewers already well trained in general
interviewing procedures, including computer
assisted interviewing
Interviewer training
(continued)
zTraining covered:
ƒmajor users and uses
ƒstructure of the questionnaire
ƒmain survey populations
ƒquestions and sequencing
ƒcoding of conditions
ƒpractice interviewing
ƒissues relating to interviewing people with
disabilities
19
Interviewing people with

disabilities
zUnderstand that there are myths and stereotypes
about people with disabilities, but many of them
are not true!!
zAlways interview in a professional and sensitive
manner
zLanguage is important
ƒNeeds to be acceptable to people with disabilities,
for example, neutral and avoiding terms like
disability, handicap, impairment etc.
ƒNeeds to be accurate and respectful of individuals
with disabilities
Examples
Interviewing people with

disabilities (continued)
zAsk all questions
zAsk questions exactly as worded
zDo not assume or anticipate answers
zBe patient and let the respondent finish
zAsk all questions in a positive manner
zMaintain a professional attitude
20
Cultural issues
zThis is a very difficult area
zBeing considered by the WCG on disability
statistics
zSometimes there is no overall word for
"disability"
zReluctance in some cultures to identify people in
their household with a disability
zEncourage identification of people with
disabilities through use of techniques to explain
the purpose of the survey to respondents, such as
primary approach letters and brochures
21

Questionnaire Testing and Interview Techniques: Survey Cycle

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Questionnaire Testing and Interview Techniques: Survey Cycle

Uploaded by

Copyright:

Available Formats

Second Workshop for Improving Disability Statistics and Measurement

Bangkok, 27-29 September 2004

Questionnaire Testing and Interview

Survey Sample Design

Estimation Data Collection

Minimising measurement error

and non-sampling error

through attention to survey procedures, such as,

Sources on non-sample error

ƒSmall, targeted sample

the interviewers administration of the questions,

zIdeally all survey development programs

zPre-testing can minimise non-sampling error

Types of pre-testing techniques

ƒother statistical agencies and organisations

Sampling for pre-tests

z"Questions that fail in the laboratory will

Expert reviews: overview

Focus group: overview

Cognitive interviews: overview

zCognitive interviews identify problems in

ƒinterpretation of reference periods

ƒreactions to sensitive questions

zCognitive interviews can use

ƒcard sorting tasks

then asks specific probes about particular questions.

whole eg, flow and context effects and mode

thought processes as they answer or they are asked

their thought process aloud.

immediately asked to repeat the question back

Cognitive interviews: overview

point of view of a hypothetical situation.

processes, especially information about storage

Cognitive interviews: overview

how respondents think about categories, group

define concepts (in terms of inclusions and

Is cognitive interviewing the

zThe most detailed pre-testing method for

Analyzing the data

ƒsmall sample size

ƒnon standardised interview creates between

Selecting a pre-testing strategy

ƒStage of development process

ƒTypes of errors to be identified

Selected attributes of pre-

Comparison of pre-testing techniques

ƒconducted after final survey

ƒbased on qualitative and quantitative data

ƒused as a basis for next development cycle

Field Testing (continued)

ƒSample usually selected randomly

ƒ8 to 12 interviewers involved in tests

ƒunderstanding of questions (broad)

ƒquestion sequencing and overall survey flow

ƒinterviewer training manuals and procedures

ƒrespondent reaction to questions

ƒprocessing systems (if pilot test data processed)

Changes after tests

zThe dress rehearsal should be a test of all

zSurvey interviewers provide feedback on a

zPost survey review

ƒUse interviewer feedback on issues relating to the

the interviewers administration of the questions,