Interviews, Surveys, and the Problem of Ecological Validity

Author(s): Aaron V. Cicourel

Source: The American Sociologist, Vol. 17, No. 1 (Feb., 1982), pp. 11-20
Published by: American Sociological Association
Stable URL: http://www.jstor.org/stable/27702491 .
Accessed: 28/10/2013 15:50
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
.
American Sociological Association is collaborating with JSTOR to digitize, preserve and extend access to The
American Sociologist.
http://www.jstor.org
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
The Problem of Ecological Validity 11
tions that a classic can serve. We
may
believe that students' minds are
expanded
by reading
Durkheim without our
having
to believe Durkheim has
many
true
gener
alizations about the causes of suicide.
George
Herbert Mead can
symbolize
what
is distinctive in
symbolic
interactionism
even if we cannot
quite figure
out how to
test the
hypothesis
of the
independence
of
the "F from the
"me,"
and to turn it into a
puzzle
for routine science. And orte can
enjoy
the taste of Marx's famous
passage
in The 18th Brumaire about French
peas
ants
forming
a vast
mass, without that
beauty being
undermined when we find
some
regions
of modern France where the
peasants
vote Communist.
What is destructive about admiration of
the
classics, then,
is the halo
effect,
the
belief that because a book or
article is
useful for one
purpose,
it must have all the
virtues.
INTERVIEWS, SURVEYS,
AND THE PROBLEM OF
ECOLOGICAL VALIDITY*
Aaron V. Cicourel
University of California,
San
Diego
The American
Sociologist 1982,
Vol. 17
(February):
11-20
Despite
the
fact
that
virtually
all social science data are derived
from
some kind
of
discourse or
textual
materials,
sociologists
have devoted little time to
establishing explicit
theoretical
foundations for
the use
of
such instruments as interviews and
surveys.
A
key problem always
has been the lack
of
clear theoretical
concepts
about the
interpretation of
interview and
survey
question
and answer
frames.
We lack a
theory of comprehension
and communication that can
provide
a
foundation for
the
way
that
question-answer systems function,
and the
way
respondents
understand them. The
paper briefly
describes the
possible
relevance
of linguistic
and
cognitive processes
for improving
our
understanding of
interviews and
surveys.
The
theoretical
foundations of
interviews and
surveys
also must address the
way
that
artificial
circumstances become
necessary
to
guarantee adequate study designs.
These
artificial
circumstances
often
violate
ecological validity,
or the
way
interviews and
survey questions
are
constructed, understood,
and
answered,
as contrasted with the
way
that
field
notes and
tape-recordings of
natural
settings
are used to address the same or
comparable
substantive and
theoretical issues.
Social scientists have relied on inter
views for a
long
time. There is little reason
to doubt their value and
utility
for
many
theoretical and
practical purposes.
There
exists a
huge
literature on the virtues and
drawbacks of interviews that use
open
ended
questions
and
surveys
that use
close-ended
questions.
Yet there is little in
the
way
of
theory
that would link inter
*
Presented at the thematic section "Fact or Ar
tifact: Are
Surveys
Worth
Anything?"
held at the
1980 American
Sociological
Association
Meetings,
New
York,
August
27,
1980. The other
speaker
was
Howard
Schuman, taking
a less critical view of sur
vey
research. I am
grateful
to Michael
Cole, Roy
D'Andrade,
and
Hugh
Mehan for their valuable re
marks and
suggestions
on a much
longer
first draft of
the
paper. [Address correspondence
to: Aaron V.
Cicourel; Department
of
Sociology; University
of
California,
San
Diego;
La Jolla CA
92037.]
views and
surveys
to more
general
issues
of communication and
comprehension.
Those researchers who are convinced that
interviews and
surveys
are basic research
tools for the
sociologists
are concerned
about
improvements
in
survey design
and
use,
but see little
point
in
challenging
their
routine use. In this
paper
I want to
suggest
a few
cognitive
and
linguistic
issues that
can
clarify
our
understanding
of the
pro
cesses and mechanisms
underlying
the use
of interviews and
surveys.
I also want to
suggest
some theoretical ideas that can
strengthen
the
ecological validity
of inter
view and
survey
methods and
findings.
The
necessity
of
writing
a brief
paper
does not
permit
me to discuss old issues
about current interview and
survey prac
tices that I
hope
are obvious to
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
12 The American Sociologist
sociologists:
for
example,
the
way
that
preliminary qualitative
interviews
nor
mally precede
the construction of inter
view schedules and
surveys;
and the
way
that
pre-testing
with
in-depth
interviews
helps
to create
questions
that
respondents
can
understand,
while
helping
to create
answer
categories
that reflect the
thinking
of
respondents
and not
simply
the re
searcher. Hence I will avoid
discussing
the
range
of
practices
that are
necessary
to insure
quality control,
such as
using
different
types
of
questions
on the same
topic, exploring
the
significance
of
changes
in
wording,
and other
procedures
too numerous to mention here.
Interviews and
surveys usually
occur
within a broader context of interaction
that includes
complex cognitive
and lin
guistic
activities within
a set of in
stitutionalized and
emergent
socio
cultural constraints. The
questions
used in
surveys
almost
always
are framed in a
textual format that
displays
features in
common with
I.Q., aptitude,
and
reading
tests.
Virtually
all social science data are de
rived from some kind of discourse
or tex
tual materials such
as
reports,
written
ac
counts of
observations, interviews,
audio
or video
recordings
of natural
settings,
historical
or
contemporary
documents,
minutes of
meetings, newspapers, maga
zines,
and the like.
Questionnaires
mailed
to
respondents presuppose something
about the
way people
are able to
analyze
textual materials. For
example,
in re
sponding
to
questions
in a
reading
test,
the
respondent
must utilize several
sources of
knowledge
that the researcher used in
interviews and
surveys, therefore, pre
sumes a
theory
of communication and
comprehension
that seldom is addressed
by sociologists.
The remainder of the
paper
will
suggest aspects
of communica
tion and
comprehension
that can
help
es
tablish some theoretical foundations for
interviews and
survey
research.
Aspects of memory
and
comprehension
presupposed
in
surveys
and interviews
Recent work on
learning
and
reading
comprehension (Bransford
et
al., n.d.)
reminds us that our research instruments
stringently
control the information
re
sources available to the
subject
or
respon
dent. In tests and
questionnaires
"normal
procedures"
presuppose
an
agreement
or
social contract between researcher and
respondent:
the contract does not
permit
the
use of other
persons (nor
the inter
viewer)
in order to decide
on
the
meaning
of the
question
and the
appropriateness
of
a
response.
Normal
group
or
peer
sources
of
help
are blocked. The test or
question
naire item is assumed to be self
explanatory
or self-contained. These
con
ditions contrast with the
possibility
of
consulting
a friend or
colleague
or return
ing
to a textual
report,
newspaper article,
book,
and the
like, during
or after an ini
tial
reading
of the text.
The interview and
survey
seek to re
strict the
question
frame,
and in the case
of
surveys,
the answer frame. The
goal
is
to restrict the
question
in such a
way
as to
anticipate
and
even
designate (in surveys)
the
range
of
responses
that
can be used.
The
aggregation
of
responses requires
a
few choices that either
are
formally part
of
the
questionnaire
or are
imposed
on
open-ended
responses.
We
impose
infor
mation
processing
restrictions
on the re
spondent
because
they
enable
us to
aggre
gate
and summarize
a
large
amount of in
formation in a
fairly
succinct
way.
But we
pay
a
price,
and
we need to understand
the costs in order to
improve
the
reliability
and
validity
of interview and
survey
data.
We need
a better
understanding
of the
role of
memory
and the
way questions
are
comprehended.
Norman
(1973)
has noted
several
relationships
between the
organi
zation of
memory
and answers to
ques
tions: a
question
may
not evoke
an
appro
priate
recall if it is
phrased differently
from the
storage
format. Norman calls this
the
"paraphrase problem"
because the
"best" answer to a
question
may
be the
use of another
question by
the
respondent
in order to
clarify
what is intended
by
the
original question.
Norman's reference to
memory brings
up
the
problem
of how
people
store information and combine
general
and
specific
sources of informa
tion in order to reveal what
they
think is
addressed
by
the
question they
are asked.
Norman is interested in the
pre-processing
that occurs before an answer to a
question
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
The Problem of Ecological Validity 13
can be
produced.
In the case of
surveys
we are faced with the
paradox
that the
respondents
are not
encouraged
to
pro
vide us with reasons or
explanations
about
their
answers,
yet
such information
gives
us clues about how the
question
was un
derstood.
Norman
(1973)
states that retrieval of
information from
memory requires
con
struction
by
the
respondent
because of the
paraphrase problem.
It is
difficult,
there
fore,
to show the
way
questions
and an
swers are articulated
by respondents
be
cause no
simple algorithm
can be iden
tified that would enable us to
specify
a
sequence
of instructions or
steps
or ac
tions that
directly
connects
questions
and
answers. We
need, therefore,
an under
standing
of the
comprehension process
and the construction of
responses
in inter
views and
surveys.
Aspects of language presupposed
in
surveys
and interviews
I want to
suggest
a
parallel
between
as
pects
of modern
linguistic theory
and sur
veys. Language
can refer to a lexicon and
individual words as a set of
carefully
con
structed ideals that can be studied and
described
independently
of actual lan
guage
use in social
settings.
The
linguist's
syntax-based theory
of
language provides
the ideal structures the
survey
researcher
needs for
constructing
standardized
ques
tionnaires whose forced-choice
responses
can be
analyzed independently
of the
way
persons
in
daily
life
actually
discuss or
pose
and answer
questions
of each other
within the constraints of
daily practices
in
socio-cultural
organizations.
The
linguist's
normative
theory
of lan
guage
describes idealized
prescriptive
and
proscriptive
rules that are
recipes
for de
ciding
what are
socially acceptable
or un
acceptable
sentences.
Survey questions
are
equally
normative because
they
occur
in
highly
restrictive
settings
that have little
or
nothing
to do with actual discussions or
practices
in
group
or informal
organi
zational activities. But the
linguist's
and
survey
researcher's idealized
language
structures and substantive
questions
about the world of
opinions, attitudes,
be
liefs,
and moral
judgments
are an
integral
part
of the
way
researchers and the
public
governmental agencies conceptualize
and
interpret
the world around them.
The
paradox
we face is that our
surveys
and interviews
only indirectly
reflect as
pects
of the
daily
life
settings
of those we
interrogate, yet
these instruments can be
the basis for the
development
of
policy by
organizations
in
many complex
nation
states.
If we
recognize
that
questions
and an
swers are
speech
acts
(Austin, 1962;
Searle, 1969),
we can make some
sug
gestions
about the
way
that the structure
of
language
can
improve
our understand
ing
of interviews and
surveys. Speech
act
theory
seeks to combine the
analysis
of
the
propositional
content of an utterance
with its
illocutionary force;
the intention
of a
speaker
to act on the world
by
the use
of a
promise, assertion, command,
and the
like.
Speech
acts enable the researcher to
establish
a functional
meaning
for an ut
terance
by
the
way they
are classified as
statements about the
world,
a
speaker's
act on the
world,
or a
symbolic represen
tation of an event in the world. A meta
language
was felt to be
necessary
for dis
cussion of
speech
acts.
The
way
in which
surveys
and inter
views
are conducted
presupposes
a model
of conversational behavior that has been
ignored by
most
sociologists.
The notion
of a
speech
act model has been extended
to the idea of "conversational
postulates"
by
Grice
(1975),
and derived from his
more
general
notion of the
cooperative
principle.
The
principle
refers to a kind of
directive to the
speaker
to formulate all
aspects
of his or her utterance in a
way
that will
permit participants
of a conver
sation to facilitate to the utmost the
achievement of the
explicit
and
tacitly
agreed upon
aims of the conversation.
Grice identifies four
categories
that are
designed
to orient the
speaker
to be as
informative as
possible
but not more in
formative than seems
necessary.
Nor
should the
speaker
say anything
believed
to be false or
anything
that lacks sufficient
evidence. The
speaker
also is to be rele
vant and to be brief and
orderly
while
trying
not to be
ambiguous
or obscure in
his or her use of
expressions.
Notice that
the term
speaker
would
apply
to both the
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
14 The American Sociologist
interviewer and
respondent
in the case of
surveys
and interviews.
Another theoretical issue contained in
Grice's work includes the notion of con
versation
implicature.
In
everyday
con
versation,
listeners are
expected
to make
inferences that do not
necessarily
follow
from the
premises
or statements
given, yet
the statements are
necessary
for the com
prehension
of the discourse or text. The
notion of conversational
implicature
is
central to
surveys
and interviews as sub
sets of discourse and textual activities be
cause our
questions require
that the re
spondent go beyond
the information
given
in
survey questions
and
presume
that the
utterances can be
expanded
in order to
pursue
their
implications
and derive co
herence from what is said.
Speech
act
theory
can
help
us under
stand the
way
that variations in the textual
content and structure of interview and
survey questions guide
the kinds of in
terpretations
that will be
made,
and how
these attributions of
meaning
will influ
ence the construction of a
response
or the
selection of an
option
in a set of forced
choice
responses.
Some
aspects of question-answer systems
Question-answer systems
deal with a
sub-set of
speech
act
theory.
The
ques
tions
employed
are for the most
part
direct
attempts
to elicit information. In
everyday
English
we often use indirect
speech
acts
and our
questions
do not
always
follow an
interrogative
format. Scheduled inter
views and
surveys presume
a
respondent
who is aware of the
general procedure.
A
specific style
of
interrogation
is used and
respondents
assume a
response
stance
that differs
markedly
from routine
every
day
conversation. But this
response
set is
quite
similar to occasions when someone
is
being
interviewed for a
job
or is
taking
a
test.
The formal
aspects
of conversation out
lined
by
Grice
indirectly parallel
some
formal
properties
of
question-answer
systems
described
by
Harrah
(1973).
Whereas Grice
specifies
some
general
conditions
governing
all
conversations,
including aspects
of the
reasoning
neces
sary
for successful
exchanges,
Harrah in
dicates conditions that are
especially
rele
vant to the
questioner.
In Harrah's
(1973)
model the
ques
tioner:
(a)
Is
presumed
to know what the
problem
is
about.
(b)
Knows how to
express
the
question
in an
effective manner.
(c)
Knows what the set of
possible
alterna
tives can be.
(d)
Can claim that one of the alternatives is
true.
(e)
Does not know which alternative he
wants to know.
(f)
Believes the
respondent
can
help
him if a
particular question
is
put properly.
But there are various
logics
of
questions,
notes
Harrah,
and
they
will
vary according
to the social situation in which
questions
are used. Thus in a classroom
setting
a
teacher
puts questions
to students and
knows the answers
expected.
Harrah de
scribes some of the conditions of a Ph.D.
examining
committee where their
ques
tions
may
be directed as much to each
other as to the student. In the Ph.D. ex
amination the
knowledge
base of the re
spondent
is
presumed
to be the
problema
tic issue. When a
physician
asks
questions
of a
patient,
the
knowledge
base of the
respondent
may
be viewed
as a source of
new information. But this information
re
quires particular types
and
sequences
of
questions
and
answers. The
patient's
an
swers also
may
become
problematic
de
pending
on the kinds of attributions made
to the
patient
because of
age,
mental
status,
social
position,
and
physical
con
dition and
appearance,
to mention
a few
key
variables that affect all
question
answer
systems.
Questionnaire
items are not
merely
in
dividual,
self-contained
texts,
but become
the basis for
inferring
macro-structures
that resemble those
reported by
re
searchers
working
on text
comprehension
(Kintsch
and van
Dijk, forthcoming;
van
Dijk, 1972).
The
respondent
seeks a more
comprehensive understanding
of the dif
ferent
questions
asked
despite
the re
searchers
attempts
to randomize the
pre
sentation of
questions
that
are linked
by
hypotheses
in the research
design.
This
search for a
pattern
on the
part
of the
respondent
is
part
of an
attempt
to create
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
The Problem of Ecological Validity 15
an
explanation
that
goes beyond
whatever
cryptic
information is
supplied by
the
interviewer or
questionnaire.
The
respon
dent becomes an active
participant
in the
survey
or interview and seeks to
develop
his or her own
hypotheses
about what is
going
on and what intentions the re
searcher
projects by
the kinds of
ques
tions asked.
Discourse and textual
analysis
as
comprehension
A
key
issue in the
study
of
comprehen
sion is
trying
to estimate what the
respon
dent
brings
to the
reading test, interview,
or
survey.
There are several
strategies
available for
studying comprehension.
In
unpublished
work
by
David
Rumelhart,
comprehension by subjects
is measured
by asking
them to read a
story
line
by
line
while
indicating
what is
happening
after
each line. An
unpublished project
in
prog
ress on
reading comprehension by
Charles
Fillmore and Paul
Kay
uses
reading
test
items
(that strongly
resemble
survey
questions)
as the basis for
interrogating
the child about his or her
understanding
of
the text of each test
question.
More abstract
types
of textual
analysis
include the identification of
topics
or
themes and their
continuity
over a
large
textual domain
(Grimes, 1980).
A
large
and
growing
literature exists here that ad
dresses textual
analysis (Dressier, 1977;
Halliday, 1967).
I will not
attempt
to re
view current work on the
analysis
of texts
but
only
mention the broad
goal
of iden
tifying
those structures that would serve
as a basis for
interpreting specific portions
and
general aspects
of a text. Particular
conventions of
language
use,
such as fol
lowing
certain forms of
language
structure
when
writing
a
letter,
a
report,
in
structions for
filling
out the
necessary pa
pers
for a bank
loan,
and the
like, generate
expectations
about how
topics
are intro
duced, developed,
and terminated. We
need to know how interview and
survey
questions
as texts are
interpreted by
the
researcher and
respondents.
Current work
on the
analysis
of discourse and textual
materials can
help
us
develop
a theoretical
foundation for
understanding
and im
proving
interviews. The theoretical foun
dations of interviews and
surveys
must
include the
way
that artificial circum
stances
necessary
to
guarantee adequate
study designs
can violate the
ecological
validity
of
findings
vis-?-vis what takes
place
in
daily
life
settings.
Restating
the
ecological validity
issue
for
sociology
Social scientists often
are so
pre
occupied
with
creating
an
adequate study
design
that
they
overlook the
ecologi
cal
validity problem:
Do our instruments
capture
the
daily
life
conditions, opinions,
values, attitudes,
and
knowledge
base of
those we
study
as
expressed
in their natu
ral habitat?
Recent work
by
Cole et al.
(1978)
re
views the
history
and current efforts to
deal with the
problem
of
ecological
va
lidity.
Cole et al. refer to the
revolutionary
impact
of Wundt's
laboratory psychology
for the
experimental study
of mind in arti
ficially
constructed and
simplified
envi
ronments. Can we extend the
elegance
and control of
laboratory
research to field
settings?
In
sociology
it is difficult to
study
everyday settings
while
using carefully
formulated
surveys.
Much of
survey
re
search can be viewed as the
application
of
rigorous techniques
to data elicited in
simplified
and artificial social envi
ronments.
In
sociology
and
psychology ecological
validity
remains a minor issue because
studies of the social
organization
of the
laboratory
and the interview or
survey
settings
often are
relegated
to minor roles
when data are
analyzed. Psychologists
have demonstrated renewed interest in
pursuing laboratory-derived problems
in
natural
settings
and
incorporating every
day
tasks into
laboratory settings.
In
sociology
this would mean
contrasting
the
way
interview and
survey questions
are
constructed, understood,
and
answered,
with the
way
that field notes and
tape
recordings
of natural
settings
are used to
address the same or
comparable
substan
tive and theoretical issues.
Psychologists
are sensitive to
problems
associated with task
definition,
mental
overload,
and
possible
differences in the
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
16 The American Sociologist
way
remembering, thinking,
and
attending
to activities occur in
laboratory
and non
laboratory settings.
But
they
do not
study
the cultural definitions of
everyday
life
that are
part
of the
laboratory setting
and
that are invoked
necessarily
when at
tempting
to conduct controlled research in
non-laboratory settings. Psychologists
are
not sensitive to what their
subjects
must
be able to do to make themselves
appear
as normal members of a
group
and the
larger society
in order to
perform
ade
quately
in an
experiment
or in an
every
day setting.
The
subjects
and the
exper
imenter both must
rely
on their
everyday
tacit
knowledge
in order to
satisfy
stated
and unstated social conditions that must
be followed if the research is to be consid
ered successful.
Sociologists
are sensitive to the fact that
many problems
are associated with the
way questionnaires
are
administered,
coded,
and
organized
for
analysis.
But
they
are insensitive to the information
processing problems
associated with these
tasks. Because so
many surveys
are done
in the same culture in which the re
searchers also are
native,
and because we
gradually
have socialized our
respondents
to be
fairly
docile to the demands of sur
veys,
especially
since
everyday
life cir
cumstances often force them to submit to
such
activities,
we have little
knowledge
about the social
practices
of
survey
re
search within field
settings
and within re
search centers where the
analysis
takes
place.
When we administer
surveys
in
other cultures we
incorporate
natives who
have been trained in the same method and
who can
tacitly negotiate
the cultural dif
ferences.
During everyday
interaction the mem
bers of a
group
who
routinely
discuss
political, economic,
and social events are
sensitive to
group
resources of informa
tion and the
interpersonal
constraints that
are
imposed
on
exchanges,
and also are
aware of the
knowledge
limitations of the
members of the
group.
The
ecological
va
lidity
issue addresses the extent to which
responses
to interview and
survey ques
tions reflect or
represent
the
daily
actions
of a
collectivity.
We must
compare
the
way
collective discussions about
topics
covered in interviews and
surveys parallel
or differ from the
way
such themes are
presented
in the formal
setting
created
by
research
goals.
A
partial
examination of the
ecological
validity
issue can be found in recent work
by
Schuman
(1966),
and Schuman and
Presser
(1977; 1977-1978; 1979;
forth
coming)
where
they
show that
changes
in
the
wording
of the
questions
often lead to
changes
in
response patterns.
The work
by
Schuman and Presser also contains
valuable information on differences in the
use of
open
and closed
questions.
The
authors
(forthcoming:9-10)
note that if re
spondents
are
given
a
question
about
which
they
know
nothing, many
will an
swer the
question
if there is no
explicit
"don't know"
option. Many respondents
are
willing
to admit
ignorance. Respon
dents also are said to make an "educated
(though wrong) guess"
about a
topic
de
spite
their
being
uninformed about the
issue. The authors note that these re
sponses
are like "non-attitudes" in the
sense that there
probably
was no
prior
thought
about the attitude before
pre
sented
by
the interviewer.
The studies
by
Schuman and Presser
seek to resolve
possible problems
in inter
viewing
and
surveys
as a
way
of enhanc
ing
their
reliability
and
validity.
These
studies, though
limited in
scope
and
depth,
are valuable contributions to the
minimal work that has been done
by
re
searchers
working
within social science
who
strongly support
these methods in
their
existing
form.
Many
serious
prob
lems of
reliability
and
validity
remain be
cause of normal
practices
that are devoid
of
adequate
concern for theoretical foun
dations. For
example,
we restrict our col
lection of information to a few
categories
in order to restrict the number of
compari
sons we have to make. The
respondent's
knowledge
of the world is not a
problema
tic issue. In a
laboratory study
or in sur
veys
with fixed-choice
questions, subjects
or
respondents bring categorical
mech
anisms with
them,
but the actual
"pack
ages
of information"
they employ
must be
either recoded or tailored to the
particular
conditions of the
experiment
or
survey
question
and forced choices
provided.
The
range
of
speech
acts becomes se
verely
restricted in
survey
research. The
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
The Problem of Ecological Validity 17
respondent's
cultural resources available
for
searching
one's
memory
are con
strained,
and this limits the
respondent's
ability
to make
comparisons.
The
survey
question
introduces co-variation
by
the
wording
of the
question.
The
following
examples
are from
a national
survey by
E.
C.
Ladd,
Jr. and S. M.
Lipset (as
cited
by
S.
Lang (1978)).
The
questions
were sent
to
college
and
university professors.
(1)
The statements below relate to
teaching
and student
performance.
Does each
correctly
reflect
your per
sonal
judgment?
(i)
The students with whom I have close
contact are
seriously underprepared
in basic skills?such as those re
quired
for written and oral communi
cation.
(a) Definitely yes
(b) Only partly
(c) Definitely
no
(2)
"Grade inflation" is a serious academic
standards
problem
at
my
institution.
(a) Definitely yes
(b) Only partly
(c) Definitely
no
(3)
American
higher
education should ex
pand
the core
curriculum,
to increase the
number of basic courses
required
of all
undergraduates.
(a) Definitely yes
(b) Only partly
(c) Definitely
no
The
questions
assume a
sample
of re
spondents
familiar with the content of the
items
(in
the
present
case American col
lege
and
university professors)
as well as
with the
meaning
of
"definitely yes," "only
partly,"
and
"definitely
no." The
questions
and the
responses
reflect a co-variation
that enables a reader to
parse
the
question
easily.
The answers
permit easy aggrega
tion so that these and other
questions
can
be cross-tabulated with size of
school, age
of
respondent, discipline
or field of
study
and the academic rank of the
respondent.
In the
present
case these
questions
were
mailed out to the informant. There was no
way
for the
respondent
to obtain informal
clarification about
questions
from the
interviewer. The
problem solving aspects
of
answering
a fixed-choice
question
are
severely
limited. There were no
open
ended
questions
to make it
possible
to ob
serve some elements of
a
problem-solving
strategy
at work
(including
the limitations
of the
respondent's knowledge base)
as
the interviewee and interviewer
negoti
ated the
questions
and answers. The
questions
on the basic skills of
students,
grade inflation,
and the idea of
expanding
the core curriculum attribute an
expertise
to the
respondent
that cannot be chal
lenged.
There is no
possibility
of
exploring
the individual
experiences
of teachers
within the same
subject
area much less
across
disciplines.
Nor can we
distinguish
respondents by
the level of
college
or uni
versity
classes
they
teach. We have no
information on the
background
of their
students.
The
categories
created
by
the re
searcher must be
negotiated individually
by
each
respondent.
But the researcher's
categories provide ready-made
classes
and the
response
set
generates
automatic
criteria for
deciding
class
membership.
It
is difficult to
interpret
the
meaning
of
these
personal judgments
vis-?-vis the ex
periences
and
knowledge
base of a
pre
sumed
group
of
"experts."
The
expertise
is an automatic creation of
identifying
a
population
known as
being college
and
university
teachers. The
responses
we
obtain, however,
remain
ambiguous
per
sonal
judgments.
The
ecological validity
of the
response
is not addressed.
Class
membership
and interview and
survey categories
An
important aspect
of
surveys
and
open-ended
interviews is the extent to
which a
concept
or class
membership
is
presupposed
in the
way
we elicit infor
mation from
respondents.
Our
ability
to
perceive,
remember,
and talk about
some
object
or event as an instance of a class or
concept
we are
presumed
to know is fun
damental to the
way
we use
surveys
and
open-ended questions.
Public and
private
bureaucracies
so
cialize their
employees
to the use of
categories
that can subsume
a
variety
of
activities under identifiable classes and
thus confer
a
stability
on the world that
enables them to
go beyond
the informa
tion
given.
We
negotiate
the
assignment
of
an
object
or event or some
aspect
of in
formation to a class
initially
on the basis of
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
18 The American Sociologist
expectations
of its
perceptible elements,
and then
begin
to infer some of the ob
ject's
or event's
nonperceptible
attributes.
This is like
saying
that we construct the
identification of
typical
features that en
able us to claim class
membership
for an
object
or
event,
or we claim that certain
features
suggest
one or more
possible
classes that
trigger
a search for additional
elements that enable us to choose
among
several
possible
candidates of classes. In
our
perception
of
speech,
for
example,
we
may
be forced to
imagine
or recall or
search for information that extends be
yond
conventional or
dictionary interpre
tations of what is said because of the so
cial
setting.
Our
ability
to make the non
perceptible
"visible" and hence
integral
to
the invocation of a class or
concept
is a
basic
process necessary
for all social in
teraction and bureaucratic
practices.
A recent
paper by
Medin and Smith
(in
press) distinguishes
between three views
of
concepts.
The first or classical view
requires
that a
concept
have common
properties
which become
necessary
and
sufficient to define the
concept. Every
member of a class can be
specified by
the
properties they
all must
possess through
a
single description
of all members. Medin
and Smith note that attacks of this view
revolve around the
properties
of
descrip
tion that must be true of all members.
The second view of
concepts
is called
the
prototype
or
poly
the tic
position.
The
focus of this view is on the
way
instances
of a
concept
can
vary
in th?
degree
to
which
they
all share certain
properties.
This view
says
that the different instances
can
vary
in the extent to which
they
will
embody
the
concept.
A
single description
of some
may again suffice,
but the
prop
erties of the
description
are now
only
true
for most but not all of the members. Some
instances of the class will
possess
more of
the critical
properties
than will other in
stances. Those instances
possessing
more
critical
properties
are said to be more
rep
resentative of the
concept
in
question.
The third view described
by
Medin and
Smith is called an
exemplar
notion. This
view of a
concept
states that no
single
representation
exists for an
entire class or
concept,
but instead
only specific repre
sentations of the class's
exemplars
occur.
The
example given by
Medin and Smith
for this third view is that of the class of
persons
who
might
be called suicidal.
The
example
of
persons
who
might
be
suicidal is used
by
Medin and Smith to
compare
the three views
briefly. They
note that the classical view failed when
used
by
clinicians because
no
necessary
and sufficient common
properties
could
be found to define all
people
who have
suicidal tendencies. The
polythetic
view
fails, say
Medin and
Smith,
because it falls
short of
revealing
how someone decides
that a
particular person
is suicidal. Fol
lowing
a
suggestion by Twersky
and
Kahneman
(1973),
Medin and Smith
con
tend that because clinicians are not
likely
to use a
single description
of all
persons
with suicidal
tendencies, they might
in
stead make the decision about someone
being
suicidal
by comparing
the individual
to other
persons
known to be suicidal. The
exemplar
view would result in the class of
people
with suicidal tendencies
being rep
resented
by separate descriptions
for
various
people
known to be members of
the class of suicidal
persons.
Two
key questions
raised
by
Medin and
Smith here are as follows:
(1)
Is it
possible
to have a
single
or
unitary description
for all members of the
class?
(2)
Can we
say
that all of the
properties
specified
in a
unitary description
are true
of all members of the class?
According
to Medin and Smith the
classical view would answer both
ques
tions with an affirmative
response,
while
the
polythetic
view would
say yes
to
(1)
and no to
(2).
The
exemplar
view would
say
no to
(1)
and consider
(2)
to be
irrelevant.
When we use fixed-choice
question
naire items the
respondents
are
expected
to be able to
recognize
the classes of ob
jects
stated in each item as self
explanatory.
This
expectation
derives
from the assumed
pretesting
of each
questionnaire
item
prior
to
sending
out or
utilizing
the final
questionnaire.
The ex
tent to which the
respondent possesses
the
necessary knowledge
base in order to
answer the
question
is seldom a relevant
issue. The
possibility
that the
concepts
or
classes
presented
to the
respondent may
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
The Problem of Ecological Validity 19
not be defined
clearly
in his or her mind is
also seldom an issue. In both cases the
forced-choice nature of the
survey guar
antees an
"adequate" response
so
long
as
the
respondent
is
willing
to check off one
of the choices available. The researcher's
versus
the
respondent's conception
of a
particular
class or
concept
is
presumed
to
be resolved
by
the
pretest
done to the final
questionnaire.
The issue of
possible
social
classes used is not
directly
testable. What
is testable is the
way
different
respondents
can be
distinguished by
some measure of
social class as
determined
by
the
way
fixed-choice
questions
are
constructed,
answered,
and coded.
Having
created
several social classes
by
one set of
ques
tions,
we examine other
questions
an
swered
differentially
and can
attribute the
differences to social class
membership.
Surveys
are
presumed
to be
hypothesis-driven
and hence a
way
of
testing theory.
But the technical
aspects
of the instrument make it difficult to
clarify
theoretical
concepts
or classes said
to have motivated the use of the
survey.
Theoretical
concepts
are subservient to
the mechanics of
creating
and
imple
menting questions
and their
coding.
The
discourse and textual
processes
and
mechanisms that
provide
the theoretical
basis for
surveys
remain
unexamined,
as
are the social constraints and
practices
of
the
society's
social
stratification system
that enable the researcher to utilize such
an esoteric and indirect instrument to
learn about the
everyday
activities and
beliefs of the members of a
group.
Conclusion
Part of the
paper
has been critical of
survey
research. I have
slighted
several
issues. For
example,
conscientious
survey
researchers seek a form of
quasi-experi
mentation with
questionnaire
items in
which a
particular question
frame is re
tained but a
particular
word
(or perhaps
phrase)
is altered.
Important
differences
often are found because of these
changes
(cf.
Schuman and
Presser, 1981),
even if it
is not
always
clear what sort of
reasoning
we should attribute to the
respondents.
What is
important
for the
survey
re
searcher is the
patterning
that occurs or
emerges
that
gives
us more confidence in
the overall
survey.
When the same
ques
tions are used across different
groups
or
with the same
group
at different
times,
and
similar,
or the
same,
patterning
emerges,
then the researcher feels consid
erably
more confident that the
question
naire is
reflecting something significant
about the
respondents' opinions,
atti
tudes,
or beliefs.
Knowing
that some
identifiable
group
was more in favor of
some action or law this
year
than last
year
is
part
of the formal
patterning
that is
sought.
There still can be
problems
here
when
particular
researchers throw out
items that do not seem "to
work,"
but the
general
idea is to avoid
using question
naire items as if
responses
on a
given
oc
casion can be treated in some absolute
way.
The
goal
is to look at some
group
or
category
relative to other
groups
or
categories
at
specific periods
of
time,
and
not how
many respondents
endorse a
given
item at a
particular
time.
The
survey
researcher seeks to control
the
way
a data base is
generated by
creat
ing
restricted conditions under which in
formation is to be
elicited, coded,
and an
alyzed.
The conditions
simplify
and dis
tort the
daily
life activities of those
groups
and institutions we seek to understand and
predict,
but the controls and restricted
data base are
highly
valued
by many
social
scientists because
they
foster a sense of
scientific
rigor
in our research.
Another source of control in
survey
re
search can be found in the enormous ad
vances that have occurred in
sampling
theory
and the researcher's
ability
to sam
ple
different
respondents.
What is more
difficult is the
sampling
of behavior. In the
case of
voting
behavior we find a
fairly
close
correspondence
between what
people say
in
response
to a
questionnaire
item and the
way they actually
vote. But
other
topics
do not fare as
well,
and some
not well at all. We are not clear about the
behavior or activities the
survey ques
tionnaire items are said to index.
People
are not
very
accurate in
describing
their
own behavior when asked to
respond
to
direct
questions.
The
primary difficulty
remains the absence of
strong
theories.
Instead of
using strong
theories we invari
ably rely
on the detection of
patterning
in
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions
20 The American Sociologist
survey responses
in order to
guide
us in
making
theoretical
explanations
after the
fact.
Theory
seldom
guides
social re
search
explicitly;
we
depend
on research
findings
to decide which theoretical con
cepts
seem
appropriate.
Sophisticated survey
researchers
surely
can find "answers" or
"replies"
to the is
sues I have raised and will
point
to the use
of other sources of data or additional
checks or
strategies
that I have not cov
ered in the
paper.
W? need
strong
theories to decide
whether a
particular
method and the data
it
yields
tells us
something
worth know
ing.
We all are forced to deal with the
same
problem
of
interpretation regardless
of whether we use
surveys,
census mate
rials,
vital
statistics,
extensive interview
ing, participation observation,
or audio or
video
tapes.
The
interpretation
issue is
seldom the focus of
survey research,
much less
any
other
type
of research in
sociology.
Notions like limited
capacity
processing, comprehension
of discourse
and textual
materials,
and
language
use in
socially
constrained
contexts,
remain
pe
ripheral topics
in
sociology.
Yet
they
ad
dress the
interpretation
issue in several
explicit ways.
Can we afford the con
venience of
ignoring
these issues?
REFERENCES
Austin,
John L.
1962 How To Do
Things
With Words. Cam
bridge,
MA: Harvard
University.
Bransford,
John
D., Barry
S.
Stein,
Tommie S.
Shelton,
and Richard A.
Owings
n.d.
"Cognition
and
adaption:
The
importance
of
learning
to learn."
Unpublished manuscript.
Cole, Michael,
Lois
Hood,
and
Ray
P. McDermott
1978
"Ecological
niche
picking: Ecological
inva
lidity
as an axiom of
experimental cognitive
psychology." Unpublished manuscript.
Dressier, Wolfgang
U.
(ed.)
1977 Current Trends in
Textlinguistics. Berlin,
NY: de
Gruyter.
Grice,
H. Paul
1975
"Logic
and conversation."
Pp.
41-58 in P.
Cole and J.
Morgan (eds.), Syntax
and
Semantics. New York: Academic.
Grimes, Joseph
E.
1980 "Context structure
patterns." Paper pre
sented at the Nobel
Symposium
on Text
Processing, Stockholm, August.
Halliday,
Michael A. K.
1967 "Notes on
transitivity
and theme in En
glish."
Journal of
Linguistics
3:199-244.
Harrah,
David
1973 "The
logic
of
questions
and its relevance to
instructional science." Instructional Sci
ence 1:447-467.
Kintsch,
Walter and Teun A. van
Dijk
forth- "Toward a model of text
comprehension
coming
and
production." Psychological
Review.
Lang, Serge
1978 "The
professors:
A
survey
of a
survey."
The New York Review of Books 25:38-42.
Medin, Douglas
and Edward Smith
n.d.
"Representation
and
processing
of lexical
concepts." Unpublished manuscript.
Norman,
Donald A.
1973
"Memory, knowledge,
and the
answering
of
questions." Pp.
135-165 in R. S. Solso
(ed.), Contemporary
Issues in
Cognitive
Psychology:
The
Loyola Symposium.
Washington,
DC: Winston.
Schuman,
Howard
1966 "The random
probe:
A
technique
for
evaluating
the
validity
of closed
questions."
American
Sociological
Review 21:218-222.
Schuman,
Howard and
Stanley
Presser
1977
"Question wording
as an
independent
vari
able in
survey analysis." Sociological
Methods and Research 6:151-170.
1977- "Attitude measurement and the
gun
control
1978
paradox."
The Public
Opinion Quarterly
41:427-438.
1979 "The
open
and closed
question."
American
Sociological
Review 4:692-712.
1981
Questions
and Answers in Attitude Sur
veys: Experiments
on the Effects of
Ques
tion
Form, Wording
and Context. New
York: Academic Press,
forth- "Public
opinion
and
public ignorance:
The
com- fine line between attitudes and non
ing
attitudes." American Journal of
Sociology.
Searle,
John R.
1969
Speech
Acts.
Cambridge: Cambridge
Uni
versity
Press.
Tversky,
Amos and Daniel Kahneman
1973
"Availability:
A Hueristic for
judging
fre
quency
and
probability." Cognitive Psy
chology
5:207-232.
van
Dijk,
Teun A.
1972 Some
Aspects
of Text Grammers. The
Hague:
Mouton.
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM
All use subject to JSTOR Terms and Conditions