Comment: Culture, Rigor, and Science in Educational Research Author(s): Frederick Erickson and Kris Gutierrez Source: Educational

Researcher, Vol. 31, No. 8 (Nov., 2002), pp. 21-24 Published by: American Educational Research Association Stable URL: http://www.jstor.org/stable/3594390 . Accessed: 14/11/2013 14:39
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend access to Educational Researcher.

http://www.jstor.org

This content downloaded from 200.123.187.130 on Thu, 14 Nov 2013 14:39:15 PM All use subject to JSTOR Terms and Conditions

Comment

Culture, Rigor, and Science in Educational Research

by FrederickEricksonand KrisGutierrez
In this article the authors argue that both the Feuer, Towne, and Shavelson articleandthe largerNationalResearchCouncil(NRC) report on which it is based must be understood in the context of current federal discourse that focuses narrowly on experimentally derivedcausalexplanations of educationalprogrameffectiveness.Although the authors concur with much of the Feuer et al. article and the NRC report,they are concernedthat the NRC committee,by acin educationalreits chargeto definethe scientific ceptinguncritically search,produceda statementthat risksbeingread as endorsingboth of takingan evidence-basedsocial the possibilityand the desirability nationwide. to educational Finally, engineering improvement approach the layperthe authors review the consequences of not challenging it with a more comson's "whitecoat"notion of science andreplacing plicatedand realisticview of what actualscientists do and the varied and complex methods and perspectivesthey employ in their inquiry.

Culture hereis muchwith whichwe agreein "Scientific Research" andEducational (thisissue)andevenmorein the NationalResearch Council(NRC) report(2002) on whichit is based.First,the article andthe report stateforthrightly that evidence-careful research done by susprimarily descriptive tainedfirsthand observation andinterviewing-sometimescalled or ethnography-can makevaluable conor case qualitative study detributionsto educational and that evidence-careful research, in of falls within the education research methods scriptive range that can be called scientific. Second, the articleand the report makemanyreasonable for the organization of recommendations the Departmentof Educationas a federalgovernmentsponsor of research. shouldbe parThey note thatsuchresearch sponsors that in a democracy insulated from but tially politicalpressures, thatinsulation canneverbe totalnorshouldit be.We suggest two linesof additionandcorrection. The firstconcerns mattechnical tersregarding research thatwerementionedin passing qualitative in thearticle emandreport butwhichwe believe deserved greater The of there. second concerns the basic the NRC phasis premise committee'scharge. One technicalmatteris that the articleand the reportboth in orderto decausalanalysis emphasize by meansof experiment as of termineeducational the currentfedeffects, part program eraland statesystemof accountability for effectiveness. But the articleand the reportdo not makeit clearthat in the veryenteris more research effects,qualitative priseof determining program
Vol.31, No. 8, pp.21-24 Educational Researcher,

it is essentialif causalanalysisis to sucthan merelyallowable; ceed.A logicallyandempirically priorquestionto "Didit work?" as actually is "Whatwas the 'it'?"-"What was the 'treatment' aresituatedand dynamically treatments Educational delivered?" & Ball,2002). They arelointeractive (seeCohen, Raudenbush, of life social constructed ways involvingcontinualmonically and mutual adjustmentamong persons, not relatively toring like chemicalcompoundsor surgicalproceentities replicable duresor hybridseedcornor manufactured airplane wings.High of local is rarein education-for reasons fidelityimplementation the and exigency-and despite the accountabilitypressures vato avoidthis majorthreatto internal wishesof experimenters the implelimits on how "faithful" lidity, there are real-world of instructional mentationwill be of even the most structured of a research Unless considerable budget, proportions programs. aredevotedto documentformalexperiment, evenin a large-scale inferences on the ground,the causal asdelivered ing the treatment of outcomedatawill remain drawnfrominspection unwarranted, half truths."What and they arelikelyto be partially misleading is a questionbest answered was the treatment,specifically?" by esIt is expensiveand absolutely research. necessary, qualitative peciallyif the push to scaleup continues. the NRC reportacA secondtechnical matteris thatalthough for can research that provideexplanations knowledges qualitative such how crucial it does not emphasize causalprocesses, insights Rethe Educational of causalanalysis; can be for the enterprise searcher (ER)articleis even less directon this point. Light and Pillemer(1982), for example,discussa studyof day carecenters inthatthe mostimportant in whichquantitative results suggested neither was fluenceon the children'seducational development ratio,whichwerethe indepengroupsizenor the staff-to-student in the study. definedand measured dent variables operationally went to look in the day carecentersthey found When observers whilein the morethanone adultin the larger groupsof children, the In smaller there was one adult. only larger groupsthe groups thuspayinglessattentionto the chiladultstalkedto one another, wastheactual dren.In otherwordsstaff-to-staff proxschmoozing ratioor overall staff-to-student not the imalindependent variable, At issuewas the sizeof instructional groups,in andof themselves. such as groupsize not the simpleamountof a unitary"resource" ratioas generalprogram features but the speor staff-to-student resources withinthe conduct cificinteractive useof suchpotential to deof instructional Directobservation was necessary practice. to Joseph tect andexplainthis. (We areindebtedforthisexample in Cohenet al., 2002.) Maxwell,1996;see alsothe discussion the premise concerns A secondandmorebasicline of criticism of the chargeto the NRC committee in the first place, which Some in educationalresearch. was to definewhat was scientific
NOVEMBER 2002 21

This content downloaded from 200.123.187.130 on Thu, 14 Nov 2013 14:39:15 PM All use subject to JSTOR Terms and Conditions

on the scientificby the Department of Educapronouncements tion andby members of Congress, suchasRepresentative Michael Castleof Delaware, indicatethatwithinthe executive and legislative branches of the federal a leap of faithhas been government takentowardbelief in the unmixedblessings of hardsciencecausalanalysis by meansof experiment-as the only way to imThese beliefsare truly alarmingin prove educationalresearch. theirnaiveteand zeal. Feueret al.reference withoutcritical commentthe Department of Education's Plan for 2002-2007, Strategic noting that it is available at the federal website (www.ed. government publications member ofAERA gov/pubs/stratplan2002-07/index.html). Every shouldvisitthiswebsiteandreadthe language of Strategic Goal4: "Transform education into an evidence-based field." The statementbegins(p. 50), "Unlikemedicine,agriculture, and industrial the fieldof educationoperates production, largely on the basisof ideologyand professional consensus. As such,it is to fadsandis incapable of the cumulative thatfolsubject progress lows fromthe application of the scientificmethodand from the collectionanduse of systematic information in policy objective We will making. changeeducation to make it an evidencebased field." Concerning the qualObjective4.1, "Raise of research or confunded ity ducted by the Department" (p. 51), the list of implementation strategies includes "improve the peer review process clear standards by articulating andby enlistingonly thosequalified scientists who havehigh levels of methodological and substantive and "create an expertise" editorialreviewboardfor all Departmentresearch publications to ensurethat they meet the higheststandards of scientificrigor beforetheirpublication." UnderObjective the rel4.2, "Increase evanceof our researchin orderto meet the needs of our customers"(p. 52), an orientingcomment says, "the Department will create andregularly of scientifically updatean onlinedatabase research on what works in education." rigorous Note thatin the firstquotedsentence"thescientificmethod" standsunqualified as the termin contrastto "ideology," "professionalconsensus," and "fads." The next sentencesays,"Wewill field,"but it is changeeducationto make it an evidence-based clearthatonly one kindof evidenceis to be givencurrency. Considerthe effectin the nexttwo quotedsentences Obconcerning 4.1 of scholars or for "scientists" and jective substituting experts or research for "scientific Consider scholarly rigor rigorous rigor." the effectsin the lastquotedsentenceaboutan on-line database of sayingthat it would containhigh qualityresearch on issuesof currentimportance in education.Insteadthe database will show us "scientifically research on what works in education" rigorous without close and thoroughexamination of how and why those treatments "work." In addition,we haveeachwitnessedofficialsof the U.S. Office of Educational Research and Improvement curcomparing renteducational with those of medieval medicine practices (e.g., the currentstate of bleeding to cure fever)and characterizing
RESEARCHER 221 EDUCATIONAL

Real science is not

about certainty but about uncertainty.

knowledgein educationas superstition.We are not postmodernists,but we must protestthat this "highmodernist" position constitutesa substitutionof scientism, an idealization of science, for scienceitself.The prescription of a "scientific culture" as an effectiveremedyfor the ills of educational research and of "hard science"causalstudiesof program effectsas a remedyfor defects in educationpracticemust be treatedvery skeptically.Reluctantly,becausewe like so much of the ER articleand the NRC report,we must arguethat they cannot be readapartfrom the contextof the broader federal discourse the fieldof edconcerning ucation.In thatcontext,by not challenging thereigning optimism abouthardscienceas a royalroadto improvement, the article and committeereportcould be readas supporting a discourse of scientismor the appearance of rigorin educational research rather than its actualsubstance. The problemis that the house is burningaroundus and the articleand reportnevercry "Fire!" or "Danger!" Recallthat the NRC committee'schargewas not to definestandards for rigor, research but to saywhat was reason,and qualityin educational scientific. And although the report and article tried to range broadlyacrossan intellectual playing field that was profoundlytilted at the outset, they neverquestionedthe fundamental premise of their charge.Indeed, the ER article down this slippery goes farther slope than does the committee As Feueret al. sayin so report: use manywords,"Wetherefore

terms like science, research,scholarship,and inquiry as essentially interchangeable in the specific context of discussing the norms and ideals of the educational researchfield" (p. 5). Furthermore, although the article and report mention Lakatos and Musgrove (1970) in passing, they do not mention at all the substantial line of work since the 1970s called social studiesofscience,which documents the ways in which actual scientists' practice differs from that of its idealized characterizations (see, e.g., Latour & Woolgar, 1986; Lynch, 1993; Knorr-Cetina, 1999). From Lakatoson, many scholars have shown that real scientists in their daily work are anything but disinterested and canonically rational. In their daily practice they are passionate and argumentative, profoundly selective in their attention to evidence, and aesthetic in drawing conclusions from it. Many change their minds and paradigms only with great reluctance. The actual "culture of science," in other words, is far from the white coat image that appears to the layperson.The accumulation of knowledge in actual science is not at all continuous-it moves by fits and starts. Real science is not about certainty but about uncertainty. Indeed, contemporary philosophy of science is calling into question Hume's "successionist"view of cause as involving regularities among repeated sequences of events-the notion that a single temporally antecedent phenomenon (which can be operationally defined as a unitary variable) causes a single consequent result, recurrently.In social analysisespecially, it is apparent that, as Sayer observes, "the same causal power can produce different outcomes (for example, economic competition can prompt firms

This content downloaded from 200.123.187.130 on Thu, 14 Nov 2013 14:39:15 PM All use subject to JSTOR Terms and Conditions

to restructure andinnovate or to close.Sometimes different causal mechanisms can producesthe sameresult:for instance,you can loseyourjob fora variety of reasons" 2000, p. 15;seealso (Sayer, causal Salmon,1998). Bythiswayof thinking,an adequate analysis must identifythe specificmechanisms that generatespecific outcomes within particular circumstances. structural It is just suchspecification thatthe Lightand Pillemer(1982) example of schmoozingprovides-a specificationmade possibleby direct observationfrom within the local situationof complexityand of the hierarchically contingency.The varietyand changeability embeddedcontextsof sociallife aresuch that simple,consistent associations betweengenericcauseand genericeffectof the sort testedin formalsocialexperiments arenot likelyto occur.From this point of view the levelof abstraction in the operational definition of aspectsof social processas unitaryvariables-which characterizes social experiments-results in knowllarge-scale as roughapproximation or edge thatat bestcan be characterized This is not to saythatvalidconclusionscan neverbe guesswork. drawnfromexperimental studiesof "what but rather that works," those who do such researchneed to cultivate skepticismand assessment of the difficulties andlimitsthat humility-a realistic areinherentin theirapproach. The Department of Education's Strategic Plan holds up evidence-based medicinefor emulationby educational research. And yet the moment one entersthe arenaof behavioral and social aspectsof medicalservicedelivery-the contingentcircumstancesof medicalpracticemost similarto those of educational studies practice-one findsthat the conclusionsof experimental in medicalresearch do not resultdirectlyin the outcomesof curbecauseof the enduring,embarby researchers ing anticipated rassingpresenceof locally constructedsocial facts (e.g., that many patients do not follow the prescribedmedical regimen: reThey don't taketheirpills or do theirexercises). Qualitative searchers would saythatthesepatientsaremakingsensebut that it is a differentkind of sense from that of the evidence-based medicalpractitioner. we havedoubtsaboutthe appropriateness of Finally,although the practice of medicineand educationas analogous, let treating us staywith that analogyto makeone morepoint. We areconcernedthat premature conclusionsabout "whatworks"in the shortterm,withoutcareful of side effectsthatmay consideration canprovide falsewarrants forthe educational downstream, appear of thalidomide. Thatwasa medical treatment thatwas equivalent shownscientifically (i.e., by meansof randomized trials)to have clearpositive effects.A non-barbiturate hypnotic, thalidomide wasoriginally after1956 to preventmorningsickness prescribed in pregnantwomen and to help them sleep throughthe night. What a tragicirony;thalidomidepreventedmorning sickness but it was alsoeffectivein causingdeformities in veryeffectively the fetusgrowingin the mother'swomb. The lattereffectswere afterthe babieswere born, and it took yearsto only discovered trace the cause of the deformitiesback to the mothers'use of thalidomide.Will our currentdesperateattempts to discover test scoresin the short run "whatworks"to raisestandardized haveanalogous effectson ourchildrenandteachers in school,effects that areonly apparent aftermuch damagehas been done? How do we designexperiments that can predictdangerous educationalside effects?

To conclude,the unmixedoptimismaboutsciencein educationalresearch and the hopesfor evidence-based socialengineerreformappear to us to be unwise, ing as a meansof educational evensuperstitious. In ourexperience, positiveeducational change is accomplishedlocally and it is more like walking through a swamp,testingthe groundwith eachstep, than it is like driving on a superhighway or even like building one. To get smarter aboutworkingour wayin a swampwe need all kindsof research anddeliberation, scientific and nonscientific. And we needpractitionerresearch. The currentfederaldiscourseabout improvement in educationalresearchfails to emphasizepractitioner is to occurin research as a sourceof insight;yet if realprogress ourswamp-situated efforts,it seemsquiteobviousto us that the and of research must grow specialists knowledgeof practitioners in newways,a pointmadeveryrecently in ERby Hiebert, together Gallimore,and Stigler(2002). Moreover,neitherthe Departmentof Education'sStrategic Plan nor the NRC reportforegrounds the plain historicalfact that educationalresearchhas been dramatically underfunded If anywhere nearthe federalinvestmentover the past federally. and development,or that in medicenturyin militaryresearch it is likelyjust cine, had been providedfor educational research, on actuarial groundsthat considerable insightwould havebeen gained. For the federalgovernmentto criticizeeducationalresearchfor lackof progress while havinglong withheldadequate and even hypocritical. fundingfor it seemscontradictory We think it is deeplymisleading to claimthat expressways to can be engineered educational sociallyupon bedrockof progress and consistentevidenceabouteffectivesmoothlyaccumulating ness-and thatthiscanbe donereadily on the cheap.The current federal for certainty in social leap of faithto scienceas a warrant policyremindsus of the jejunecontemptheldby the Renaissance for the MiddleAges,a "presentism" thatwasrepeated in the Enlightenment.Formanyat the beginningof the 20th centuryand at its endingaswell, thathope in progress asthe resultof the conof new knowledge tinuousdevelopment cameto appear asa delusion. Our concerncan be summarized in a paraphrase of the old Those who do not know theirintellectual aphorism: historyare condemnedto repeatit.
REFERENCES S. W., & Ball, D. W. (2002). Resources, Cohen, D. K., Raudenbush, In R. Boruch& F. Mosteller(Eds.),Eviinstruction,and research. dencematters: Randomized trialsin educational research. Washington, DC: The Brookings Institution. Hiebert,J., Gallimore,R., & Stigler,J. (2002). A knowledgebase for the teachingprofession: Whatwould it looklike,and how canwe get one?Educational Researcher, 31(5), 3-15. cultures: How the sciences make Knorr-Cetina,K. (1999). Epistemic MA: Harvard knowledge. UniversityPress. Cambridge, A. (Eds.).(1970). Criticism and thegrowthof Lakatos, I., & Musgrove, UK: Press. knowledge. University Cambridge, Cambridge Latour,B., & Woolgar,S. (1986). Laboratory life: Theconstruction of facts.Princeton,NJ: PrincetonUniversityPress scientific Light,R., & Pillemer,D. (1982). Numbersand narrative: Combining their strengthsin researchreviews. HarvardEducationalReview, 52(1), 1-26. andordinary action: EthnomethodLynch,M. (1993). Scientficpractice and social studies UK: ology ofscience.Cambridge, CambridgeUniversityPress.
NOVEMBER 2002 [

This content downloaded from 200.123.187.130 on Thu, 14 Nov 2013 14:39:15 PM All use subject to JSTOR Terms and Conditions

Maxwell, J. (1996, October). Using qualitative researchto developcausal explanations. Cambridge, MA: Harvard Project on Schooling and Children. National Research Council. (2002). Scientific research in education. R. J. Shavelson & L. Towne (Eds.) Committee on Scientific Principles for Education Research. Washington, DC: National Academy Press. Salmon, W. (1998). Causalityand Explanation. New York: Oxford University Press. Sayer,A. (2000). Realismand socialscience.London: SAGE Publications.

video-based of tion, the studyof learningin socialinteraction, analysis oral discourse,and contemporary research inmethods, ethnographic of teachingpractice. cludingdigitalmultimediadocumentation KRIS GUTIERREZ is a fullprofessor at the University of California, Los and PolAngeles,UrbanSchooling,Curriculum, Teaching,Leadership, icy Studies,1026 Moore Hall, LosAngeles,CA 90095-1521; krisgu@ ucla.edu.Her research interests focuson understanding the relationship between language,culture,and human developmentand include the study of the social and cognitive consequencesof literacy,practices, and policies in urbanschools. Her work also examinesthe effects of currenteducationalreforms,includingnew languagepolicies on poor and working-class students-particularly those who areEnglish LanguageLearners. received June 10, 2002 Manuscript Revisionsreceived July7, 2002 Accepted July I I, 2002

AUTHORS
FREDERICK ERICKSON is the George F. Kneller Professor of Anthropology of Education in the Social Research Methodology Division, University of California, Los Angeles, Graduate School of Education and Information Studies, 1002 Moore Hall, Box 951521, Los Angeles, CA 90095-1521; ferickson@gseis.ucla.edu. He is also the Director of Research at the Corinne A. Seeds University Elementary School at UCLA. His research interests include issues of equity in urban educa-

Curriculum Inquiry
AJournal Institute from TheOntario for Studiesin Education of the University of Toronto

A librarysubscriptionto the print volume entitlesreadersto:

Edited CONNELLY by F. MICHAEL

tothestudy ofeducational Curriculum Inquiry is dedicated Sample Free and This research, evaluation, theory. leading development, influential international academics and journal brings together from a variety of disciplines theworld researchers around to provide andlively debate. Articles ideas, expert commentary explore important
and each issue also includes issues, trends,andproblemsin education, and critically editorials provocative analytical topics such as covering

Freeaccesstofull textarticles online Freecopying for non-commercial course packs Freeaccessto all available electronic backvolumes

Select

curriculum educational andteacher education. development, policy,

"Oneof the few education journals

for FREE your register updates simplyvisi... Toviewa FREE online visit: To issue, sample blackwellpublishers.co. uk/journals/ci httpV/select.Iackwelpubishers.co.ukI oremail toreceiiiiiii want hday@blackwellpub.com SELECTexactlywhatu ofyour ch contents JOURN-AL: fronthejournals SELECT
and ofbooks news SELECT josualsby subjecii when SELECT your messages arrive-rigtdo

E-mail Updates e-maild to you ......... Contents tornilly,

Have CuriculumInquiry '

...:.._ i .. .....iii.. i iiii-i iii{ -iiii{}

Blackwell Publhing's

education." Maxine Columbia USA Greene, University,

to open upcontemporary ongeneral theoretical perspectives

Publishing
Inquiry-s
website
a

Blackwel Phone: 1-800-835-6770 Fax: (781)-388-8232

www.blockwellpub.com

e-mail: subscrip@blackwellpub.com

tothe ofthe week day

ca/,.ci/

Visit

Curriculum

tl

-h-ttp://home.Oise?

L'Itl

utoronto
I

24

EDUCATIONAL RESEARCHER

This content downloaded from 200.123.187.130 on Thu, 14 Nov 2013 14:39:15 PM All use subject to JSTOR Terms and Conditions