JointCommitteeonQuantitativeAssessmentofResearch

CitationStatistics
AreportfromtheInternationalMathematicalUnion(IMU)in
cooperationwiththeInternationalCouncilofIndustrialand
AppliedMathematics(ICIAM)andtheInstituteofMathematical
Statistics(IMS)

Correctedversion,
6/12/08

RobertAdler,JohnEwing(Chair),PeterTaylor

6/11/2008

June2008 CitationStatistics IMUICIAMIMS

ExecutiveSummary
Thisisareportabouttheuseandmisuseofcitationdataintheassessmentofscientificresearch.The
ideathatresearchassessmentmustbedoneusing"simpleandobjective"methodsisincreasingly
prevalenttoday.The"simpleandobjective"methodsarebroadlyinterpretedasbibliometrics,thatis,
citationdataandthestatisticsderivedfromthem.Thereisabeliefthatcitationstatisticsareinherently
moreaccuratebecausetheysubstitutesimplenumbersforcomplexjudgments,andhenceovercome
thepossiblesubjectivityofpeerreview.Butthisbeliefisunfounded.

Relyingonstatisticsisnotmoreaccuratewhenthestatisticsareimproperlyused.Indeed,
statisticscanmisleadwhentheyaremisappliedormisunderstood.Muchofmodern
bibliometricsseemstorelyonexperienceandintuitionabouttheinterpretationandvalidityof
citationstatistics.
Whilenumbersappeartobe"objective",theirobjectivitycanbeillusory.Themeaningofa
citationcanbeevenmoresubjectivethanpeerreview.Becausethissubjectivityislessobvious
forcitations,thosewhousecitationdataarelesslikelytounderstandtheirlimitations.
Thesolerelianceoncitationdataprovidesatbestanincompleteandoftenshallow
understandingofresearchanunderstandingthatisvalidonlywhenreinforcedbyother
judgments.Numbersarenotinherentlysuperiortosoundjudgments.

Usingcitationdatatoassessresearchultimatelymeansusingcitationbasedstatisticstorankthings
journals,papers,people,programs,anddisciplines.Thestatisticaltoolsusedtorankthesethingsare
oftenmisunderstoodandmisused.

Forjournals,theimpactfactorismostoftenusedforranking.Thisisasimpleaveragederived
fromthedistributionofcitationsforacollectionofarticlesinthejournal.Theaveragecaptures
onlyasmallamountofinformationaboutthatdistribution,anditisarathercrudestatistic.In
addition,therearemanyconfoundingfactorswhenjudgingjournalsbycitations,andany
comparisonofjournalsrequirescautionwhenusingimpactfactors.Usingtheimpactfactor
alonetojudgeajournalislikeusingweightalonetojudgeaperson'shealth.
Forpapers,insteadofrelyingontheactualcountofcitationstocompareindividualpapers,
peoplefrequentlysubstitutetheimpactfactorofthejournalsinwhichthepapersappear.They
believethathigherimpactfactorsmustmeanhighercitationcounts.Butthisisoftennotthe
case!Thisisapervasivemisuseofstatisticsthatneedstobechallengedwheneverandwherever
itoccurs.
Forindividualscientists,completecitationrecordscanbedifficulttocompare.Asa
consequence,therehavebeenattemptstofindsimplestatisticsthatcapturethefullcomplexity
ofascientist'scitationrecordwithasinglenumber.Themostnotableoftheseisthehindex,
whichseemstobegaininginpopularity.Butevenacasualinspectionofthehindexandits
variantsshowsthatthesearenaveattemptstounderstandcomplicatedcitationrecords.While
theycaptureasmallamountofinformationaboutthedistributionofascientist'scitations,they
losecrucialinformationthatisessentialfortheassessmentofresearch.

Thevalidityofstatisticssuchastheimpactfactorandhindexisneitherwellunderstoodnorwell
studied.Theconnectionofthesestatisticswithresearchqualityissometimesestablishedonthebasisof
"experience."Thejustificationforrelyingonthemisthattheyare"readilyavailable."Thefewstudiesof
thesestatisticsthatweredonefocusednarrowlyonshowingacorrelationwithsomeothermeasureof
qualityratherthanondetermininghowonecanbestderiveusefulinformationfromcitationdata.

2
June2008 CitationStatistics IMUICIAMIMS

Wedonotdismisscitationstatisticsasatoolforassessingthequalityofresearchcitationdataand
statisticscanprovidesomevaluableinformation.Werecognizethatassessmentmustbepractical,and
forthisreasoneasilyderivedcitationstatisticsalmostsurelywillbepartoftheprocess.Butcitationdata
provideonlyalimitedandincompleteviewofresearchquality,andthestatisticsderivedfromcitation
dataaresometimespoorlyunderstoodandmisused.Researchistooimportanttomeasureitsvalue
withonlyasinglecoarsetool.

Wehopethoseinvolvedinassessmentwillreadboththecommentaryandthedetailsofthisreportin
ordertounderstandnotonlythelimitationsofcitationstatisticsbutalsohowbettertousethem.Ifwe
sethighstandardsfortheconductofscience,surelyweshouldsetequallyhighstandardsforassessing
itsquality.

JointIMU/ICIAM/IMSCommitteeonQuantitativeAssessmentofResearch
RobertAdler,TechnionIsraelInstituteofTechnology
JohnEwing(Chair),AmericanMathematicalSociety
PeterTaylor,UniversityofMelbourne

Fromthecommitteecharge
Thedrivetowardsmoretransparencyand
accountabilityintheacademicworldhas
createda"cultureofnumbers"inwhich
institutionsandindividualsbelievethatfair
decisionscanbereachedbyalgorithmic
evaluationofsomestatisticaldata;unable
tomeasurequality(theultimategoal),
decisionmakersreplacequalitybynumbers
thattheycanmeasure.Thistrendcallsfor
commentfromthosewhoprofessionally
dealwithnumbersmathematiciansand
statisticians.

3
June2008 CitationStatistics IMUICIAMIMS

Introduction
Scientificresearchisimportant.Researchunderliesmuchprogressinourmodernworldandprovides
hopethatwecansolvesomeoftheseeminglyintractableproblemsfacinghumankind,fromthe
environmenttoourexpandingpopulation.Becauseofthis,governmentsandinstitutionsaroundthe
worldprovideconsiderablefinancialsupportforscientificresearch.Naturally,theywanttoknowtheir
moneyisbeinginvestedwisely;theywanttoassessthequalityoftheresearchforwhichtheypayin
ordertomakeinformeddecisionsaboutfutureinvestments.

Thismuchisn'tnew:Peoplehavebeenassessingresearchformanyyears.Whatisnew,however,isthe
notionthatgoodassessmentmustbe"simpleandobjective,"andthatthiscanbeachievedbyrelying
primarilyonmetrics(statistics)derivedfromcitationdataratherthanavarietyofmethods,including
judgmentsbyscientiststhemselves.Theopeningparagraphfromarecentreportstatesthisviewstarkly:

ItistheGovernmentsintentionthatthecurrentmethodfordeterminingthequalityof
universityresearchtheUKResearchAssessmentExercise(RAE)shouldbereplacedafterthe
nextcycleiscompletedin2008.Metrics,ratherthanpeerreview,willbethefocusofthenew
systemanditisexpectedthatbibliometrics(usingcountsofjournalarticlesandtheircitations)
willbeacentralqualityindexinthissystem.[EvidenceReport2007,p.3]

Thosewhoargueforthissimpleobjectivitybelievethatresearchistooimportanttorelyonsubjective
judgments.Theybelievecitationbasedmetricsbringclaritytotherankingprocessandeliminate
ambiguitiesinherentinotherformsofassessment.Theybelievethatcarefullychosenmetricsare
independentandfreeofbias.Mostofall,theybelievesuchmetricsallowustocompareallpartsofthe
researchenterprisejournals,papers,people,programs,andevenentiredisciplinessimplyand
effectively,withouttheuseofsubjectivepeerreview.

Butthisfaithintheaccuracy,independence,andefficacyofmetricsismisplaced.

First,theaccuracyofthesemetricsisillusory.Itisacommonmaximthatstatisticscanliewhen
theyareimproperlyused.Themisuseofcitationstatisticsiswidespreadandegregious.Inspite
ofrepeatedattemptstowarnagainstsuchmisuse(forexample,themisuseoftheimpact
factor),governments,institutions,andevenscientiststhemselvescontinuetodraw
unwarrantedorevenfalseconclusionsfromthemisapplicationofcitationstatistics.

Second,solerelianceoncitationbasedmetricsreplacesonekindofjudgmentwithanother.
Insteadofsubjectivepeerreviewonehasthesubjectiveinterpretationofacitation'smeaning.
Thosewhopromoteexclusiverelianceoncitationbasedmetricsimplicitlyassumethateach
citationmeansthesamethingaboutthecitedresearchits"impact".Thisisanassumptionthat
isunprovenandquitelikelyincorrect.

Third,whilestatisticsarevaluableforunderstandingtheworldinwhichwelive,theyprovide
onlyapartialunderstanding.Inourmodernworld,itissometimesfashionabletoasserta
mysticalbeliefthatnumericalmeasurementsaresuperiortootherformsofunderstanding.
Thosewhopromotetheuseofcitationstatisticsasareplacementforafullerunderstandingof
researchimplicitlyholdsuchabelief.Wenotonlyneedtousestatisticscorrectlyweneedto
usethemwiselyaswell.

4
June2008 CitationStatistics IMUICIAMIMS

Wedonotarguewiththeefforttoevaluateresearchbutratherwiththedemandthatsuchevaluations
relypredominantlyon"simpleandobjective"citationbasedmetricsademandthatoftenis
interpretedasrequiringeasytocalculatenumbersthatrankpublicationsorpeopleorprograms.
Researchusuallyhasmultiplegoals,bothshorttermandlong,anditisthereforereasonablethatits
valuemustbejudgedbymultiplecriteria.Mathematiciansknowthattherearemanythings,bothreal
andabstract,thatcannotbesimplyordered,inthesensethateachtwocanbecompared.Comparison
oftenrequiresamorecomplicatedanalysis,whichsometimesleavesoneundecidedaboutwhichoftwo
thingsis"better".Thecorrectanswerto"Whichisbetter?"issometimes:"Itdepends!"

Thepleatousemultiplemethodstoassessthequalityof
researchhasbeenmadebefore(forexample[Martin1996]
or[CareyCowlingTaylor2007]).Publicationscanbejudged Researchusuallyhasmultiple
inmanyways,notonlybycitations.Measuresofesteem goalsanditistherefore
suchasinvitations,membershiponeditorialboards,and reasonablethatitsvaluemust
awardsoftenmeasurequality.Insomedisciplinesandin
somecountries,grantfundingcanplayarole.Andpeer bejudgedbymultiplecriteria.
reviewthejudgmentoffellowscientistsisanimportant
componentofassessment.(Weshouldnotdiscardpeer
reviewmerelybecauseitissometimesflawedbybias,anymorethanweshoulddiscardcitation
statisticsbecausetheyaresometimesflawedbymisuse.)Thisisasmallsampleofthemultiplewaysin
whichassessmentcanbedone.Therearemanyavenuestogoodassessment,andtheirrelative
importancevariesamongdisciplines.Inspiteofthis,"objective"citationbasedstatisticsrepeatedly
becomethepreferredmethodforassessment.Thelureofasimpleprocessandsimplenumbers
(preferablyasinglenumber)seemstoovercomecommonsenseandgoodjudgment.

Thisreportiswrittenbymathematicalscientiststoaddressthemisuseofstatisticsinassessingscientific
research.Ofcourse,thismisuseissometimesdirectedtowardsthedisciplineofmathematicsitself,and
thatisoneofthereasonsforwritingthisreport.Thespecialcitationcultureofmathematics,withlow
citationcountsforjournals,papers,andauthors,makesitespeciallyvulnerabletotheabuseofcitation
statistics.Webelieve,however,thatallscientists,aswellasthegeneralpublic,shouldbeanxioustouse
soundscientificmethodswhenassessingresearch.

Someinthescientificcommunitywoulddispensewithcitationstatisticsaltogetherinacynicalreaction
topastabuse,butdoingsowouldmeandiscardingavaluabletool.Citationbasedstatisticscanplaya
roleintheassessmentofresearch,providedtheyareusedproperly,interpretedwithcaution,andmake
uponlypartoftheprocess.Citationsprovideinformationaboutjournals,papers,andpeople.Wedon't
wanttohidethatinformation;wewanttoilluminateit.

Thatisthepurposeofthisreport.Thefirstthreesectionsaddressthewaysinwhichcitationdatacanbe
used(andmisused)toevaluatejournals,papers,andpeople.Thenextsectiondiscussesthevaried
meaningsofcitationsandtheconsequentlimitationsoncitationbasedstatistics.Thelastsection
counselsaboutthewiseuseofstatisticsandurgesthatassessmentstempertheuseofcitationstatistics
withotherjudgments,eventhoughitmakesassessmentslesssimple.

"Everythingshouldbemadeassimpleaspossible,butnotsimpler,"AlbertEinsteinoncesaid.1 This
advicefromoneoftheworld'spreeminentscientistsisespeciallyaptwhenassessingscientificresearch.

5
June2008 CitationStatistics IMUICIAMIMS

Rankingjournals:Theimpactfactor 2
Theimpactfactorwascreatedinthe1960sasawaytomeasurethevalueofjournalsbycalculatingthe
averagenumberofcitationsperarticleoveraspecificperiodoftime.[Garfield2005]Theaverageis
computedfromdatagatheredbyThomsonScientific(previouslycalledtheInstituteforScientific
Information),whichpublishesJournalCitationReports.ThomsonScientificextractsreferencesfrom
morethan9,000journals,addinginformationabouteacharticleanditsreferencestoitsdatabaseeach
year.[THOMSON:SELECTION]Usingthatinformation,onecancounthowoftenaparticulararticleis
citedbysubsequentarticlesthatarepublishedinthecollectionofindexedjournals.(Wenotethat
ThomsonScientificindexeslessthanhalfthemathematicsjournalscoveredbyMathematicalReviews
andZentralblatt,thetwomajorreviewingjournalsinmathematics. 3 )

Foraparticularjournalandyear,thejournalimpactfactoriscomputedbycalculatingtheaverage
numberofcitationstoarticlesinthejournalduringtheprecedingtwoyearsfromallarticlespublishedin
thatgivenyear(intheparticularcollectionofjournalsindexedbyThomsonScientific).Iftheimpact
factorofajournalis1.5in2007,itmeansthatonaveragearticlespublishedduring2005and2006were
cited1.5timesbyarticlesinthecollectionofallindexedjournalspublishedin2007.

ThomsonScientificitselfusestheimpactfactorasonefactorinselectingwhichjournalstoindex.
[THOMSON:SELECTION]Ontheotherhand,Thomsonpromotestheuseoftheimpactfactormore
generallytocomparejournals.

"Asatoolformanagementoflibraryjournalcollections,theimpactfactorsuppliesthelibrary
administratorwithinformationaboutjournalsalreadyinthecollectionandjournalsunder
considerationforacquisition.Thesedatamustalsobecombinedwithcostandcirculationdata
tomakerationaldecisionsaboutpurchasesofjournals."[THOMSON:IMPACTFACTOR]

Manywritershavepointedoutthatoneshouldnotjudgetheacademicworthofajournalusingcitation
dataalone,andthepresentauthorsverymuchagree.Inadditiontothisgeneralobservation,theimpact
factorhasbeencriticizedforotherreasonsaswell.(See[Seglen1997],[AminMabe2000],
[Monastersky2005],[Ewing2006],[Adler2007],and[Hall2007].)

(i)Theidentificationoftheimpactfactorasanaverageisnotquitecorrect.Becausemanyjournals
publishnonsubstantiveitemssuchaslettersoreditorials,whichareseldomcited,theseitemsarenot
countedinthedenominatoroftheimpactfactor.Ontheotherhand,whileinfrequent,theseitemsare
sometimescited,andthesecitationsarecountedinthenumerator.Theimpactfactoristhereforenot
quitetheaveragecitationsperarticle.Whenjournalspublishalargenumberofsuch"nonsubstantial"
items,thisdeviationcanbesignificant.Inmanyareas,includingmathematics,thisdeviationisminimal.

(ii)Thetwoyearperiodusedindefiningtheimpactfactorwasintendedtomakethestatisticcurrent.
[Garfield2005]Forsomefields,suchasbiomedicalsciences,thisisappropriatebecausemostpublished
articlesreceivemostoftheircitationssoonafterpublication.Inotherfields,suchasmathematics,most
citationsoccurbeyondthetwoyearperiod.Examiningacollectionofmorethan3millionrecent
citationsinmathematicsjournals(theMathReviewsCitationdatabase)oneseesthatroughly90%of
citationstoajournalfalloutsidethis2yearwindow.Consequently,theimpactfactorisbasedonamere
10%ofthecitationactivityandmissesthevastmajorityofcitations.4

6
June2008 CitationStatistics IMUICIAMIMS

CitationCurves
14%
Impact
Percentof2003citations
12% Factor

10%
CellBiology
8%
Education
6%
Economics
4%
Mathematics
2%
0%
2003 2002 2001 2000 1999 1998 1997 1996 1995 1994

Agraphshowingtheageofcitationsfromarticlespublishedin2003coveringfourdifferentfields.
Citationstoarticlepublishedin20012002arethosecontributingtotheimpactfactor;allother
citationsareirrelevanttotheimpactfactor. Datafrom Thomson Scientific.

Doesthetwoyearintervalmeantheimpactfactorismisleading?Formathematicsjournalstheevidence
isequivocal.ThomsonScientificcomputes5yearimpactfactors,whichitpointsoutcorrelatewellwith
theusual(2year)impactfactors.[Garfield1998]UsingtheMathReviewscitationdatabase,onecan
compute"impactfactors"(thatis,averagecitationsperarticle)foracollectionofthe100mostcited
mathematicsjournalsusingperiodsof2,5,and10years.Thechartbelowshowsthat5and10year
impactfactorsgenerallytrackthe2yearimpactfactor.

Top100MathematicsJournals
3

2.5
"ImpactFactors"

2
2year
1.5 5year

1 10year

0.5

"Impactfactors"for2,5,and10yearsfor100mathematicsjournals.DatafromMath

Reviewscitationdatabase.

Theonelargeoutlierisajournalthatdidnotpublishpapersduringpartofthistime;thesmalleroutliers
tendtobejournalsthatpublisharelativelysmallnumberofpaperseachyear,andthechartmerely
reflectsthenormalvariabilityinimpactfactorsforsuchjournals.Itisapparentthatchangingthe

7
June2008 CitationStatistics IMUICIAMIMS

numberof"targetyears"whencalculatingtheimpactfactorchangestherankingofjournals,butthe
changesaregenerallymodest,exceptforsmalljournals,whereimpactfactorsalsovarywhenchanging
the"sourceyear"(seebelow).

(iii)Theimpactfactorvariesconsiderablyamongdisciplines.[AminMabe2000]Partofthisdifference
stemsfromtheobservation(ii):Ifinsomedisciplinesmanycitationsoccuroutsidethetwoyear
window,impactfactorsforjournalswillbefarlower.Ontheotherhand,partofthedifferenceissimply
thatthecitationculturesdifferfromdisciplinetodiscipline,andscientistswillcitepapersatdifferent
ratesandfordifferentreasons.(Weelaborateonthisobservationlaterbecausethemeaningofcitations
isextremelyimportant.)Itfollowsthatonecannotinanymeaningfulwaycomparetwojournalsin
differentdisciplinesusingimpactfactors.

Averagecitationsperarticle
Mathematics/Computer
Socialscience
Materialsscience
Biologicalsciences
Environmentalsciences
EarthSciences
Chemistry
Physics
Pharmacology
ClinicalMedicine
Neuroscience
Lifesciences

0 1 2 3 4 5 6 7
Citations

Averagecitationsperarticlefordifferentdisciplines,showingthatcitation
practicesdiffermarkedly.DatafromThomsonScientific[AminMabe2000].

(iv)Theimpactfactorcanvaryconsiderablyfromyeartoyear,andthevariationtendstobelargerfor
smallerjournals.[AminMabe2000]Forjournalspublishingfewerthan50articles,forexample,the
averagechangeintheimpactfactorfrom2002to2003wasnearly50%.Thisiswhollyexpected,of
course,becausethesamplesizeforsmalljournalsissmall.Ontheotherhand,oneoftencompares
journalsforafixedyear,withouttakingintoaccountthehighervariationforsmalljournals.

(v)JournalsthatpublisharticlesinlanguagesotherthanEnglishwilllikelyreceivefewercitations
becausealargeportionofthescientificcommunitycannot(ordonot)readthem.Andthetypeof
journal,ratherthanthequalityalone,mayinfluencetheimpactfactor.Journalsthatpublishreview
articles,forexample,willoftenreceivefarmorecitationsthanjournalsthatdonot,andthereforehave
higher(sometimes,substantiallyhigher)impactfactors.[AminMabe2000]

8
June2008 CitationStatistics IMUICIAMIMS

(vi)Themostimportantcriticismoftheimpactfactoristhatitsmeaningisnotwellunderstood.When
usingtheimpactfactortocomparetwojournals,thereisnoapriorimodelthatdefineswhatitmeansto
be"better".Theonlymodelderivesfromtheimpactfactoritselfalargerimpactfactormeansabetter
journal.Intheclassicalstatisticalparadigm,onedefinesamodel,formulatesahypothesis(ofno
difference),andthenfindsastatistic,whichdependingonitsvaluesallowsonetoacceptorrejectthe
hypothesis.Derivinginformation(andpossiblyamodel)fromthedataitselfisalegitimateapproachto
statisticalanalysis,butinthiscaseitisnotclearwhatinformationhasbeenderived.Howdoesthe
impactfactormeasurequality?Isitthebeststatistictomeasurequality?Whatpreciselydoesit
measure?(Ourlaterdiscussionaboutthemeaningofcitationsisrelevanthere.)Remarkablylittleis
knownaboutamodelforjournalqualityorhowitmightrelatetotheimpactfactor.

Theabovesixcriticismsoftheimpactfactorareallvalid,buttheymeanonlythattheimpactfactoris
crude,notuseless.Forexample,theimpactfactorcanbeusedasastartingpointinrankingjournalsin
groupsbyusingimpactfactorsinitiallytodefinethegroupsandthenemployingothercriteriatorefine
therankingandverifythatthegroupsmakesense.Butusingtheimpactfactortoevaluatejournals
requirescaution.Theimpactfactorcannotbeusedtocomparejournalsacrossdisciplines,forexample,
andonemustlookcloselyatthetypeofjournalswhenusingtheimpactfactortorankthem.Oneshould
alsopaycloseattentiontoannualvariations,especiallyforsmallerjournals,andunderstandthatsmall
differencesmaybepurelyrandomphenomena.Anditisimportanttorecognizethattheimpactfactor
maynotaccuratelyreflectthefullrangeofcitationactivityinsomedisciplines,bothbecausenotall
journalsareindexedandbecausethetimeperiodistooshort.Otherstatisticsbasedonlongerperiods
oftimeandmorejournalsmaybebetterindicatorsofquality.Finally,citationsareonlyonewayto
judgejournals,andshouldbesupplementedwithotherinformation(thecentralmessageofthisreport).

Theseareallcautionssimilartothoseonewouldmakeforanyrankingbasedonstatistics.Mindlessly
rankingjournalsaccordingtoimpactfactorsforaparticularyearisamisuseofstatistics.Toitscredit,
ThomsonScientificagreeswiththisstatementand(gently)cautionsthosewhousetheimpactfactor
aboutthesethings.

"ThomsonScientificdoesnotdependontheimpactfactoraloneinassessingtheusefulnessofa
journal,andneithershouldanyoneelse.Theimpactfactorshouldnotbeusedwithoutcareful
attentiontothemanyphenomenathatinfluencecitationrates,asforexampletheaverage
numberofreferencescitedintheaveragearticle.Theimpactfactorshouldbeusedwith
informedpeerreview."[THOMSON:IMPACTFACTOR]

Unfortunately,thisadviceistoooftenignored.

Rankingpapers
Theimpactfactorandsimilarcitationbasedstatisticscanbemisusedwhenrankingjournals,butthereis
amorefundamentalandmoreinsidiousmisuse:Usingtheimpactfactortocompareindividualpapers,
people,programs,orevendisciplines.Thisisagrowingproblemthatextendsacrossmanynationsand
manydisciplines,madeworsebyrecentnationalresearchassessments.

Inasense,thisisnotanewphenomenon.Scientistsareoftencalledupontomakejudgmentsabout
publicationrecords,andonehearscommentssuchas,"Shepublishesingoodjournals"or"Mostofhis

9
June2008 CitationStatistics IMUICIAMIMS

papersareinlowleveljournals."Thesecanbesensibleassessments:Thequalityofjournalsinwhicha
scientistgenerally(orconsistently)publishesisoneofmanyfactorsonecanusetoassessthescientist's
overallresearch.Theimpactfactor,however,hasincreasedthetendencytoascribethepropertiesofan
individualjournaltoeacharticlewithinthatjournal(andtoeachauthor).

ThomsonScientificimplicitlypromotesthispractice:

"Perhapsthemostimportantandrecentuseofimpactisintheprocessofacademicevaluation.
Theimpactfactorcanbeusedtoprovideagrossapproximationoftheprestigeofjournalsin
whichindividualshavebeenpublished."[THOMSON:IMPACTFACTOR]

Herearesomeexamplesofthewaysinwhichpeoplehaveinterpretedthisadvice,reportedfrom
mathematiciansaroundtheworld:

Example1:Myuniversityhasrecentlyintroducedanewclassificationofjournalsusingthe
ScienceCitationIndexCorejournals.Thejournalsaredividedintothreegroupsbasedonlyon
theimpactfactor.Thereare30journalsinthetoplist,containingnomathematicsjournal.The
secondlistcontains667,whichincludes21mathematicsjournals.Publicationinthefirstlist
causesuniversitysupportofresearchtotriple;publicationinthesecondlist,todouble.
Publicationinthecorelistawards15points;publicationinanyThomsonScientificcovered
journalawards10.Promotionrequiresafixedminimumnumberofpoints.

Example2:Inmycountry,universityfacultywithpermanentpositionsareevaluatedeverysix
years.Sequentialsuccessfulevaluationsarethekeytoallacademicsuccess.Inadditiontoa
curriculumvitae,thelargestfactorinevaluationconcernsrankingfivepublishedpapers.In
recentyears,thesearegiven3pointsiftheyappearinjournalsinthetopthirdoftheThomson
Scientificlist,2pointsifinthesecondthird,and1pointinthebottomthird.(Thethreelistsare
createdusingtheimpactfactor.)

Example3:Inourdepartment,eachfacultymemberisevaluatedbyaformulainvolvingthe
numberofsingleauthorequivalentpapers,multipliedbytheimpactfactorofthejournalsin
whichtheyappear.Promotionsandhiringarebasedpartlyonthisformula.

Intheseexamples,aswellasmanyothersreportedtous,theimpactfactorisbeingusedeitherexplicitly
orimplicitlytocompareindividualpapersalongwiththeirauthors:IftheimpactfactorofjournalAis
greaterthanthatofjournalB,thensurelyapaperinAmustbesuperiortoapaperinB,andauthorA
superiortoauthorB.Insomecases,thisreasoningisextendedtorankdepartmentsorevenentire
disciplines.

Ithaslongbeenknownthatthedistributionofcitationcountsforindividualpapersinajournalishighly
skewed,approximatingasocalledpowerlaw.([Seglen1996],[Garfield1987])Thishasconsequences
thatcanbemadeprecisewithanexample.

ThedistributionforpapersintheProceedingsoftheAmericanMathematicalSocietyovertheperiod
20002004canbeseenbelow.TheProceedingspublishesshortpapers,normallyshorterthantenpages
inlength.Duringthisperiod,itpublished2,381papers(about15,000pages).Using2005journalsinthe
MathReviewscitationdatabase,theaveragecitationcountperarticle(thatis,theimpactfactor)is.434.

10
June2008 CitationStatistics IMUICIAMIMS

ProceedingsoftheAMS
80%
70%
Percentofarticles 60%
50%
40%
30%
20%
10%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12
Citations


TheTransactionsoftheAMSpublisheslongerarticlesthatareusuallymoresubstantial,bothinscope
andcontent.Overthesameperiodoftime,theTransactionspublished1,165papers(morethan25,000
pages),withcitationcountsrangingfrom0to12.Theaveragenumberofcitationsperarticlewas.846
abouttwicethatoftheProceedings.

60%
TransactionsofAMS
50%
Percentofarticles

40%
30%
20%
10%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12
Citations

Nowconsidertwomathematicians,onepublishingapaperintheProceedingsandtheotherapaperin
theTransactions.Usingsomeoftheinstitutionalpracticescitedabove,thesecondwouldbejudged
superiortothefirst,publishingapaperinajournalwithhigherimpactfactorinfact,twiceashigh!Is
thisavalidassessment?ArepapersintheTransactionoftheAMStwiceasgoodasthoseinthe
Proceedings?

WhenweassertthatanindividualTransactionspaperisbetter(inthesenseofcitations)thanan
individualProceedingspaper,weneedtoasknotaquestionaboutaverages,butratheraquestion
aboutprobabilities:Whatistheprobabilitythatwearewrong?Whatistheprobabilitythatarandomly
selectedProceedingspaperhasatleastasmanycitationsasarandomlyselectedTransactionspaper?

Thisisanelementarycalculation,andtheansweris62%.Thismeansthatwearewrong62%ofthetime,
andarandomlyselectedProceedingspaperwillbejustasgoodas(orbetterthan)arandomlyselected

11
June2008 CitationStatistics IMUICIAMIMS

TransactionspaperinspiteofthefactthattheProceedingsimpactfactorisonlyhalfthatofthe
Transactions!Wearemoreoftenwrongthanright.Mostpeoplefindthissurprising,butitisa
consequenceofthehighlyskeweddistributionandthenarrowwindowoftimeusedtocomputethe
impactfactor(whichisthereasonforthehighpercentageofuncitedpapers). 5 Itshowsthevalueof
precisestatisticalthinkingratherthanintuitiveobservation.

Thisistypicalbehaviorforjournals,andthereisnothingspecialaboutthechoicesofthesetwojournals.
(Forexample,theJournaloftheAMSoverthesameperiodhasanimpactfactor2.63sixtimesthatof
theProceedings.YetarandomlyselectedProceedingsarticleisatleastasgoodasaJournalarticle,in
thesenseofcitations,32%ofthetime.)

Thus,whileitisincorrecttosaythattheimpactfactorgivesnoinformationaboutindividualpapersina
journal,theinformationissurprisinglyvagueandcanbedramaticallymisleading.

Itfollowsthatthekindsofcalculationsperformedinthethreeexamplesaboveusingtheimpactfactor
asaproxyforactualcitationcountsforindividualpapershavelittlerationalbasis.Makingassertions
thatareincorrectmorethanhalfthetime(orathirdofthetime)issurelynotagoodwaytocarryout
anassessment.

Onceonerealizesthatitmakesnosensetosubstitutethe
impactfactorforindividualarticlecitationcounts,itfollows
thatitmakesnosensetousetheimpactfactortoevaluatethe Whileitisincorrecttosaythat
authorsofthosearticles,theprogramsinwhichtheywork,and theimpactfactorgivesno
(mostcertainly)thedisciplinestheyrepresent.Theimpact
factorandaveragesingeneralaretoocrudetomakesensible
informationaboutindividual
comparisonsofthissortwithoutmoreinformation. papersinajournal,the
informationissurprisingly
Ofcourse,rankingpeopleisnotthesameasrankingtheir vagueandcanbedramatically
papers.Butifyouwanttorankaperson'spapersusingonly
citationstomeasurethequalityofaparticularpaper,youmust
misleading.
beginbycountingthatpaper'scitations.Theimpactfactorof
thejournalinwhichthepaperappearsisnotareliablesubstitute.

Rankingscientists
Whiletheimpactfactorhasbeenthebestknowncitationbasedstatistic,thereareothermorerecent
statisticsthatarenowactivelypromoted.Hereisasmallsampleofthreeofthesestatisticsmeantto
rankindividuals.

hindex:Ascientist'shindexisthelargestnforwhichhe/shehaspublishednarticles,eachwithatleast
ncitations.
Thisisthemostpopularofthestatisticsmentionedhere.ItwasproposedbyJ.E.Hirsch[Hirsch
2006]inordertomeasure"thescientificoutputofaresearcher"byfocusingonthehighend
"tail"ofaperson'scitationdistribution.Thegoalwastosubstituteasinglenumberfor
publicationscountsandcitationdistributions.

mindex:Ascientist'smindexisthehindexdividedbythenumberofyearssincehis/herfirstpaper.

12
June2008 CitationStatistics IMUICIAMIMS

ThiswasalsoproposedbyHirschinthepaperabove.Theintentionistocompensatejunior
scientistsbecausetheyhavenothadtimetopublishpapersorgainmanycitations.

gindex:Ascientist'sgindexisthelargestnforwhichthenmostcitedpapershaveatotalofatleastn2
citations.
ThiswasproposedbyLeoEgghein2006[Egghe2006].Thehindexdoesnottakeintoaccount
thefactthatsomepapersinthetopnmayhaveextraordinarilyhighcitationcounts.Thegindex
ismeanttocompensateforthis.

Therearemoreindicesmanymoreofthemincludingvariantsofthoseabovethattakeintoaccount
theageofpapersorthenumberofauthors.([BatistaCampiteliKinouchiMartinez2005],[Batista
CampiteliKinouchi2006],[SidiropoulsKatsarosManolopoulos2006])

Inhispaperdefiningthehindex,Hirschwrotethatheproposedthehindexas"aneasilycomputable
index,whichgivesanestimateoftheimportance,significance,andbroadimpactofascientist's
cumulativeresearchcontributions."[Hirsch2005,p.5]Hewentontoaddthat"thisindexmayprovidea
usefulyardsticktocomparedifferentindividualscompetingforthesameresourcewhenanimportant
evaluationcriterionisscientificachievement."

Neitheroftheseassertionsissupportedbyconvincingevidence.Tosupporthisclaimthatthehindex
measurestheimportanceandsignificanceofascientist'scumulativeresearch,Hirschanalyzestheh
indexforacollectionofNobelPrizewinners(and,separately,membersoftheNationalAcademy).He
demonstratesthatpeopleinthesegroupsgenerallyhavehighhindices.Onecanconcludethatitis
likelyascientisthasahighhindexgiventhescientistisaNobelLaureate.Butwithoutfurther
information,weknowverylittleaboutthelikelihoodsomeonewillbecomeaNobelLaureateora
memberoftheNationalAcademy,giventhattheyhaveahighhindex.Thatisthekindofinformation
onewantsinordertoestablishthevalidityofthehindex.

Inhisarticle,Hirschalsoclaimsthatonecanusethehindextocomparetwoscientists:

"Iarguethattwoindividualswithsimilarharecomparableintermsoftheiroverallscientific
impact,eveniftheirtotalnumberofpapersortheirtotalnumberofcitationsisverydifferent.
Conversely,thatbetweentwoindividuals(ofthesamescientificage)withsimilarnumberof
totalpapersoroftotalcitationcountandverydifferenthvalue,theonewiththehigherhis
likelytobethemoreaccomplishedscientist."[Hirsch2005,p.1]

Theseassertionsappeartoberefutedbycommonsense.(Thinkoftwoscientists,eachwith10papers
with10citations,butonewithanadditional90paperswith9citationseach;orsupposeonehasexactly
10papersof10citationsandtheotherexactly10papersof100each.Wouldanyonethinkthem
equivalent?)6

Hirschextolsthevirtuesofthehindexbyclaimingthat"hispreferabletoothersinglenumbercriteria
commonlyusedtoevaluatescientificoutputofaresearcher"[Hirsch2005,p.1],butheneitherdefines
"preferable"norexplainswhyonewantstofind"singlenumbercriteria."

Whiletherehasbeensomecriticismofthisapproach,therehasbeenlittleseriousanalysis.Muchofthe
analysisconsistsofshowing"convergentvalidity,"thatis,thehindexcorrelateswellwithother

13
June2008 CitationStatistics IMUICIAMIMS

publication/citationmetrics,suchasthenumberofpublishedpapersorthetotalnumberofcitations.
Thiscorrelationisunremarkable,sinceallthesevariablesarefunctionsofthesamebasicphenomenon
publications.Inonenotablepaperaboutthehindex[LehmannJacksonLautrup2006]theauthors
carryoutamorecarefulanalysisanddemonstratethatthehindex(actually,themindex)isnotas
"good"asmerelyconsideringthemeannumberofcitationsperpaper.Evenhere,however,theauthors
donotadequatelydefinewhattheterm"good"means.Whentheclassicalstatisticalparadigmis
applied[LehmannJacksonLautrup2006],thehindexprovestobelessreliablethanothermeasures.

Anumberofvariantsofthehindexhavebeendevisedto
comparethequalityofresearchersnotonlywithina
disciplinebutacrossdisciplinesaswell.([BatistaCampiteli
Kinouchi2006],[MolinariMolinari2008])Othersclaimthat
Understandingoughttobethe
thehindexcanbeusedtocompareinstitutesand goalwhenassessingresearch,
departments.[Kinney2007]Theseareoftenbreathtakingly notmerelyensuringthatany
naveattemptstocaptureacomplexcitationrecordwitha twopeoplearecomparable.
singlenumber.Indeed,theprimaryadvantageofthesenew
indicesoversimplehistogramsofcitationcountsisthatthe
indicesdiscardalmostallthedetailofcitationrecords,and
thismakesitpossibletorankanytwoscientists.Evensimpleexamples,however,showthatthe
discardedinformationisneededtounderstandaresearchrecord.Surelyunderstandingoughttobethe
goalwhenassessingresearch,notmerelyensuringthatanytwopeoplearecomparable.

Insomecases,nationalassessmentbodiesaregatheringthehindexoroneofitsvariantsaspartof
theirdata.Thisisamisuseofthedata.Unfortunately,havingasinglenumbertorankeachscientistisa
seductivenotiononethatmayspreadmorebroadlytoapublicthatoftenmisunderstandstheproper
useofstatisticalreasoninginfarsimplersettings.

Themeaningofcitations
Thosewhopromotecitationstatisticsasthepredominantmeasureofresearchqualitydonotanswer
theessentialquestion:Whatdocitationsmean?Theygatherlargeamountsofdataaboutcitation
counts,processthedatainordertoderivestatistics,andthenasserttheresultingassessmentprocessis
"objective".Yetitistheinterpretationofthestatisticsthatleadstoassessment,andtheinterpretation
reliesonthemeaningofcitations,whichisquitesubjective.

Intheliteraturepromotingthisapproach,itissurprisinglydifficulttofindclearstatementsaboutthe
meaningofcitations.

"Theconceptbehindcitationindexingisfundamentallysimple.Byrecognizingthatthevalueof
informationisdeterminedbythosewhouseit,whatbetterwaytomeasurethequalityofthe
workthanbymeasuringtheimpactitmakesonthecommunityatlarge.Thewidestpossible
populationwithinthescholarlycommunity(i.e.anyonewhousesorcitesthesourcematerial)
determinestheinfluenceorimpactoftheideaanditsoriginatoronourbodyofknowledge."
[THOMSON:HISTORY]

14
June2008 CitationStatistics IMUICIAMIMS

"Althoughquantifyingthequalityofindividualscientistsisdifficult,thegeneralviewisthatitis
bettertopublishmorethanlessandthatthecitationcountofapaper(relativetocitationhabits
inthefield)isausefulmeasureofitquality."[LehmanJacksonLautrup2006,p.1003]

"Citationfrequencyreflectsajournal'svalueandtheusemadeofit"[Garfield1972,p.535]

"Whenaphysicianorabiomedicalresearchercitesajournalarticle,itindicatesthatthecited
journalhasinfluencedhimorherinsomemanner."[Garfield1987,p.7]

"Citationsareanacknowledgementofintellectualdebt."[THOMSON:FIFTYYEARS]

Therelevanttermsare"quality","value","influence",and"intellectualdebt".Theterm"impact"has
becomethegenericwordusedtoassignmeaningtocitationsatermthatfirstaroseinashortpaper
writtenin1955byEugeneGarfieldtopromotetheideaofcreatingacitationindex.Hewrote:

"Thus,inthecaseofahighlysignificantarticle,thecitationindexhasaquantitativevalue,forit
mayhelpthehistoriantomeasuretheinfluenceofthearticlethatis,its'impactfactor.'"
[Garfield1955,p.3]

Itisfairlyclearthathere,aselsewhere,theterm"impactfactor"isintendedtosuggestthattheciting
paperhasbeen"builtupon"theworkofthecitedthatcitationsarethemechanismbywhichresearch
propagatesitselfforward.

Thereisarichliteratureabouttheactualmeaningofcitationsthatsuggestscitationsaremore
complicatedthanthesevaguestatementsleadustobelieve.Forexample,intheir1983paperon
assessingresearch,MartinandIrvinewrite:

"Underlyingalltheseproblemswiththeuseofcitationsasameasureofqualityisourignorance
ofthereasonswhyauthorsciteparticularpiecesofworkandnotothers.Theproblems
describedabove...Simplecitationanalysispresupposesahighlyrationalmodelofreference
giving,inwhichcitationsareheldtoreflectprimarilyscientificappreciationofpreviousworkof
highqualityorimportance,andpotentialcitersallhavethesamechancetociteparticular
papers..."[MartinIrvine1983,p.69]

Inher1988paperonthemeaningofcitations[Cozzens1989],Cozzensassertsthatcitationsarethe
resultoftwosystemsunderlyingtheconductofscientificpublication,onea"reward"systemandthe
other"rhetorical."Thefirstkindhavethemeaningmostoften
associatedwithacitationanacknowledgmentthattheciting
paperhas"intellectualdebt"tothecited.Thesecond, Themeaningofcitationsisnot
however,haveameaningquitedifferentareferencetoa simpleandcitationbased
previouspaperthatexplainssomeresult,perhapsnotaresult statisticsarenotnearlyas
ofthecitedauthoratall.Suchrhetoricalcitationsaremerelya
waytocarryonascientificconversation,notestablish
"objective"asproponents
intellectualindebtedness.Ofcourse,insomecases,acitation assert.
canhavebothmeanings.

15
June2008 CitationStatistics IMUICIAMIMS

Cozzensmakestheobservationthatmostcitationsarerhetorical.Thisisconfirmedbytheexperienceof
mostpracticingmathematicians.(IntheMathReviewscitationsdatabase,forexample,nearly30%of
themorethan3millioncitationsaretobooksandnottoresearcharticlesinjournals.)Whyisthis
important?Becauseunlike"reward"citations,whichtendtorefertoseminalpapers,thechoiceof
whichpapertociterhetoricallydependsonmanyfactorstheprestigeofthecitedauthor(the"halo"
effect),therelationshipofthecitingandcitedauthors,theavailabilityofthejournal(Areopenaccess
journalsmorelikelytobecited?),theconvenienceofreferencingseveralresultsfromasinglepaper,
andsoforth.Fewofthesefactorsaredirectlyrelatedtothe"quality"ofthecitedpaper.

Evenwhencitationsare"reward"citations,theycanreflectavarietyofmotives,including"currency,
negativecredit,operationalInformation,persuasiveness,positivecredit,readeralert,andsocial
consensus".[Brooks1996]Inmostcases,citationsweremotivatedbymorethanoneofthese.Some
notableresultscansufferthe"obliteration"effect,immediatelybeingincorporatedintotheworkof
others,whichthenservesasthebasisforfurthercitations.Othercitationsarenotrewardsfor
outstandingresearch,butratherwarningsaboutflawedresultsorthinking.Thepresentreportprovides
manyexamplesofsuch"warning"citations.

Thesociologyofcitationsisacomplexsubjectonethatisbeyondthescopeofthisreport.Eventhis
cursorydiscussion,however,showsthatthemeaningofcitationsisnotsimpleandthatcitationbased
statisticsarenotnearlyas"objective"asproponentsassert.

Somemightarguethatthemeaningofcitationsisimmaterialbecausecitationbasedstatisticsarehighly
correlatedwithsomeothermeasureofresearchquality(suchaspeerreview).Forexample,the
Evidencereportmentionedearlierarguesthatcitationstatisticscan(andshould)replaceotherformsof
evaluationbecauseofthiscorrelation:

"Evidencehasarguedthatbibliometrictechniquescancreateindicatorsofresearchqualitythat
arecongruentwithresearcherperception."[EvidenceReport2007,p.9]

Theconclusionseemstobethatcitationbasedstatistics,regardlessoftheirprecisemeaning,should
replaceothermethodsofassessment,becausetheyoftenagreewiththem.Asidefromthecircularityof
thisargument,thefallacyofsuchreasoningiseasytosee.

Usingstatisticswisely
Thezealousoverrelianceonobjectivemetrics(statistics)toassessresearchisneitheranewnoran
isolatedphenomenon.Itiseloquentlydescribedinthe2001popularbook,Damnedliesandstatistics,
writtenbythesociologistJoelBest:

"Thereareculturesinwhichpeoplebelievethatsomeobjectshavemagicalpowers;
anthropologistscalltheseobjectsfetishes.Inoursociety,statisticsareasortoffetish.Wetend
toregardstatisticsasthoughtheyaremagical,asthoughtheyaremorethanmerenumbers.
Wetreatthemaspowerfulrepresentationsofthetruth;weactasthoughtheydistillthe
complexityandconfusionofrealityintosimplefacts.Weusestatisticstoconvertcomplicated
socialproblemsintomoreeasilyunderstoodestimates,percentages,andrates.Statisticsdirect
ourconcern;theyshowuswhatweoughttoworryaboutandhowmuchweoughttoworry.In
asense,thesocialproblembecomesthestatisticand,becausewetreatstatisticsastrueand

16
June2008 CitationStatistics IMUICIAMIMS

incontrovertible,theyachieveakindoffetishlike,magicalcontroloverhowweviewsocial
problems.Wethinkofstatisticsasfactsthatwediscover,notnumberswecreate."[Best2001,
p160]

Thismysticalbeliefinthemagicofcitationstatisticscanbefoundthroughoutthedocumentationfor
researchassessmentexercises,bothnationalandinstitutional.Itcanalsobefoundintheworkofthose
promotingthehindexanditsvariants.

Thisattitudeisalsoevidentinrecentattemptstoimproveontheimpactfactorusingmoresophisticated
mathematicalalgorithms,includingpagerankalgorithms,toanalyzecitations.([Bergstrom2007],
[StringerSalesPardoNunes2008])Theirproponentsmakeclaimsabouttheirefficacythatare
unjustifiedbytheanalysisanddifficulttoassess.Becausetheyarebasedonmorecomplicated
calculations,the(oftenhidden)assumptionsbehindthemarenoteasyformostpeopletodiscern. 7 We
aremeanttotreatthenumbersandrankingswithaweastruthsratherthancreations.

Researchisnotthefirstpubliclyfundedactivitytocomeunder
scrutiny,andoverthepastdecadespeoplehavetriedtocarry
outquantitativeperformanceassessmentsofeverythingfrom
educationalsystems(schools)tohealthcare(hospitalsand Ifoneconsultswithdoctors
evenindividualsurgeons).Insomecases,statisticianshave whenpracticingmedicine,
steppedintoadvisethosedoingthemeasuringaboutsensible surelyoneoughttoconsult
metricsandtheproperuseofstatistics.Ifoneconsultswith
withstatisticianswhen
doctorswhenpracticingmedicine,surelyoneoughttoconsult
with(andheedtheadviceof)statisticianswhenpracticing practicingstatistics.
statistics.Twoexcellentexamplescanbefoundin[Bird2005]
and[GoldsteinSpiegelhalter1996].Whiletheyeachdealwith
performanceassessmentofthingsotherthanresearchpublicsectorperformancemonitoringinthe
firstandhealthcare/educationinthesecondeachprovidesinsightaboutthesensibleuseofstatistics
inassessingresearch.

ThepaperbyGoldsteinandSpiegelhalterinparticulardealswiththeuseofLeagueTables(rankings)
basedonsimplenumbers(forexample,studentachievementsormedicaloutcomes),anditis
particularlyrelevanttoassessingresearchbyrankingjournals,papers,orauthorsusingcitation
statistics.Intheirpaper,theauthorsoutlineathreepartframeworkforanyperformanceassessment:

Data
"Noamountoffancystatisticalfootworkwillovercomebasicinadequaciesineitherthe
appropriatenessortheintegrityofthedatacollected."[GoldsteinSpiegelhalter1996,p.389]

Thisisanimportantobservationforcitationbasedperformanceassessment.Theimpactfactor,for
example,isbasedonasubsetofdata,whichincludesonlythosejournalsselectedbyThomsonScientific.
(Wenotethattheimpactfactoritselfisthemajorpartoftheselectioncriterion.)Somehavequestioned
theintegrityofthisdata[RossnerVanEppsHill2007].Otherspointoutthatotherdatasetsmightbe
morecomplete.[MehoYang2007]SeveralgroupshavepushedtheideaofusingGoogleScholarto
implementcitationbasedstatistics,suchasthehindex,butthedatacontainedinGoogleScholaris
ofteninaccurate(sincethingslikeauthornamesareautomaticallyextractedfromwebpostings).
Citationstatisticsforindividualscientistsaresometimesdifficulttoobtainbecauseauthorsarenot

17
June2008 CitationStatistics IMUICIAMIMS

uniquelyidentified,andinsomesettingsandcertaincountries,thiscanbeanenormousimpedimentto
assemblingaccuratecitationdata.Theparticularcollectionofdataoneusesforcitationanalysisis
frequentlyoverlooked.Oneislikelytodrawfaultyconclusionsfromstatisticsbasedonfaultydata.

StatisticalAnalysisandPresentation
"Weshallpayparticularattentiontothespecificationofanappropriatestatisticalmodel,the
crucialimportanceofuncertaintyinthepresentationofallresults,techniquesforadjustmentof
outcomesforconfoundingfactorsandfinallytheextenttowhichanyreliancemaybeplacedon
explicitrankings."[GoldsteinSpiegelhalter1996,p.390]

Aswehavewrittenpreviously,inmostcasesinwhichcitationstatisticsareusedtorankpapers,people,
andprograms,nospecificmodelisspecifiedinadvance.Instead,thedataitselfsuggestsamodel,which
isoftenvague.Acircularprocessseemstorankobjectshigherbecausetheyarerankedhigher(inthe
database).Thereisfrequentlyscantattentiontouncertaintyinanyoftheserankings,andlittleanalysis
ofhowthatuncertainty(forexample,annualvariationsintheimpactfactor)wouldaffecttherankings.
Finally,confoundingfactors(forexample,theparticulardiscipline,thetypeofarticlesajournal
publishes,whetheraparticularscientistisanexperimentalistortheoretician)arefrequentlyignoredin
suchrankings,especiallywhencarriedoutinnationalperformanceassessments.

InterpretationandImpact
"Thecomparisonsdiscussedinthispaperareofgreatpublicinterest,andthisisclearlyanarea
wherecarefulattentiontolimitationsisbothvitalandlikelytobeignored.Whetheradjusted
outcomesareinanywayvalidmeasuresofinstitutional'quality'isoneissue,whileanalysts
shouldalsobeawareofthepotentialeffectoftheresultsintermsoffuturebehavioural
changesbyinstitutionsandindividualsseekingtoimprovetheirsubsequent'ranking'."
[GoldsteinSpiegelhalter1996,p.390]

Theassessmentofresearchisalsoofgreatpublicinterest.Foranindividualscientist,anassessmentcan
haveprofoundandlongtermeffectsonone'scareer;foradepartment,itcanchangeprospectsfor
successfarintothefuture;fordisciplines,acollectionofassessmentscanmakethedifferencebetween
thrivingandlanguishing.Foratasksoimportant,surelyoneshouldunderstandboththevalidityandthe
limitationsofthetoolsbeingusedtocarryitout.Towhatextentdocitationsmeasurethequalityof
research?Citationcountsseemtobecorrelatedwithquality,andthereisanintuitiveunderstanding
thathighqualityarticlesarehighlycited.Butasexplainedabove,somearticles,especiallyinsome
disciplines,arehighlycitedforreasonsotherthanhighquality,anditdoesnotfollowthathighlycited
articlesarenecessarilyhighquality.Thepreciseinterpretationofrankingsbasedoncitationstatistics
needstobebetterunderstood.Inaddition,ifcitationstatisticsplayacentralroleinresearch
assessment,itisclearthatauthors,editors,andevenpublisherswillfindwaystomanipulatethesystem
totheiradvantage.[MacdonaldKam2007]Thelongtermimplicationsofthisareunclearand
unstudied.

ThearticlebyGoldsteinandSpiegelhalterisvaluabletoreadtodaybecauseitmakesclearthattheover
relianceonsimplemindedstatisticsinresearchassessmentisnotanisolatedproblem.Governments,
institutions,andindividualshavestruggledwithsimilarproblemsinthepastinothercontexts,andthey
havefoundwaystobetterunderstandthestatisticaltoolsandtoaugmentthemwithothermeansof
assessment.GoldsteinandSpiegelhalterendtheirpaperwithapositivestatementofhope:

18
June2008 CitationStatistics IMUICIAMIMS

"Finally,althoughwehavebeengenerallycriticalofmanycurrentattemptstoprovide
judgmentsaboutinstitutions,wedonotwishtogivetheimpressionthatwebelievethatall
suchcomparisonsarenecessarilyflawed.Itseemstousthatthecomparisonofinstitutionsand
theattempttounderstandwhyinstitutionsdifferisanextremelyimportantactivityandisbest
carriedoutinaspiritofcollaborationratherthanconfrontation.Itisperhapstheonlysure
methodforobtainingobjectivelybasedinformationwhichcanleadtounderstandingand
ultimatelyresultinimprovements.Therealproblemwiththesimplisticprocedureswhichwe
havesetouttocriticizeisthattheydistractbothattentionandresourcesfromthisworthier
aim."[GoldsteinSpiegelhalter1996,p.406]

Itwouldbehardtofindabetterstatementtoexpressthegoalsthatshouldbesharedbyeveryone
involvedintheassessmentofresearch.

JointIMU/ICIAM/IMSCommitteeonQuantitativeAssessmentofResearch
RobertAdler,TechnionIsraelInstituteofTechnology
JohnEwing(Chair),AmericanMathematicalSociety
PeterTaylor,UniversityofMelbourne

19
June2008 CitationStatistics IMUICIAMIMS

References

Adler,Robert.2007.Theimpactofimpactfactors.IMSBulletin,Vol.36,No.5,p.4.
http://bulletin.imstat.org/pdf/36/5

Amin,M.;Mabe,M.2000.Impactfactor:useandabuse.PerspectivesinPublishing,No.1,October,pp.
16.
http://www.elsevier.com/framework_editors/pdfs/Perspectives1.pdf

Batista,PabloDiniz;Campiteli,MonicaGuimaraes;Kinouchi,Osame;Martinez,AlexandreSouto.2005.
Universalbehaviorofaresearchproductivityindex.arXiv:physics,v1,pp.15.
arXiv:physics/0510142v1

Batista,PabloDiniz;Campiteli,MonicaGuimaraes;Kinouchi,Osame;.2006.Isitpossibletocompare
researcherswithdifferentscientificinterests?.Scientometrics,Vol68,No1,pp.179189.
http://dx.doi.org/10.1007/s1119200600904

Bergstrom,Carl.Eigenfactor:measuringthevalueandpresitigeofscholarlyjournals.College&Research
LibrariesNews,Vol68,No.5,May2007
http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/may07/eigenfactor.cfm
(Seealsohttp://www.eigenfactor.org/methods.pdf.)

Best,Joel.2001.Damnedliesandstatistics:untanglingthenumbersfromthemedia,politicians,and
activists.UniversityofCaliforniaPress,Berkeley.

Bird,Sheila;etal.2005.Performanceindicators:good,bad,andugly;Reportofaworkingpartyon
performancemonitoringinthepublicservices.J.R.Statist.Soc.A(2005),168,Part1,pp.127.
http://dx.doi.org/10.1111/j.1467-985X.2004.00333.x

Brooks,Terrence.1986.Evidenceofcomplexcitermotivations.JournaloftheAmericanSocietyfor
InformationScience,Vol37,No.1,pp.3436,1986.
http://dx.doi.org/10.1002/asi.4630370106

Carey,AlanL.;Cowling,MichaelG.;Taylor,PeterG.2007.Assessingresearchinthemathematical
sciences.GazetteoftheAustralianMathSociety,A.L.Carey,Vol.34,No.2,May,pp.8489.
http://www.austms.org.au/Publ/Gazette/2007/May07/084CommsCarey.pdf

Cozzens,SusanE.1989.Whatdocitationscount?Therhetoricfirstmodel.Scientometrics,Vol15,Nos5
6,(1989),pp.437447.
http://dx.doi.org/10.1007/BF02017064

Egghe,Leo.2006.Theoryandpracticeofthegindex.Scientometrics,vol.69,No1,pp.131152.
http://dx.doi.org/10.1007/s11192-006-0144-7

EvidenceReport.2007.TheuseofbibliometricstomeasureresearchqualityintheUKhighereducation
system.(AreportproducedfortheResearchPolicyCommitteeofUniversities,UK,byEvidenceLtd.,a

20
June2008 CitationStatistics IMUICIAMIMS

companyspecializinginresearchperformanceanalysisandinterpretation.EvidenceLtd.has"strategic
alliance"withThomsonScientific.)
http://bookshop.universitiesuk.ac.uk/downloads/bibliometrics.pdf

Ewing,John.2006.Measuringjournals.NoticesoftheAMS,vol.53,no.9,pp.10491053.
http://www.ams.org/notices/200609/commewing.pdf

Garfield,Eugene.1955.Citationindexesforscience:Anewdimensionindocumentationthrough
associationofideas."Science,122(3159),p.10811,July1955.
http://garfield.library.upenn.edu/papers/science1955.pdf

______.1972.Citationanalysisasatoolinjournalevaluation.Science,178(4060),pp.471479,1972.
http://www.garfield.library.upenn.edu/essays/V1p527y196273.pdf

______.1987.Whyaretheimpactsoftheleadingmedicaljournalssosimilarandyetsodifferent?
CurrentComments#2,p.3,January12,1987.
http://www.garfield.library.upenn.edu/essays/v10p007y1987.pdf

______. 1998.Long-term vs. short-term journal impact (part II). The Scientist
12(14):12-3 (July 6, 1998).
http://garfield.library.upenn.edu/commentaries/tsv12(14)p12y19980706.pdf

______.2005.Agonyandtheecstasythehistoryandmeaningofthejournalimpactfactor.Presented
attheInternationalCongressonPeerReviewandBibliomedicalPublication,Chicago,September16,
2005.
http://garfield.library.upenn.edu/papers/jifchicago2005.pdf

Goldstein,Harvey;Spiegelhalter,DavidJ.1996.Leaguetablesandtheirlimitations:Statisticalissuesin
comparisonsofinstitutionalperformance.J.R.Statist.Soc.A,159,No.3.(1996),pp385443.
http://links.jstor.org/sici?sici=09641998%281996%29159%3A3%3C385%3ALTATLS%3E2.0.CO%3B25
http://dx.doi.org/10.2307/2983325

Hall,Peter.2007.MeasuringresearchperformanceinthemathematicalsciencesinAustralian
universities.TheAustralianMathematicalSocietyGazette,Vol.34,No.1,pp.2630.
http://www.austms.org.au/Publ/Gazette/2007/Mar07/26HallMeasuring.pdf

Hirsch,J.E.2006.Anindextoquantifyanindividual'sscientificresearchoutput.ProcNatlAcadSciUSA,
Vol.102,No.46,pp.1656916573.
http://dx.doi.org/10.1073/pnas.0507655102

Kinney,A.L.2007.Nationalscientificfacilitiesandtheirscienceimpactonnonbiomedicalresearch.Proc
NatlAcadSciUSA,Vol.104,No.46,pp.1794317947.
http://dx.doi.org/10.1073/pnas.0704416104

Lehmann,Sune;Jackson,AndrewD.;Lautrup,BennyE.2006.Measuresformeasures,Nature,Vol444,
No.21,pp.10031004.
http://www.nature.com/nature/journal/v444/n7122/full/4441003a.html

21
June2008 CitationStatistics IMUICIAMIMS

Macdonald,Stuart;Kam,Jacqueline.2007.Aardvarketal.:qualityjournalsandgamesmanshipin
managementstudies.JournalofInformationScience,Vol.33,pp.702717.
http://dx.doi.org/10.1177/0165551507077419

Martin,BenR.1996.Theuseofmultipleindicatorsintheassessmentofbasicresearch,Scientometrics,
Vol36,No.3(1996),pp.343362.
http://dx.doi.org/10.1007/BF02129599

Martin,BenR.,Irvine,John.1983.Assessingbasicresearch.ResearchPolicy,Vol12(1983),pp.6190.
http://dx.doi.org/10.1016/00487333(83)900057

Meho,Lokman;Yang,Kiduk.2007.ImpactofdatasourcesoncitationcountsandrankingsofLISfaculty:
WebofSciencevs.ScopusandGoogleScholar.JournaloftheAmericanSocietyforInformationScience
andTechnology,Vol58,No13,pp.21052125.
http://dx.doi.org/10.1002/asi.20677

Molinari,J.F.,Molinari,A.2008.Anewmethodologyforrankingscientificinstitutions.Toappearin
Scientometrics.
http://imechanica.org/files/paper.pdf

Monastersky, R. 2005.Thenumberthatsdevouringscience.ChronicleHigherEd.Vol. 52, No. 8.

http://chronicle.com/free/v52/i08/08a01201.htm

Rossner,Mike;VanEpps,Heather;Hill,Emma.2007.Showmethedata.JournalofCellBiology,Vol179,
No6,December17,pp.10911092.
http://dx.doi.org/10.1083/jcb.200711140

Seglen,P.O.1997.Whytheimpactfactorforjournalsshouldnotbeusedforevaluatingresearch;BMJ,
314:497 (15 February).
http://www.bmj.com/cgi/content/full/314/7079/497

Sidiropoulos,Antonis;Katsaros,Dimitrios;Manolopoulos,Yannis.2006.Generalizedhindexfor
disclosinglatentfactsincitationnetworks.V1,arXiv:cs.
arXiv:cs/0607066v1[cs.DL]

StringerMJ,SalesPardoM,NunesAmaralLA(2008)Effectivenessofjournalrankingschemesasatool
forlocatinginformation.PLoSONE3(2):e1683
http://dx.doi.org/10.1371/journal.pone.0001683

THOMSON:JOURNALCITATIONREPORTS.2007.(ThomsonScientificwebsite)
http://scientific.thomson.com/products/jcr/

THOMSON:SELECTION.2007. (ThomsonScientificwebsite)
http://scientific.thomson.com/free/essays/selectionofmaterial/journalselection/

THOMSON:IMPACTFACTOR(ThomsonScientificwebsite)
http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor/

22
June2008 CitationStatistics IMUICIAMIMS

THOMSON:HISTORY(ThomsonScientificwebsite)
http://scientific.thomson.com/free/essays/citationindexing/history/

THOMSON:FIFTYYEARS(ThomsonScientificwebsite)
http://scientific.thomson.com/free/essays/citationindexing/50ycitationindexing/

23
June2008 CitationStatistics IMUICIAMIMS

Endnotes
1
ThisquotewasattributedtoEinsteinintheReader'sDigest.Oct.1977.Itappearstobederivedfrom
hisactualquote:"Itcanscarcelybedeniedthatthesupremegoalofalltheoryistomaketheirreducible
basicelementsassimpleandasfewaspossiblewithouthavingtosurrendertheadequate
representationofasingledatumofexperience."From"OntheMethodofTheoreticalPhysics"The
HerbertSpencerLecture,deliveredatOxford(10June1933);alsopublishedinPhilosophyofScience,
Vol.1,No.2(April1934),pp.163169.
2
WhileweconcentrateontheThomsonScientificimpactfactorinthissection,wenotethatThomson
promotestheuseoftwootherstatistics.Also,similarstatisticsbasedonaveragecitationcountsfor
journalscanbederivedfromotherdatabases,includingScopus,Spires,GoogleScholar,and(for
mathematics)theMathReviewscitationdatabase.Thelatterconsistsofcitationsfromover400
mathematicsjournalsfromtheperiod2000present,identifiedasitemsthatwerelistedinMath
Reviewssince1940;itincludesmorethan3millioncitations.
3
ThomsonScientificindicates(March2008)thatitindexesjournalsinthefollowingcategories:
MATHEMATICS(217)
MATHEMATICSAPPLIED(177)
MATHEMATICSINTERDISCIPLINARY(76)
PHYSICS,MATHEMATICAL(44)
PROBABILITYANDSTATISTICS(96)
Thecategoriesoverlap,andthetotalnumberofjournalsisapproximately400.Bycontrast,
MathematicalReviewsincludesitemsfromwellmorethan1200journalseachyear,andconsidersmore
than800journalsas"core"(inthesensethateveryiteminthejournalisincludedinMathReviews).
Zentralblattcoversasimilarnumberofmathematicsjournals.

4
TheMathematicalReviewscitationdatabaseincludes(March2008)morethan3millionreferencesin
approximately400journalspublishedfrom2000tothepresent.Thereferencesarematchedtoitemsin
theMRdatabaseandextendovermanydecades.UnliketheScienceCitationIndex,citationsbothto
booksandjournalsareincluded.Itisacuriousfactthatroughly50%ofthecitationsaretoitems
appearinginthepreviousdecade;25%citearticlesappearinginthedecadebeforethat;12.5%cite
articlesinthepriordecade;andsoon.Thissortofbehaviorisspecialtoeachdiscipline,ofcourse.
5
Theskeweddistributioncombinedwiththenarrowwindow(usingonlyoneyear'sjournalsasthe
sourceofcitationsandfiveyearsasthetarget)meansthatalargenumberofarticleshaveeithernone
orveryfewcitations.Thismakesitintuitivelyobviousthatrandomlychosenarticlesareoften
equivalent.

Thefactthatmanyarticleshavenocitations(oronlyafew)isalsoaconsequenceofthelongcitation
timeformathematicsarticlesoftentakemanyyearstoaccumulatecitations.Ifwechooselonger
periodsoftimeforbothsourcejournalsandtargetyears,thenthecitationcountsincreasesubstantially
anditbecomeseasiertodistinguishjournalsbycitationbehavior.Thisistheapproachusedin[Stringer
etal.2008]toanalyzecitations.Theyshowthatforsufficientlylongperiodsoftime,thedistributionof
citationcountsforindividualarticlesappearstobelognormal.Thisprovidesamechanismfor
comparingtwojournalsbycomparingthedistributions,andiscertainlymoresophisticatedthanusing
impactfactors.Again,however,itconsidersonlycitationsandnothingelse.

24
June2008 CitationStatistics IMUICIAMIMS

6
Toillustratehowmuchinformationoneloseswhenusingonlythehindex,hereisareallifeexample
ofadistinguishedmidcareermathematicianwhohaspublished84researchpapers.Thecitation
distributionlookslikethefollowing:

Scientist'sCitationRecord
(84articles)
30%
25%
Percentofarticles

20%
15%
10%
5%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 >15
Citations

Noticethatslightlyunder20%ofthepublicationshave15ormorecitations.Thedistributionofactual
citationcountsforthese15papersis:

Top15papers
200
Numberofcitations

150

100

50

0
0 5 10 15 20
Papers

InHirsch'sanalysis,however,allthisinformationisthrownaway.Oneonlyremembersthatthehindex
is15,meaningthatthetop15papershave15ormorecitations.

7
Thealgorithmin[Bergstrom2007]usesapagerankalgorithmtogiveeachcitationaweight,andthen
computesan"impactfactor"byusingtheweightedaveragesforcitations.Pagerankalgorithmshave
meritbecausetheytakeintoaccountthe"value"ofcitations.Ontheotherhand,theircomplexitycan
bedangerousbecausethefinalresultsarehardertounderstand.Inthiscase,all"selfcitations"are
discardedthatis,allcitationsfromarticlesinagivenjournalJtoarticlespublishedinJduringthe
precedingfiveyearsarediscarded.Thesearenot"selfcitations"inanynormalsenseoftheword,anda

25
June2008 CitationStatistics IMUICIAMIMS

glanceatsomedatafromtheMathReviewsCitationsdatabasesuggeststhatthisdiscardsroughlyone
thirdofallcitations.

Thealgorithmin[Stringeretal.2008]isinteresting,inpartbecauseitattemptstoaddressthediffering
timescalesforcitationsaswellastheissueofcomparingrandomlyselectedpapersinonejournalwith
thosefromanother.Again,thecomplexityofthealgorithmsmakesithardformostpeopletoevaluate
theirresults.Onenotablehypothesisisslippedintothepaperonpage2:"Ourfirstassumptionisthat
thepaperspublishedinjournalJhaveanormaldistributionof'quality'..."Thisseemstocontradict
commonexperience.

26