Journal of Public Administration Research and Theory Advance Access published April 28, 2016

Journal of Public Administration Research And Theory, 2016, 1–13

doi:10.1093/jopart/muw028
Article

Article

Relative Performance Information and

Perceptions of Public Service Quality: Evidence
From American School Districts
Samuel Barrows,* Michael Henderson,† Paul E. Peterson,‡
Martin R. West§

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

*
Harvard Kennedy School of Government; †Louisiana State University; ‡Harvard University; §Harvard Graduate School
of Education

Abstract
The publication of information about public service performance has expanded dramatically in
recent decades. Multiple experiments have shown that relative performance information influ-
ences citizens’ evaluations of local services. It is unclear whether this information leads citizens
to update their beliefs or temporarily employ different criteria, however, and we do not know if
the influence of performance information will extend to contexts in which citizens have access to
other information sources. We examine these questions using two experiments within nationally
representative surveys. In the first, we find that providing respondents with information about the
performance of their local schools relative to schools in the state, the nation, and other developed
countries depresses average evaluations of local school quality.The second experiment shows
that relative performance information depresses average evaluations in part by priming
respondents to consider how the academic performance of their communities’ schools
compares to others, but also by leading some respondents with erroneous prior beliefs about
school performance to update their beliefs.

Introduction by voicing their concerns (Baekgaard 2015; Chingos,

Recent decades have seen a dramatic expansion in the
publication of information about the performance of
public services across many policy domains and coun-
tries (Fung 2008; Linn, Nagler, and Morales 2010;
Stecher 2010; Walker 2014). Underpinning these
reforms is the assumption that performance infor-
mation will influence citizens’ perceptions of public
services and lead them to pressure service providers
to improve, either through their service choices or
We thank the Program on Education Policy and Governance, located
within the Harvard Kennedy School’s Taubman Center on State and
Local Government, for facilitating the collection of the survey data used
for this research. As survey director for the 2013 Education Next
Survey, William G. Howell contributed to the design of the
experiments here. Antonio Wendland provided valuable administrative
support.
Address correspondence to the author
samuel_barrows@hks.harvard.edu.
Henderson, and West 2012; Fung, Graham, and Weil
2008; James and Moseley 2014). Although scholars
of public administration have long been concerned
with understanding the effects of performance infor-
mation on citizens’ perceptions and behavior, for
many decades, the topic received only modest empiri-
cal attention.
Recent years have seen a renaissance in the study
of citizens’ responses to information about public ser-
vice performance and the growing use of experimen-
tal methods to examine the impacts of performance
information. Multiple studies have examined how
citizens’ responses to information are shaped by their
prior expectations and attitudes regarding public sec-
tor organizations, as well as the source and presenta-
tional format of the information. Scholars have also
at

© The Author 2016. Published by Oxford University Press on behalf of the Journal of Public Administration Research and Theory, Inc. 1
All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
2 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx
Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 3

examined the effects of local public schools, public service Scholars have also long
different types of rather than hypothetical performance. Simon and highlighted the potential
performance services. Residents have Ridley (1938, 466–7) importance of relative
information and have access to numerous note that policymakers comparisons. Simon
consistently found that sources of information play a crucial role in (1939,
infor- mation about about public school deciding what 106) argues that “the
performance relative to performance, includ- ing information to provide only sound basis for
other localities influences their own experiences because “the voter has decisions about
people’s perceptions of and state test results, but neither the time nor the numbers is numerical
and attitudes toward rel- ative performance information to factual information
service quality in their information is difficult interpreted undigested about past experiences
own communities. to acquire. We isolate statistical tables in an or the experiences of
The extent to which learning effects by intelligent manner.” others— nothing more
the finding that relative providing information or less than comparative
per- formance about the rank of local statistics” (see also
information influences schools, rather than March and Simon 1958;
citizens’ perceptions of priming a specific Simon 1937). Yet in the

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

public services comparison, and decades since,
generalizes beyond the employing a recent remarkably little
experimen- tal settings methodo- logical empirical work has
of existing studies is innovation to separate examined the impact of
unclear, however. learning from priming either relative
Previous studies have effects. comparisons or
primed respondents to Below, we first performance
draw particular consider existing information more
comparisons, for evidence regard- ing the broadly (Olsen
example, to the average impact of comparative 2
performance of similar information about the 0
services, or have framed performance of public 1
per- formance measures services and discuss our 5
in a positive or negative expec- tations regarding a
light. We therefore do the effects of school )
not know whether performance .
simply learning about information. We then Recent years have
the relative performance describe the survey seen a new interest in
of local services will be experiments and data using experimental
suf- ficient to influence employed in our methods to identify the
citizens’ perceptions. In analysis, before effects of infor- mation.
addition, existing studies presenting our results. Researchers have
have provided We conclude by examined how the
information about considering the implica- effects of absolute
hypothetical services or tions of our findings for performance
services about which performance reporting information are shaped
citizens have little prior in both education and by citi- zens’ prior
information. Whether the public sector more expectations and
relative per- formance broadly. attitudes regarding
measures influence public sector
citizens when they have organizations
Responses (Baekgaard and
access to information
to Serritzlew 2016;
from other sources
Performanc Jacobsen, Snyder, and
remains an open
e Saultz 2015; Marvel
question.
Information 2016), as well as the
This article builds on
recent advances in the Public administration source and
litera- ture by exploring scholars have long been presentational format
the impact of providing con- cerned with of the information
US residents with understanding how (Jacobsen, Snyder, and
information about the citizens respond to Saultz 2015; James and
true relative different types of Van Ryzin 2015; Van
performance of their information about Ryzin and Lavena
 4 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx

2013). Studies have also information in the for-

begun to examine the mation of citizens’
impact of the preferences. There is
informational content of evidence, for example,
performance data and that personal accounts
have consistently found can have a greater
that information about influence on citizens’
the relative performance perceptions than
of local services affects statistical data (Olsen
people’s perceptions of 2015d).
and attitudes toward Beyond the issue of
those services external validity, the
(Baekgaard 2015; literature on relative
Charbonneau and Van performance
Ryzin 2015; James 2011; information has yet to
James and Moseley disen- tangle the
2014; Olsen 2015a). potential priming and
Although existing learning effects that may
studies point to the be at work. Learning
potential influ- ence of occurs when an
relative performance intervention provides
information, the extent previously unknown
to which this generalizes information. For exam-
to real-world settings is ple, an instance in which
unclear. Several studies a person who initially
have examined judges
hypothetical situations,
for example, by
providing respondents
with informa- tion about
a hypothetical school
(Charbonneau and Van
Ryzin 2015; Jacobsen,
Snyder, and Saultz
2015; Olsen 2015a).
Other studies have
considered low-
information
environments, such as
the proportion of waste
recycled by English
councils (James 2011;
James and Moseley
2014). In practice,
however, citizens will
often have access to
numerous sources of
information in forming
their evaluations of
public services, includ-
ing their own
experiences, media
reports, and other
sources of publicly
reported performance
informa- tion. It may
well be that these
alternative data sources
dominate relative
performance
Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 5
6 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx

a school’s quality based profound implications positive or negative light, information about the
for example, describing local
on an inaccurate for the design of public services as being in the
true performance of the
perception of the accountability systems. “best” or “worst” third of school districts in which
school’s academic If information about the services nationally they live, rather than
performance but adjusts rela- tive performance of (Baekgaard 2015; James and hypothetical schools. In
Mosely 2014). There are
his judg- ment when he local services does not order to separate the
strong theoretical and
is provided accurate lead citizens to learn and empirical grounds to expect effect of citizens learn-
information would be an update their beliefs, this such framing to mediate ing from priming effects,
example of learning. In would call into question citizens’ responses to we provide respondents
contrast, priming occurs the value of providing performance information with information about
(Olsen 2015c, 2016). We do not
when an intervention such information. At the the percentile rank of
test framing effects in this
“heightens the salience same time, if priming study. their local schools, rather
of a par- ticular specific comparisons 2 There have been numerous than a comparison to a
consideration and pushes explains the substantial scandals concerning the particular refer- ence
other relevant consid- effects observed, this manipulation of school point, and employ a
performance information, for
erations to the would likely rein- force example, in Indiana in 2012
recently developed
background of concerns about the (Associated Press 2013). methodo- logical

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

respondents’ minds in a potential for technique to separate
way that influences policymakers to learning from priming
their survey responses” manipulate performance effects.
(Schueler and West information so as to
forthcoming). In the shift residents’ judgments
context of public edu- to the most favorable Performance
cation, Andersen and basis.2 Information and
Hjortskov We build on the Public Schools
(forthcoming) have existing literature in We expect information
found that asking several respects. We about the relative
parents to recall their explore the impact of academic performance of
experiences of the information about the local schools to influence
performance of their relative performance of citizens’ eval- uations of
child’s school influences US public schools. This those schools. First,
their subsequent is an environment in citizens lack informa-
evaluations of the which citizens already tion about the relative
school. have access to a performance of local
Several previous considerable amount of schools. Although the
studies have primed performance No Child Left Behind
respondents to draw a information, but Act of 2001 requires
particular comparison information about states to publish
by providing informa- relative national information about the
tion about performance performance is limited. pro- portion of students
relative to a specific We also seek to in each school who are
reference point, for strengthen the external proficient on state tests,
example, the average validity of our findings states are not required
performance of simi- lar by providing to indicate how they
or nearby services respondents with compare to other schools.
(Charbonneau and Van Further, each state is left
Ryzin to define its own
2015; Olsen 2015a). It 1 It is important to note that
priming effects are closely standards and measures
is therefore unclear related to framing effects, but of student proficiency.
whether the effects the two are not equivalent. As a result, there is
reported in existing Framing effects occur when substantial variation in
studies are a conse- subjects are provided with
“substantively equivalent
state academic
quence of priming standards, and
descriptions of the
individuals to base their performance level but with information about how
judgments on specific varying valence, i.e. negative, local schools compare to
comparisons or of neutral, and positive” that
schools across the
learning about the rela- have differential effects on
citizens’ perceptions (Olsen country is difficult to
tive performance of local obtain (Peterson and
2015b, 3). Several previous
services.1 Yet this studies have framed Lastra-Anadon 2010).
distinction has performance information in a
 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 7

Second, an
inconsistency between 3 Although not as large,
citizens’ evalua- tions of surveys consistently reveal
local schools and similar disparities across a
schools nationally range of topics including
crime rates, environmental
suggests that people do
quality, and quality of
not evaluate schools on healthcare (Daniels et al. 2012;
the basis of accu- rate Newport 2013; Saad 2011).
knowledge about
relative academic
performance. Surveys
routinely find that when
asked to grade public
schools with the five-
point scale traditionally
used to grade students,
more people assign an
“A” or “B” to the schools
in their own community
than to schools across
the United States as a
whole. Indeed, this
evaluation gap has
widened since Gallup
began asking respond-
ents to grade schools
nationally in 1981,
reaching a high of 35
percentage points in
2013 (figure 1).3 This
gap represents a
paradox, as services
nationally are the sum of
local services, and thus
indicates that citizens
cannot be evaluating
local schools and schools
nation- ally on the basis
of accurate information
about relative academic
performance.
A number of factors,
however, could mitigate
the impact of relative
school performance
information. First,
citizens may use
dimensions of quality
other than relative
academic performance
to evaluate local schools.
Americans tend to value
multiple qualities in their
schools, including
academic quality, safety,
demo- graphic mix, and
extracurricular
activities, each of which
may be emphasized to a
greater or lesser extent
8 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx

Figure 1. Proportion of respondents in national poll grading local schools and schools nationally A or B.
Source: PDK/Gallup poll on the public’s attitudes toward the public schools, 1974–2013.
Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 9

in evaluations in their local area; for In sum, there is good c

(Hochschild and example, more than 50% reason to think that in i
Scovronick 2004; Moe of Americans report the absence of any p
a
2001). Jacobsen, having visited a public intervention citizens lack n
Snyder, and Saultz school within the past information about the t
(2015) find that when 12 months (Bushaw and relative performance of s
citizens are primed to Lopez 2012). Citizens’ their local schools and The two surveys were
consider which goals also have reason to be fail to evaluate local administered to
they believe it is most informed about their schools on the basis of nationally repre- sentative
important for schools to local schools; parents accu- rate information samples of the general
pursue and then and homeowners in about relative school population of American
provided with school particular have strong performance. We adults as well as to
performance data, their incentives to seek out therefore expect that oversamples of teachers,
subsequent evaluations information about the providing relative parents of school-age
of school quality vary quality of nearby schools perfor- mance children, African
with the goals that they (Chingos, Henderson, information will Americans, and
value most highly. and West 2012; Moe influence citizens’ Hispanics. The total

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

Second, citizens may 2001; Peterson, evaluations of their local sample sizes (including
use information other Henderson, and West schools. Whether this is general population and
than performance on 2014). the case, how- ever, oversamples) are 5,567
standardized tests to remains an open and 5,190 for 2013 and
evaluate relative empirical question. 2014, respectively. To
academic performance. adjust both for the
In particular, individuals D design that includes
mak- ing evaluations a oversamples and for
tend to rely on an t nonresponse, survey
“availability heuris- tic,” a weights were employed
a systematic perceptual in estimating all of the
bias whereby people a results reported below.
base their perceptions n
on examples from their d 4 The Education Next surveys
daily lives and familiar have previously been
images from the media employed to study effects of
M information about educational
that come imme- diately e inputs, specifically spending
to mind (Evans and t and teacher salaries, on
Kelley 2004; Evans, h support for increasing
Kelley, and Kolosi 1992; o funding for public schools
Tversky and Kahneman and teacher salaries (Schueler
d
and West forthcoming).
1973). Consequently, s
individuals tend to To investigate the impact
respond more posi- of relative performance
tively to concrete infor- mation on
examples and specific residents’ evaluations,
information than to survey experiments were
abstract examples and embedded in two
uncertainty (Alicke et al. national surveys of
1995; Alvarez 1998; public opin- ion on
Olsen 2015d). It seems education policy
likely that many administered in 2013 and
individuals have access 2014 under the auspices
to such examples of local of Education Next and
school performance from conducted by the polling
multiple sources. For one, firm Gfk (formerly
they live near public Knowledge Networks).4
schools, permitting
regular observa- tion of P
facilities, neighborhoods, a
and students. Indeed, r
citizens have frequent t
contact with the schools i
10 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx
Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 11

E higher than [X] performance, such as the live provided by Gfk.

x percent of [students national average, nor Data on relative school
p in Y/ American were they otherwise perfor- mance were
e primed to draw a
r students/students obtained from the
i in those nations] comparison to a Global Report Card
m and lower than particular reference index developed by
e [100-X] percent of point. Second, the Greene and McGee
n [students in Y/ statement explaining (2011, 2012) to facilitate
t American how to inter- pret the comparisons of student
a percentile of the local
l
students/students achievement levels in
in those nations]. district was framed in specific school districts to
neutral terms. Local national and
S A final treatment group
t
district scores were international norms
was informed of the described as “higher”
r despite the wide
propor- tion of pupils and “lower” than a
u variation in states’
c within their local school percentage of students, definitions of
t district graduat- ing rather than better or proficiency.5 Data on
u from high school within

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

worse, and respondents each district’s high
r 4 years of entering ninth were pre- sented with school graduation rate
e grade, in order to test both the “higher” and were obtained from
At the start of the 2013 whether absolute (i.e., “lower” framings. annual Diplomas Count
survey, respondents were not rela- tive) Finally, respondents reports produced by
ran- domly assigned to performance are provided with Education Week (2011,
one of five treatment information has a information about the 2
conditions. A control similar influence on true performance of the 0
group was given no evaluations of local actual public school 1
information about the schools (n = 1,096). The districts in which they 2
performance of their text stated: live. We were able to )
local schools (n =
According to the link each respondent to .
1,075). Three treatment
most recent their local public school
groups were told where
available informa- district using data D
their local school district e
tion, [X] percent reporting the census
ranked in math in the p
of public school block in which
state (n = 1,055), nation- e
students in your respondents
ally (n = 1,001), or n
internationally (n = local district d
1,075). The text stated: graduate within e
four years of n
A recent study t
entering 9th grade.
compared the This means that
math test scores V
[100-X] per- cent a
of students in your of entering 9th r
local school grade public school i
districts to the test students fail to a
scores of students graduate high b
across the United school within four l
States. The study e
years.
showed that Respondents to both
students in your Three features of the surveys were next asked
district perform at experimental treatments to grade their local
the [X]th percentile report- ing the percentile public schools and
of students in the of local schools are of public schools across
[state of Y/ United note. First, respondents the country on the
States/group of 25 were provided with standard A to F grading
nations with the percentile of their scale. The same letter
developed local schools relative to a grade scale has been
economies]. This given reference group. employed in several
means that the Respondents were not, recent studies of citizens’
average public however, provided with evaluations of public
school student in infor- mation about a schools (Chingos,
the district scores benchmark measure of Henderson, and West
 12 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx

2012; Clinton and dimen- sion of quality

Grissom 2015; Jacobsen, used to evaluate schools
Snyder, and Saultz as respondents are led to
2015). emphasize this particular
consideration. In short,
I each relative
d performance treatment
e is in fact two
n
treatments; they are both
t
i information and primes.
f Estimating the effect
y of information alone
i requires distinguishing
n the learning effect from a
g priming effect. To do
so, we conducted a
L
second experiment in
e
a the
r 2014 survey. All
n respondents were first
i asked to esti- mate the
n percentile rank of their
g local district among the
nation as a whole on
E math scores. The text
f
f stated:
e
c 5 Greene and McGee’s (2011,
t 2012) Global Report Card
s index uses state
Although the above accountability test results
held by the American
experiment permits
Institutes for Research. To
identification of the obtain comparable results
overall effect of relative from different state tests, a
performance informa- student-weighted average
tion, it cannot achievement score is
computed for each school
distinguish whether the district, and the district
effect arises from averages are then placed on
changing the a normal distribution within
information available to each state. The distribution
respond- ents or in each state is next shifted
up or down by the amount
changing the dimensions that their state exceeds or
of quality they use to lags the national average on
evaluate schools. The the National Assessment of
treatments in the relative Educational Progress, a
national test administered to
perfor- mance
a representative sample in
conditions provide each state. To obtain the
information that may international percentile of
yield a learning effect by each state, test results from
updating the the Programme for
International Student
information respond-
Assessment (PISA) are used
ents have available to to shift every district up or
them about their local down relative to the
schools. However, by developed economies in the
calling attention to a PISA sample.
particular consid- eration
for evaluating schools,
the treatments may also
yield a priming effect—
that is, a shift in the
Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx 13
14 Journal of Public Administration Research and Theory, 2016, Vol. xx, No. xx
Students in different parts of the country perform
differently in math. The average student in your
district performs better than what percent of stu-
dents across the country?
A treatment group was then informed of the national
percentile in math of their local school district
(n = 2,562), using identical wording to that employed
in the 2013 survey, and a control group was given no
information (n = 2,628). We then asked respondents to
evaluate the quality of their local schools, again assign-
ing a letter grade on the A to F scale.
This procedure allows us to isolate learning effects,
first, by priming all respondents and, second, by explor-
ing how treatment effects vary with respondents’ prior
beliefs. To the extent that there is a learning effect, the
effect of treatment should become increasingly posi-
tive in the degree to which respondents underestimate
the performance of their local schools and likewise
increasingly negative as respondents overestimate local
performance. This approach to identifying learning
effects was first developed in an observational context
by Lenz (2009) and has subsequently been applied in
survey experiments by Cruces, Perez-Truglia, and Tetaz
(2013), Clinton and Grissom (2015), and Schueler and
West (forthcoming).
Data Analysis
For each experiment, we first explore whether there is
a statistically significant difference between the school
grades assigned by each active treatment group and the
control group. In order to do this, we create data sets
of the control group and each treatment group and fit
a proportional odds model with treatment status as the
only predictor. The proportional odds model assumes
that there is a latent response variable, Zi, underlying
the categorical outcome that we observe, Yi, such that
Yi = j is observed if θj−1 < Zi < θj. The proportional odds
model also assumes a logit model for the cumulative
probabilities of γij, where γij(xi) = P(Yi < j|xi). We there-
fore fit the model:
γ j (x i )
log = θj − βT xi j = 1,…, J −
1 − γ j (xi ) 1
We initially fit this model with a single predictor, which
is equal to 1 if the respondent was assigned to treatment
and 0 otherwise. We report differences between treat-
ment and control groups in the predicted probabilities
of a given outcome, together with bootstrapped confi-
dence intervals. We use likelihood ratio tests to exam-
ine whether the difference between a given treatment
group and the control group is statistically significant.
We also explore how the effect of treatment var-
ies with the relevant underlying performance meas-
ure and, in the 2014 survey, with the gap between
respondents’ prior beliefs and the true value of the
performance measure. In order to do this, we employ
models including the underlying performance measure,
or the difference between respondents’ prior beliefs
and this performance measure, as well as an interac-
tion between this variable and the treatment dummy.
Further details are provided below.
Results
Prior Information About School Quality
Before reporting the experimental results, we first use
responses to the 2014 survey to examine the perfor-
mance-related information that individuals already
have. Figure 2, which displays the difference between
respondents’ perceptions of the national percentile rank
of their local schools and the actual percentile, provides
Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016
no evidence of systematic bias in perceptions. The differ-
ence between individuals’ estimates and the true percen-
tile is approximately normally distributed around zero.
However, just because perceived percentile approxi-
mates actual percentile on average, does not mean the
difference is close to zero for most individuals. Indeed,
as can be seen in the figure, many individuals are off
by a great deal. Sixty percent miss the mark by more
than 10 percentile points in either direction. For 32%,
the errors are 20 points or more in either direction. The
large proportion of respondents with errors of this mag-
nitude opens the door for substantial learning effects
from new information about relative performance.
Further, individuals’ errors in estimating the
national percentile of the schools in their communi-
ties are correlated with the true national percentile of
local schools (r = −0.39, p ≤ .001). Eighty-one percent
of respondents living near schools with test scores in
the bottom quintile nationally overestimate the rela-
tive performance of the schools in their community, for
example, compared to 28% of respondents who live
near schools in the top quintile. To the extent that the
provision of information leads citizens to correct their
beliefs about the relative performance of local schools,
therefore, this may change the distribution of people’s
evaluations of local schools.
Evaluations of School Quality
We begin our examination of the experimental data
by considering baseline evaluations of local school
quality in the control group in the 2013 survey. The
proportions of letter grades given to local schools by
respondents who are not provided with performance
information are approximately 12% A, 38% B, 32%
C, 12% D, and 6% F. The proportions of letter grades
given to schools nationally are approximately 3% A,
20% B, 55% C, 16% D, and 6% F. The distribution
of letter grades is broadly consistent with the results
Figure 2. Distribution of differences between respondent estimate of the national percentile of local schools and the true national percentile.
of recent Phi Delta treatment group gap. Likelihood ratio with the state, national,
Kappa/Gallup polls that (supplementary table A1) tests confirm that these or international
7 Results are also reported
asked the same question, supplementary table A2. treatment effects are percentile of local
reported in figure 1. Test for the proportional statistically significant schools lowers the
We first test the odds assumption for all (table 1, Rows 1–3).8 evaluations of
hypothesis that models are reported in We also examine respondents near poorly
supplementary figure A5.
providing respond- ents whether the provision of performing schools but
8 Recent evidence indicates
with information about that that the impacts of informa- tion about the has little or no effect on
local school performance information can absolute performance of the evaluations of
performance relative to vary with citizens’ prior local schools, rather respondents near the
different reference political beliefs (Baekgaard than information about highest performing
and Serritzlew 2016). We
groups depresses aver- therefore confirmed that
local school perfor- schools. Likelihood ratio
age evaluations of these treatment effects report mance relative to other tests confirm that the
school quality, using and all others reported below groups, has a similar interac- tion between
2013 survey data.6 are substantively similar for effect on respondents’ treatment and the
Figure 3 reports Republicans and Democrats evaluations of local national percentile is
when we divide the sample
differences between each school qual- ity. Table statistically significant at

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

according to party
treat- ment group and identification (results 1, Row 4, reports the the .05 level, whereas
the control group in the available upon request). result of a likelihood interac- tions between
predicted probability of ratio test for a treatment and the state
assigning local schools difference between the and international
A or B, together with evaluations of
bootstrapped confidence respondents provided
intervals.7 Informing with the graduation
respondents of local rates of local schools
school performance and respondents given
relative to the state, no information. There is
nation, or no evidence that
internationally depresses providing respondents
the probabil- ity of with the graduation rate
assigning local schools of local schools affects
A or B by evaluations of local
approximately school quality.
7, 10, and 11 percentage We next consider
points, respectively. The whether the positive
provi- sion of relative effect of rela- tive
performance performance
information thus information on the
reduces the gap between evaluations of
average evaluations of respondents living near
local schools and lower performing
schools nationally but schools is larger than
does not entirely close the negative effect of
this performance infor-
mation on the
6 We confirm that evaluations of
randomization has resulted in
balance on observable
respondents near higher
characteristics between the performing schools.
control group and each of the Figure 4 plots how the
treatment groups by predicted probabilities
comparing group means for of assigning local
continuous and binary
covariates (supplementary
schools A or B in each
table A1) and the distribution treatment group vary
in the control group to the with each of the school
distribution across different perfor- mance measures,
categories in each treatment
using the full models in
group for categorical
variables (supplementary table 1, Rows
figures A1–A4). We also 5, 6, 8, and 10.
provide a summary of the Providing respondents
mean responses of each
percentiles are statistically significant at the .10 level
(table 1, Rows 5–9). A likelihood ratio test also con-
firms that there is no statistically significant interaction
between the absolute performance information treat-
ment and the graduation rate of local schools (table 1,
Row 10).
We also consider the effect of the different infor-
mational treatments on evaluations of national school
quality. In the control group, the proportions of letter
grades given to schools nationally are approximately
3% A, 20% B, 55% C, 16% D, and 6% F. Our results
confirm that none of the informational treatments
has a statistically significant effect on evaluations of
the quality of schools nationally (supplementary table
A3). In other words, relative performance information
shifts perceptions of schools in one’s own community
but not perceptions of schools across the country in
Learning Effect vs. Priming Effect
In 2014, respondents were either informed of the
actual national percentile rank of their local school
district or given no additional information, only after
all were first asked to offer their own estimate of the
percentile rank. The grades assigned to local schools
by respondents in the control group are slightly less
favorable than in the 2013 sample: 12% A, 36% B,
33% C, 12% D, and 6% F.9 This difference between
the grade distributions in the control groups from the
two survey years could suggest that priming respond-
ents to think about the relative academic performance
of local schools depresses evaluations of local school
quality.
The estimated difference in the probability of assign-
ing local schools a grade of A or B between the treat-
ment and control groups is considerably smaller than

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

general. was found in 2013. Treatment depresses the estimated
probability of assigning local schools an A or B by
.018, as compared to .069 in 2013. The bootstrapped
confidence interval for this estimate indicates that the
difference in probabilities is not statistically significant
at the .05 level, and a likelihood ratio test confirms
that the treatment indicator is not a statistically signifi-
cant predictor (table 2, Model 1).
The finding that after providing a prime to all
respondents to think about relative performance the
informational treatment no longer depresses average
evaluations of school quality might appear to suggest
that priming alone explains the negative effects on
average evaluations in the 2013 experiment reported
above. However, the small and statistically insignifi-
cant average treatment effect masks heterogeneity

9 Supplementary table A1 provides a summary of responses for the

control and treatment groups and confirms that randomization has
Figure 3. Difference in predicted probabilities of assigning local result in balance on observable characteristics between the control
schools A or B for treatment versus control. and treatment group.

Table 1. Likelihood Ratio Tests to Compare Models for Grade Assigned to Local Schools

Full Model Reduced Model LRT p

(1) st 1 .0004
(2) nt 1 <.0001
(3) it 1 <.0001
(4) gt 1 .575
(5) st + sp + st: sp st + sp .067
(6) nt + np + nt: np nt + np .012
(7) nt + np + ng + nt: np nt + np + ng .012
(8) it + ip + it : ip it + ip .077
(9) it + ip + ig + it : ip it + ip + ig .076
(10) gt + gr + gt: gr gt + gr .447

Note: st, nt, it, and gt are the state, national, international, and graduation rate treatments. sp, np, ip, and gr are the state, national, and interna-
tional percentiles, and the graduation rate. ng and ig are the gap between the national or international percentile, and the state percentile. LRT
p-value reports the p-value for a likelihood ratio test that compares the likelihood of observing the data under the full and reduced models.
A low p-value indicates that the data were not as likely to occur under the reduced model as under the full model.
P(Grade Schools A/B)
1.00
 0.75
T rea tment

0.50 No Information

State Percentile
0.25

0.00
0 25 50 75 100

State Percentile

1.00
P(Grade Schools A/B)

0.75
T reatm ent

0.50 No Information

National Percentile
0.25

0.00
0 25 50 75 100

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

National Percentile

1.00
P(Grade Schools A/B)

0.75
T reatm ent

0.50 No Information

Internat. Percentile
0.25

0.00
0 25 50 75 100

Internat. Percentile

1.00
P(Grade Schools A/B)

0.75
T rea tm ent

0.50 No Information

Graduation Rate
0.25

0.00
0 25 50 75 100

Graduation Rate

Figure 4. Predicted probabilities of grading local schools A or B against school performance.

Table 2. Likelihood Ratio Tests to Compare Models across respondents. Figure 5 plots predicted probabili-
for Grade Assigned to Local Schools Using 2014 ties of assigning local schools A or B against the actual
Survey Data
national percentile rank, using a model including an
Full Model Reduced Model
interaction between treatment and the national per-
LRT p
centile rank (table 2, Row 2, full model). As with the
(1) nt 1 .149 2013 data, the effect of treatment on the probability
(2) nt + np + nt: np nt + np .047 of assigning A or B becomes increasingly positive as
(3) nt + gp + nt: gp nt + gp <.0001 the national percentile rank of local schools increases.
Note: nt is an indicator for the national percentile treatment,
A likelihood ratio test confirms that the interaction
np is the actual national percentile, and gp is the gap between the between treatment and the national percentile rank is
actual and the estimated national percentile. LRT p-value reports a statistically significant predictor (table 2, Row 2).
the p-value for a likelihood ratio test that compares the likelihood of A test to identify learning effects in the context of
observing the data under the full and reduced models. A low p-value potential priming requires examining how the treat-
indicates that the data were not as likely to occur under the reduced
model as under the full model.
ment effect varies with prior misperception. Larger
treatment effects among respondents who lack relative
performance information (as measured by their own
estimates of where their local schools rank nationally)
indicate a learning effect. Because both groups received
the prime, these effects cannot be attributed to prim-
ing. We therefore fit a model that includes the differ-
ence between actual national percentile and estimated
percentile along with an interaction between this differ-
ence and the treatment indicator (table 2, Row 3, full
model). Figure 6 uses this model to plot how the prob-
ability of assigning local schools A or B varies with the
gap between the estimates and true national percentile.
In figure 6, we observe that the more a respondent
underestimates the national percentile rank of local
schools, the greater the amount by which providing
the respondent with true percentile rank increases the
probability of grading local schools A or B. Likewise,
the more a respondent overestimates the true percen-
tile, the more treatment depresses the probability of
assigning local schools A or B. A likelihood ratio test
confirms that the interaction between treatment and
the respondent estimated percentile is a statistically
significant predictor (table 2, Model 3). These results
indicate that respondents who err in their prior beliefs

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

Figure 5. Predicted probabilities of grading local schools A or B against the actual national percentile: 2014 survey data.

Figure 6. Predicted probabilities of grading local schools A or B against the gap between estimated and actual national percentile: 2014
survey data.
about local school Presenting respondents nation’s, rather than on data. Indeed, our results
performance update with information about other considerations. are consistent with a
those beliefs, and in turn the academic Specifically, the average scenario in which the
their evaluations, in performance of local treatment effect of public evaluates local
response to new schools relative to information found in the schools based on high
information. schools in the state, 2013 sur- vey shrinks school graduation rates
This learning effect nation- ally, or and loses statistical and has reasonably
also helps to explain internationally depresses significance in the 2014 accurate information on
why the effect of average evaluations of survey, when both them. We likewise
treatment on local school quality. The treatment and control cannot exclude the
respondents’ effect is especially large groups are primed to possibility that in the
evaluations becomes among those living near consider relative absence of treatment
increasingly positive as the lowest scoring performance. respondents evaluate
the national percentile schools. Absolute The observed learning schools based on criteria
rank of local schools information about the effects occur in a other than academic
increases, as reported in performance of local context in which citizens performance, such as a
figures schools, as measured by already have access to school’s facilities or

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

4 and 5. In the absence graduation rates, has no multiple sources of demographic makeup.
of treatment, similar effect. information about The evidence indicates
respondents liv- ing near Relative performance school performance, that relative performance
to schools with a lower information influences from their own and is not typically one of
national percentile rank evalu- ations in part others’ experiences, as those considerations,
tend to overestimate through a learning well as publicly reported however, and the
the performance of effect. Respondents who provision of relative
their local schools to a overestimate the performance information
greater degree than relative performance of leads citizens to update
respondents near higher the schools in their their views.
performing schools. community lower their The findings suggest
Consequently, evaluations of local that relative
respondents living near school quality in comparisons can play a
to worse performing response to new role in shaping citizens’
schools change their informa- tion, beyond evaluations of public
beliefs and evaluations any impact of the service quality, with
more when provided prime provided to both important implications
with the true national treatment and control of demo- cratic
percentile of their local groups in the 2014 sur- policymaking. In the
schools. vey. This learning effect absence of this informa-
also leads respondents tion, the public appears
who underestimate the to evaluate public
C
performance of their institutions and services
o
local schools to raise in such a way as to
n
c their evaluations, overestimate quality in
l however, and therefore their local communities
u can- not account for relative to the nation as
s the overall reduction in a whole. Citizens living
i evaluations observed in near the lowest
o 2013. performing ser- vices
n There is suggestive tend to exhibit the
evidence that relative greatest positive biases in
The experimental results
per- formance their judgments of local
reported in this article
information also works services. To the extent
pro- vide evidence that
through priming that citizens’ evaluations
information about how
respondents to base of the quality of local
local pub- lic services
their evaluations of services influence their
rank compared to
local school quality on support for reforms,
services nationwide (or
how the academic these biases may reduce
other comparison
performance of their sup- port for changes
groups) influences
community’s schools locally, particularly near
citizens’ evalua- tions of
compares to the the worst performing
local service quality.
services. to close the gap between
This study’s findings per- ceptions of local
also show that informing schools and schools
citi- zens of how their nationally.
local services rank
relative to other services
S
is sufficient to influence
u
citizens’ evaluations. It is
p
not necessary to prime p
citizens to draw l
comparisons to a e
particular reference m
point. Again, this has e
signifi- cant implications n
for policymaking. t
Informing citizens of the a
ranking of local services r
in neutral terms offers y
less opportunity for
manipulation by M
bureaucrats or a
policymakers than t
priming a particular e
comparison. Relative r
performance i
information in the form a
of rank- ings may l
therefore be more Supplementary material
acceptable to and is available at the
widely trusted by the Journal of Public
public. Administration Research
The implications of and Theory online
this study’s findings for (www.jpart.oxfordjourn
demo- cratic als.org).
policymaking are
apparent in the context
of the US public school
system. In public
education, each state sets
its own academic
standards and
thresholds for student
proficiency. As a
consequence,
comparisons of local
public schools to schools
beyond the state are all
but impossible for most
citizens. The evidence
here indi- cates that
many Americans would
be less pleased with their
schools if they were able
to make these compari-
sons. This study also
indicates that informing
citizens of the ranking of
their local schools
relative to schools
nationally would begin
 cost information and service Press.
F Hochschild, Jennifer L., and
use affect the relationship? Education Week. 2011. Diplomas Nathan Scovronick. 2004. The
u count 2011—Beyond high
International Public American dream and the public
n Management Journal 18:228– school, before baccalaureate: schools. New York, NY:
d 45. Meaningful alternatives to a Oxford University Press.
i Baekgaard, Martin, and Søren four-year degree. Bethesda, Jacbosen, Rebecca, Jeffrey W.
n Serritzlew. 2016. Interpreting MD: Editorial Projects in Snyder, and Andrew Saultz.
Education.
g performance information: 2015. Understanding
Motivated reasoning or ——. 2012. Diplomas count 2012 satisfaction with schools: The
The 2013 and 2014 unbiased comprehension. —Trailing behind, moving role of expectations. Journal
Education Next Public Administration Review forward: Latino students in of Public Administration
U.S. schools. Bethesda, MD:
Surveys were 76:73–82. Research and Theory 25:831–
Bushaw, Willaim J., and Shane J. Editorial Projects in Education. 48.
administered by Gfk Lopez. 2012. Public education in Evans, M. D. R., and Jonathan James, Oliver. 2011.
(formerly Knowledge the United Kelley. 2004. Subjective social Performance measures and
Networks) and funded State: A location: Data from 21 democracy: Information
nation
by the William Simon divided.
nations. International Journal effects on citizens in field and
of Public Opinion Research
Foundation and the Bill Kappan laboratory experiments.
Magazine 1 Journal of Public
& Melinda Gates 94:8–25. 6
Administration Research and
Foundation. Charbonneau, Étienne, and Gregg :

Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016

3 Theory 21:399–418.
G. Van Ryzin. 2015.
– James, Oliver, and Alice Moseley.
Benchmarks and citi- zen 3 2014. Does performance
R judgments of local government 8
information about public
Performance: Findings from a .
e Evans, M. D. R., Jonathan Kelley,
services affect citizens’
survey experiment. Public
f Management Review 17:288– and Tamas Kolosi. 1992.
perceptions, satisfaction, and
e voice behav- iour? Field
304. Images of class: Public
experiments with absolute and
r Chingos, Matthew M., Michael perceptions in Hungary and
relative performance informa-
e Henderson, and Martin R. Australia. American
tion. Public Administration
Sociological Review 57:461–
n West. 2012. Citizen perceptions
92:493–511.
of government service quality: 82.
c Fung, Archon, Mary Graham, and
James, Oliver, and Gregg G. Van
Evidence from public schools.
e Quarterly Journal of Political David Weil. 2008. Full disclosure:
Ryzin. 2015. Incredibly good
s Science 7:1–35. The perils and promise of
performance: An experimental
study of source and level
Alicke, Mark D., M. L. Klotz, Clinton, Joshua D., and Jason A. transparency. New York, NY:
effects on the credibility of
David L. Breitenbecher, Tricia Grissom. 2015. Public Cambridge University Press.
government. American Review
J. Yurak, and Debbie S. information, public learning, Greene, Jay P., and Josh B.
of Public Administration.
Vredenburg. 1995. Personal and public opinion: McGee. 2011. George W. Bush
Lenz, Gabriel S. 2009. Learning
contact, individuation, and Democratic accountability in Institute Global Report Card:
and opinion change, not
the better-than-average effect. education pol- icy. Journal of Technical appendix.
priming: Reconsidering the
Journal of Personality and Public Policy. Advance online http://globalreportcard.org/docs/
priming hypothesis. American
Social Psychology publication. http://journals. AboutTheIndex/Global-Report-
Journal of Political Science
6 cambridge.org/abstract_S0143 Card-Technical-Appendix-9-28-
53:821–37.
8 814X14000312 (accessed April 11.pdf
: Linn, Suzanna, Jonathan Nagler,
1, 2016). (
8 a and Marco A. Morales. 2010.
0 Cruces, Guillermo, Ricardo Perez-
c Economics, elections, and
4 Truglia, and Martin Tetaz. c voting behavior. In The Oxford
– 2013. Biased perceptions of e
2 handbook of American elec-
income distribution and s
5 s tions and political behavior, ed.
preferences for redistribu-
. e Jane E. Leighley, 375–96. New
tion: Evidence from a survey d
Alvarez, R. Michael. 1998. York, NY: Oxford University
experiment. Journal of Public
Information and elections. Press.
Economics
Ann Arbor, MI: University of A March, James G., and Herbert A.
9 p
Michigan Press. Simon. 1958. Organizations.
8 r
Andersen, Simon Calmar, and : New York, NY: Wiley.
i
Morten Hjortskov. 1 l Marvel, John D. 2016. Unconscious
Forthcoming. Cognitive biases 0 bias in citizens evaluations of
in performance evaluations. 0
public sec- tor performance.
– 1
Journal of Public 1 , Journal of Public
Administration Research and 2 Administration Research and
Theory. . Theory
2
Associated Press. 2013. Indiana: Daniels, David P., Jon A. 26:143–58.
0
Grade manipulation found. Krosnick, Michael P. Tichy, and
1 Moe, Terry M. 2001. Schools,
Trevor Tompson.
New York Times. 6 vouchers, and the American
2012. Public opinion on )
http://ww w.nytimes.com/201 public. Washington, DC:
environmental policy in the .
3/08/08/us/indiana-grade- Brookings Institution Press.
United States. In Handbook ——. 2012. When the
manipulation- found.html best is mediocre. Newport, Frank. 2013. Americans’
of U.S. environmental policy,
(accessed April 1, 2016). Education Next 12:34– view of healthcare quality,
ed. Michael Kraft and Sheldon
Baekgaard, Martin. 2015. 40. cost, and cov- erage: Public
Kamieniecki, 461–86. New
Performance information and still more positive about their
York, NY: Oxford University
citizen service attitudes: Do situations than the nation’s.
 Gallup. ——. 1939. The administrator in
http://www.gallup.com/poll/16 search of statistics. Public
Management
5998/americans-views-
21:106–9.
healthcare- quality-cost-
Simon, Herbert A., and Clarence E.
coverage.aspx (accessed April
Ridley. 1938. Trends in
1, 2016).
municipal report- ing. The
Olsen, Asmus Leth. 2015a.
Public Opinion Quarterly
Compared to what?
2:465–8.
Experimental evidence on
Stecher, Brian M., Frank Camm,
social and historical reference
Cheryl L. Damberg, Laura S.
points in performance
Hamilton, Kathleen J. Mullen,
evaluation. Paper presented at
Christopher Nelson, Paul
the 11th Public Management
Sorensen, Martin Wachs,
Research Conference, Madison,
2013.
——. 2015b. Negative
performance information
causes asymmetrical evalua-
tions and elicits strong
responsibility attributions.
Working Paper.
——. 2015c. Citizen
(dis)satisfaction: An
experimental equivalence
framing study. Public
Administration Review
75:469–78.
——. 2015d. Human interest or
hard numbers? Experiments on
citizens’ selec- tion, exposure,
and recall of performance
information. Working Paper.
——. 2016. The numerical
psychology of performance
information: Implications for
citizens, managers, and policy
makers. Public Performance
and Management Review
39:100–15.
Peterson, Paul. E., Michael
Henderson, and Martin R.
West. 2014. Teachers versus
the public: What Americans
think about schools and how
to fix them. Washington, DC:
Brookings Institution Press.
Peterson, Paul E., and Carlos Xabel
Lastra-Anadon. 2010. State
standards rise in reading, fall in
math. Education Next 10:12–
6.
Saad, Lydia. 2011. Most Americans
believe crime in U.S. is
worsening: Slight majority rate
U.S. crime problem as highly
serious; 11% say this about
local crime. Gallup.
http://www.gallup.com/poll/15
0464/americans- believe-crime-
worsening.aspx (accessed April
1, 2016).
Schueler, Beth E., and Martin R.
West. 2016. Sticker shock:
How informa- tion affects
citizen support for public
school funding. Public
Opinion Quarterly 80:90–113.
Simon, Herbert A. 1937.
Comparative statistics and the
measurement of effi- ciency.
National Municipal Review
26:524–7.
 Allison Yoh, and Gail L.
Zellman. 2010. Toward a
culture of conse- quences:
Performance-based
accountability systems for
public services. Santa Monica,
CA: RAND Corporation.
Tversky, Amos, and Daniel
Kahneman. 1973. Availability:
A heuristic for judg- ing
frequency and probability.
Cognitive Psychology 5:207–
32.
Van Ryzin, Gregg G., and Cecilia
F. Lavena. 2013. The
credibility of govern- ment
performance reporting: An
experimental test. Public
Performance and Management
Review 37:87–103.
Walker, Samuel E., and Carol A.
Archbold. 2014. The new
world of police accountability,
2nd ed. London: Sage.
Downloaded from http://jpart.oxfordjournals.org/ at University of Leeds on June 5, 2016