Response Bias Project

Zoe Ervolino
Introduction
In this study, Brandon, Nell, and I wanted to see whether a persons judgment of a quotation
was related to the person who said the quote. In order to do this, we explored whether or not
changing the identity of the person who said a quotation would impact the way that people
would perceive the quote. Specifically, we tested to see if there was a notable distinction
between Donald Trump and Peter J. Douglas, a person who was created specifically for this
experiment, in order to see if connotations with Trump would cause students to respond
differently. Because the quotation was the same for both Trump and Douglass, tracking the
change in the response as the names changed would be a way to examine the association
between the name and the reaction. As Packer is considered an institution that primarily
consists of liberals, we predicted that students would be more likely to be neutral or disagree
with the quotation when it came from Trump, and more likely to agree with the quotation when it
came from Peter J. Douglas, because we assumed the students would have fewer (or no)
preconceived notions of Douglas. The specific quotation used came from Donald Trumps Time
to Get Tough, and says that Any system that penalizes success and accomplishment is wrong.
Any system that discourages work, discourages productivity, discourages economic progress, is
wrong.[1]
Data/Results
Figure 1. Survey results
Trump
Strongly Agree

Douglas
15

22
Agree

65

43

Neither Agree nor

Disagree

29

35

Disagree

14

Strongly Disagree

Total

119

111

Figure 2. Results grouped by response type

Affirmative Response
Neutral Response

Trump

Douglas

87

58

29

35

Negative Response

Analysis and Conclusions

Test for Homogeneity
Conditions: Random? The surveys were assigned randomly. Large sample size? Expected
counts: 75.022, 69.978, 33.113, 30.887, 10.865, 10.135, which are all larger than 5.
Independent? The sample was closer to a census than a true sample, and no response affected
any other response. Proceed with caution, however.
H0 = Survey A and Survey B follow the same distribution of affirmative, neutral, and negative
responses.
Ha = Survey A and Survey B do not follow the same distribution of affirmative, neutral, and
negative responses.
X2 = 1.912 + 2.05 + .511 + .548 + 5.694 + 6.109 = 16.819
Degrees of freedom = (2)(1) = 2
P-value = 2.23 x 10-4
Because the p-value(2.23 x 10-4) is less than .05, we can reject our null hypothesis that Survey
A and Survey B would follow the same distribution of affirmative, neutral, and negative
responses follow the same distribution. There is evidence to suggest that the distribution of
affirmative, neutral, and negative responses to both surveys were different.
Two-sample z test for pa pb
Conditions: Random? The data was taken from a randomized experiment. Normal? (119*.73)=
86.87, (119*.27) = 32.13, (111*.53)= 58.83, (111*.47) = 52.17; all values are greater than 10.
Independent? The sample was closer to a census than a true sample, thus this condition should
be disregarded. Proceed with caution, however.
H0 = The true difference in the proportion of affirmative responses to the quote on Survey A and
the proportion of affirmative responses to the quote on Survey B is equal to 0.
Ha = The true difference in the proportion of affirmative responses to the quote on Survey A and
the proportion of affirmative responses to the quote on Survey B is not equal to 0.
ppa = 87/119
ppb= 58/111
SE = .0624
Z = 3.2746
P = 1.1 x 10-3

Because the p-value (1.1 x 10-3) is less than .05, we can reject our null hypothesis that the true
difference in the proportion of affirmative responses to the quote on Survey A and the proportion

of affirmative responses to the quote on Survey B is equal to 0. There is evidence to suggest

that the true difference in the proportion of affirmative responses to the quote on Survey A and
the proportion of affirmative responses to the quote on Survey B is not equal to 0. In other
words, there is statistically significant evidence that the true proportion of Packer students who
agreed with a quotation was higher when it was attributed to Trump.
Reflection
Though our experiment was relatively successful and produced interesting findings, there were
a few issues. For one thing, the inclusion of a Neither Agree nor Disagree category was
unnecessary as, in some sense, it allowed people to avoid participating in the specific question
we were addressing. Instead, we should have omitted such a category and forced participants
to make a decision about their agreement or disagreement with the quotation. Another issue
was that we used the name Peter J. Douglas to serve as a character with which subjects
would have no associations (because he was not a real character), however this decision was a
bit ignorant; it is quite conceivable that students might have biases associated with Douglas
because, in context, he appears a middle-aged, probably white, man. Instead, we should have
rewritten the alternate question to contain solely the quote to completely eliminate bias with an
individual.
Extra Credit:
Test for Association
Figure 3. Cheating by amount of years spent at Packer
I have not cheated

I have cheated, but

not within the past
year

I have cheated
within the past year

0-2 years

22

11

3-4 years

19

12

10

5+ years

51

33

24

Total

77

67

45

H0 = There is no association between the amount of years a student has spent at Packer and
their cheating habits.
Ha = There is an association between the amount of years a student has spent at Packer and
their cheating habits.
Conditions: Random? The surveys were randomly assigned. Independent? The sample taken
was essentially a census but independence is still met because no response affected another.
Large sample size? Expected counts: 16.296, 14.179, 16.703, 14.534, 44, 38.286, which are all
larger than 5.

X2 = 5.303 + 4.313 + .316 + .442+ 5.694 + .729 = 12.566

Degrees of freedom = (3-1)(3-1) = 4
P-value = 1.36 x 10-2
Because the p-value (1.36 x 10-2) is less than .05, we can reject our null hypothesis that there is
no association between the amounts of years a student has spent at Packer and their cheating
habits. There is statistically significant evidence to suggest that there is an association between
the amount of years a student has spent at Packer and their cheating habits.
** It should be noted that technically the cheating data were derived through different
treatments, however there was no statistically evidence that indicated that the treatment led to a
difference among the responses, so the data was treated as if it came from the same treatment.

[1] http://www.ontheissues.org/Celeb/Ronald_Reagan_Tax_Reform.htm