Chapter 11

Hypothesis Testing Using the

One-Way Analysis of Variance
ANOVA
The analysis of variance (abbreviated ANOVA) is a
powerful hypothesis testing procedure that extends
the capability of t-tests beyond just two samples.
There are many types of ANOVAs, today we will learn
about a one-way independent-measures ANOVA
Well learn one-way repeated-measures ANOVA next
week.
Well learn two-factor ANOVA the week after.
These ANOVAs are by no means all of them! There
are a LOT more types!
One-Way ANOVA
The independent measures ANOVA is used in the same
types of situations that the independent measures t-test
had been used, except that the ANOVA allows for the
comparison of more than just two groups.
Before the advent of the computer, if someone had three
groups in an experiment, they would often employ a series
of t-tests to compare all possible combinations of means.
If you had three samples to compares then, using t-tests,
we would have to do M
1
vs. M
2
, M
1
vs. M
3
, and M
2
vs. M
3
Each time we do a t-test, the type I error rate is equal to o.
The experiment-wise error rate (o) is held at .05 in an
ANOVA.
One-Way ANOVA
Preliminary Example
Pretend you wanted to know the effects of different temperatures
on the ability to learn. You assigned n = 5 subjects to each of
three treatment conditions. Each of the five subjects in each group
were placed into a room of the appropriate temperature and were
asked to solve problems. The DV is the number of problems
correctly solved.
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
One-Way ANOVA
Preliminary Example
New Statistical Notation
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
Some slightly new statistical symbols and notation will be
introduced in ANOVA.
A factor (a.k.a. IV) is something that is manipulated by an
experimenter. In this experiment we have a single factor,
temperature of room.
Since a factor is manipulated, it must have at least two levels. In
this experiment, the factor has three levels. The number of levels
of a factor is symbolized by the letter k. If a statistical formula
required us to use k, for this experiment, we would use 3.
One-Way ANOVA
Preliminary Example
New Statistical Notation
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
T
1
=5 T
2
=20 T
3
=5
Another new notation is T, it refers to the total of all of the
scores in a particular treatment. For this experiment T
1
=
5, T
2
= 20, and T
3
= 5. We have been calling this particular
thing EX up until now.
One-Way ANOVA
Preliminary Example
New Statistical Notation
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
T
1
=5 T
2
=20 T
3
=5
G = T
1
+ T
2
+ T
3
= 30
The capital letter G refers to the grand total of all the
individual scores. G also equals T
1
+ T
2
+ T
3
.
One-Way ANOVA
Preliminary Example
New Statistical Notation
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
n
1
=5 n
2
=5 n
3
=5
N = n
1
+ n
2
+ n
3
= 15
The lowercase letter n indicates the number of scores in each group, n
1

indicates the number of scores in the first group, n
2
the number of scores
in the second group, and so on. The capital letter N represents the total
number of scores in all groups. For this experiment N = n
1
+ n
2
+ n
3
.
There does not have to be the same number of scores in each group.
Theory of ANOVA
ANOVA uses the notion of variability in its
calculations. In a set of scores there is a certain
amount of variability. This total variability can be
split into two pieces, the good stuff and the bad
stuff.
As a general rule, we want the bad variability to
be very small and the good variability to be very
large.
Theory of ANOVA
Bad Variability
In an ideal experiment, the subjects within each treatment
condition are treated exactly the same, and therefore
every single one should respond exactly the same.
However, people dont respond in exactly the same way.
This different responding creates something called error
variance.
Error variance has two pieces, a portion caused by
individual differences, and a portion caused by
experimental error.
Experimental error refers to the fact that no matter how
hard an experimenter tries to treat every subject in a group
exactly the same, inevitably, they will be treated slightly
differently.
Error variance is also called within-groups variance.
Theory of ANOVA
Good Variability
The second thing which creates variability in an
experiment is the fact that the subjects were
assigned to different treatments. We want this
to be large because that is the entire point of
designing an experiment.
This is called between-groups variance.
Theory of ANOVA
Total Variability
X (X M)
2

0 4
1 1
3 1
1 1
0 4
4 4
3 1
6 16
3 1
4 4
1 1
2 0
2 0
0 4
0 4
EX = 30
M
All
= 2
( )
Total
SS X X = =

46
2
Theory of ANOVA
Bad Variability
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
( )
6
5
5
11
2
1
2
1
1
2
1
= = =

n
X
X SS
EX
1
= 5
EX
2
1
= 11
EX
2
= 20
EX
2
2
= 86
EX
3
= 5
EX
2
3
= 9
( )
6
5
20
86
2
2
2
2
2
2
2
= = =

n
X
X SS
( )
4
5
5
9
2
3
2
3
3
2
3
= = =

n
X
X SS
16 4 6 6
3 2 1
= + + = + + = SS SS SS SS
Within
Theory of ANOVA
Good Variability
Temperature of room
50
o
70
o
90
o
0 4 1
1 3 2
3 6 2
1 3 0
0 4 0
M
1
= 1
M
2
= 4
M
3
= 1
M
All
= 2
30 5 ) 2 1 ( 5 ) 2 4 ( 5 ) 2 1 (
) (
) (
) (
2 2 2
3
2
3
2
2
2
1
2
1
= - + - + -
= - +
- +
- =
n X X
n X X
n X X SS
All
All
All
Between
30 = =
Within Total Between
SS SS SS
Theory of ANOVA
F-ratio
In order to do an ANOVA we need to calculate something
called a mean square.
A mean square (abbreviated MS) is simply an average sum of
square. We have used the idea of the MS before, but we called
it variance which is the average squared deviation about the
mean.
We need MS from two different sources, an MS calculated from
the scores within the groups, symbolized as MS
within
, and an MS
calculated from the differences between the groups, symbolized
MS
between
.
MS
within
is a measure of variance that includes two types of
variability, that caused by individual differences, and that
caused by experimental error. Both the sources of bad
variability. Because of this, the MS
within
is also called MS
error
.
MS
between
is a measure of variance that includes both the
sources of error, plus it includes variability due to the fact that
people were assigned to different treatments.
Theory of ANOVA
F-ratio
ANOVA uses a ratio of these two measures of variance to
create an F-ratio. (F = MS
between
/MS
within
). This is equivalent to
saying:


In the ANOVA, as in the t-test, the H
0
is that there is no
differences between the s of the treatments. If this is true,
then the treatment variability = 0 and the ratio becomes two
things over themselves and the ratio becomes 1. If the ratio is
larger than 1, then the treatment variability must be something
other than 0.
If H
0
is true, then all means are the same and MS
between
results
from error.
If H
0
is false, then MS
between
results because the means are
different.
F =
treatment variability+individual differences variability+experimental error variability
individual differences variability+experimental error variability
One-Way ANOVA
Computational Formulas
k N df
k df
N df
n
T
X SS
N
G
n
T
SS
N
G
X SS
within
between
total
within
between
total
=
=
=
E =
=
E =

1
1
2
2
2 2
2
2
within
between
Obt
within
within
within
between
between
between
MS
MS
F
df
SS
MS
df
SS
MS
=
=
=
One-Way ANOVA
Full Example
A random sample of n = 6 students from each of four
different statistics classes are asked to take a
cumulative statistics test at the end of the semester. Do
the students in different classes score differently? Test
with o = .05.
Class Instructor
Brown Smith Jones Johnson
98 76 67 54 k = 4
79 69 80 76 n = 6
87 80 96 32 N = 24
88 60 70 90 G = 1747
76 78 64 59 EX
2
= 132127
64 57 70 77
T
1
=492 T
2
=420 T
3
=447 T
4
=388

One-Way ANOVA
Full Example
Step 1: State the Hypotheses
H
0
: The scores in all classes are the same.
H
a
: The scores in the classes are not all the
same, at least one is different from the others.

H
0
:
1
=
2
=
3
=
4
H
a
: Not all s are equal
One-Way ANOVA
Full Example
Step 3: Compute the Statistic




20 4 24
3 1 4 1
23 1 24 1
= = =
= = =
= = =
k N df
k df
N df
within
between
total
83 . 3990
6
388
6
447
6
420
6
492
132127
2 2 2 2 2
2
=
|
|
.
|

\
|
+ + + = E =

n
T
X SS
within
125 . 969
24
1747
6
388
6
447
6
420
6
492
2 2 2 2 2 2 2
= + + + = =

N
G
n
T
SS
between
96 . 4959
24
1747
132127
2 2
2
= = E =
N
G
X SS
total
One-Way ANOVA
Full Example
Step 3: Compute the Statistic
Using the numbers, well create an ANOVA source
table:

Source SS df MS F
obt
Between 969.125 3 323.04 1.62
Within 3990.83 20 199.54
Total 4959.96 23
One-Way ANOVA
Full Example
Step 2: Define the Critical Value
The F distribution is a probability distribution just like any
other probability distribution.
Just like the t-distribution, the exact shape depends on the
df, but df from two sources in this case (between and
within).
However, the value of F can never be negative (no such
thing as a directional ANOVA), so it has a funny shape!
Regardless of the funny shape, the same principles apply.
See Table 4 (Page 278).

F
crit
(3, 20)
o = .05
= 3.10
One-Way ANOVA
Full Example
Step 4: Make a Decision About H
0
Because F
obs
< F
crit
, we will fail to reject H
0
and
conclude that the scores students receive after
taking different classes do not significantly
differ.
Single-Factor Independent Measures
ANOVA
Unequal ns
If the size of the samples in each of the groups
is unequal, the ANOVA can still be used. This is
particularly true if they arent very different.
If they are different, be careful when performing
the calculations that as your calculating sum
T/n you match the appropriate Ts with the
appropriate ns.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
If we perform the ANOVA and the F
obt
is larger than
the F
crit
then we would reject H
0
and state At least one
treatment mean is different.
At this point however, we have no idea which
treatment means are different from which.
When we performed a t-test and obtained a significant
result we were able to make some kind of specific
statement such as
1
is significantly larger than
2
".
As long as there are only two means, this can be
done.
As soon as we have more than two treatment
means, we can no longer simply make these
statements without further exploring.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
When performing the general ANOVA, the initial
F
obt
is often called an Omnibus F-ratio.
Omnibus meaning large.
If the Omnibus F-ratio is significant, then we
must perform some additional tests to find out
which treatment means differ from which other
ones.
There are two main types.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
a priori tests (coming from Latin for before the fact)
With a priori tests, you decide before you ever run
the initial experiment which treatments you expect
to differ from which other treatments.
Very powerful procedures.
We wont be doing anything with these for now.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
post hoc tests (coming from Latin for after the fact)
With post hoc tests, you run them only if you find
that the Omnibus F-ratio is significant.
For this class there will be two types of pairwise
comparisons used:
Tukeys honestly significant difference test (hsd)
Must have equal ns in each group.
Fishers protected t-tests
ns can be different.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Tukeys hsd
The Tukeys hsd test is a commonly used post hoc
test in psychology (the one I use most often) and
requires that the sample size of each treatment be
equal.
In the Tukeys test we will first determine how much
the means from the different groups differ.
Well create a table of mean differences.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Tukeys hsd
Then we calculate an hsd which is the minimum
amount two means need to differ before the difference
is considered to be significant.
We need to find q (Table 5 on page 281 using the
values of the df
error
(df
within
) and k) in order to
calculate hsd.
Finally, we will compare the minimum to the actual
differences to see which pairs of means actually differ.
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Example Tukeys hsd
11-3 of the homework (different stress levels leading
to viral infections)
neg min mod sev
(2.00) (3.00) (5.75) (6.00)
(2.00) ------------ 1.00 3.75 4.00
(3.00) ------------ ----------- 2.75 3.00
(5.75) ------------ ----------- ------------ 0.25
(6.00) ------------ ------------ ------------ ------------
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Example Tukeys hsd
MS
within
= 1.729
k = 4
df
within
= 12
o = .05
q = 4.20
n
MS
q hsd
within
- =
76 . 2
4
729 . 1
20 . 4 = - = hsd
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Example Tukeys hsd
Finally, well compare the minimum hsd (2.76) with the actual
mean differences.
Having negligible stress (M = 2.0) leads to significantly fewer
viral infections than having moderate stress (M = 5.75).
Having negligible stress (M = 2.0) leads to significantly fewer
viral infections than having severe stress (M = 6.0).
Having minimal stress (M = 3.0) leads to significantly fewer
viral infections than having severe stress (M = 6.0).
Single-Factor Independent Measures
ANOVA
Hypothesis Testing
Fishers Protected t-tests
If the samples are different sizes we can do a
separate t-test between each pair of means.
Fishers protected t-tests assure that the
probability of making one or more Type I errors
in all of the t-tests is equal to o.
This is called the family-wise Type I error
rate.
Single-Factor Independent Measures
ANOVA
Effect Size
Just because an ANOVA effect is significant does not
mean it is important.




The larger the value of q
2
, the more consistently the
factor caused changes in the dependent variable.
6968 .
4375 . 68
6875 . 47
2
= = =
total
between
SS
SS
q
Single-Factor Independent Measures
ANOVA
Confidence Interval for each
It is possible to compute a confidence interval for the
value of population mean () for each level of the
factor.
I have never seen it done, but it is nice to know that
you can.
X t
n
MS
X t
n
MS
crit
within
crit
within
+ + -
|
|
.
|

\
|
s s + -
|
|
.
|

\
|
) ( ) (
Single-Factor Independent Measures
ANOVA
The Relationship Between t and F
If you have a situation where k = 2, then you could
do either an ANOVA or a t-test. If one revealed a
significant difference, then the other one would as
well.
In fact, the obtained F would be identical to the
obtained t
2
.
Single-Factor Independent Measures
ANOVA
Assumptions
The observations within each sample must be
independent.
Absolutely critical
The populations from which the samples are selected
must be normal.
Not too important, especially with large samples.
The populations from which the samples are selected
must have equal variances.
Can be tested with Hartleys F-max test for
homogeneity
Single-Factor Independent Measures
ANOVA
Example
A researcher wants to know whether three different
breeds of rats display different levels of curiosity. The
researcher chooses Long-Evans hooded rats, Sprague-
Dawley rats, and Feral rats. Several rats of each breed
are placed into an experimental chamber filled with
novel objects and the number of objects that the rats
contact in a one minute period is recorded. Do the
different breeds show different levels of curiosity as
measured by novel object contacting? Use o = .05.
Single-Factor Independent Measures
ANOVA
Example
Long-Evans Sprague-Dawley Feral
3 4 5
4 2 6
2 4 7
4 3 6
2 2
2
T
1
= 17 T
2
= 15 T
3
= 24
n
1
= 6 n
2
= 5 n
3
= 4
M
1
= 2.83 M
2
= 3 M
3
= 6
k = 3
N = 15
G = 56
EX
2
= 248
Single-Factor Independent Measures
ANOVA
Example
Step 1: State the Hypotheses
H
0
: The different types of rats do not differ in the
number of object contacts.
H
a
: The different types of rats do differ in the
number of object contacts.

H
0
:
LE
=
SD
=
F
H
a
: Not all s are equal
Single-Factor Independent Measures
ANOVA
Example
Step 3: Compute the Test Statistic
9333 . 38
15
56
248
8333 . 10
4
24
5
15
6
17
248
1 . 28
15
56
4
24
5
15
6
17
2 2
2
2 2 2 2
2
2 2 2 2 2 2
= = E =
=
|
|
.
|

\
|
+ + = E =
=
|
|
.
|

\
|
+ + = =

N
G
X SS
n
T
X SS
N
G
n
T
SS
Total
within
between
Source SS df MS F
obt
Between 28.1
Within 10.83
Total 38.93
Single-Factor Independent Measures
ANOVA
Example
Step 3: Compute the Test Statistic
14 1 15 1
12 3 15
2 1 3 1
= = =
= = =
= = =
N df
k N df
k df
total
within
between
Source SS df MS F
obt
Between 28.1 2
Within 10.83 12
Total 38.93 14
Single-Factor Independent Measures
ANOVA
Example
Step 3: Compute the Test Statistic
9025 .
12
83 . 10
05 . 14
2
1 . 28
= = =
= = =
within
within
within
between
between
between
df
SS
MS
df
SS
MS
Source SS df MS F
obt
Between 28.1 2 14.05
Within 10.83 12 .9025
Total 38.93 14
Single-Factor Independent Measures
ANOVA
Example
Step 3: Compute the Test Statistic
57 . 15
9025 .
05 . 14
= = =
within
between
obt
MS
MS
F
Source SS df MS F
obt
Between 28.1 2 14.05 15.57
Within 10.83 12 .9025
Total 38.93 14
Single-Factor Independent Measures
ANOVA
Example
Step 2: Compute the Critical Value
F
crit
(2, 12)
o=.05
= 3.88

Single-Factor Independent Measures
ANOVA
Example
Step 4: Make a Decision about H
0
F
obt
is larger than F
crit
, therefore reject H
0
and
conclude that at least one mean is different
from the others.

However, we have no idea which mean is
significantly different from which other!
Single-Factor Independent Measures
ANOVA
Example

In step 4 we rejected H
0
, which means that not all
s are equal, at least one is different from the
others. However, we have no idea which one is
different from which other one. If the ns were the
same we would perform a Tukeys hsd test.
However, our ns are different sizes so we must
perform Fishers protected t-tests between each
of the possible combinations of the levels. The
probability of making one or more Type I errors in
all of these comparisons is equal to .05 (o).
Single-Factor Independent Measures
ANOVA
Example

Long-Evans vs. Sprague-Dawley
Step 1: H
0
:
LE
=
SD
H
a
:
LE

SD
Step 2: t
crit
(12) = 2.179

Step 3:


Step 4: Fail to reject H
0
, Long-Evans rats do not
significantly differ from Sprague-Dawley rats in number
of object contacts.
296 .
5
1
6
1
9025 .
3 83 . 2
1 1
=
|
.
|

\
|
+ -

=
|
|
.
|

\
|
+ -

=
SD LE
within
SD LE
obt
n n
MS
X X
t
Single-Factor Independent Measures
ANOVA
Example

Long-Evans vs. Feral
Step 1: H
0
:
LE
=
F
H
a
:
LE

F
Step 2: t
crit
(12) = 2.179

Step 3:



Step 4: Reject H
0
, Long-Evans rats contact significantly
fewer objects than Feral rats.
169 . 5
4
1
6
1
9025 .
6 83 . 2
1 1
=
|
.
|

\
|
+ -

=
|
|
.
|

\
|
+ -

=
F LE
within
F LE
obt
n n
MS
X X
t
Single-Factor Independent Measures
ANOVA
Example

Sprague-Dawley vs. Feral
Step 1: H
0
:
SD
=
F
H
a
:
SD

F
Step 2: t
crit
(12) = 2.179

Step 3:



Step 4: Reject H
0
, Sprague-Dawley rats contact
significantly fewer objects than Feral rats.
708 . 4
4
1
5
1
9025 .
6 3
1 1
=
|
.
|

\
|
+ -

=
|
|
.
|

\
|
+ -

=
F SD
within
F SD
obt
n n
MS
X X
t
Single-Factor Independent Measures
ANOVA
Example
Finally, we want to know whether the factor of different
species is an important determinant of curiosity so we
must compute the effect size (eta-squared or q2).



Roughly, 72.2% of the variance in novel object
contacts is predictable by knowing the species of rat.
722 .
93 . 38
1 . 28
2
= = =
total
between
SS
SS
q