Professional Documents
Culture Documents
David Lohman
Institute for Research and Policy on Acceleration
Belin-Blank Center
&
Iowa Testing Programs
University of Iowa
http://faculty.education.uiowa.edu/dlohman/
Topics
Distinguishing between ability & achievement
Overview of CogAT
Comparing CogAT with other ability tests
Interpreting CogAT scores
General issues in selection
Identification of talent in special populations
Combining Achievement, Ability, & Teacher
ratings: the Lohman – Renzulli matrix
Distinguishing between
ability and achievement
Puzzlements for common interpretations
of ability & achievement
Is ability more biologically based?
Most studies show same heritability for IQ (Gf) and
achievement tests (Gc)
Lower relative achievement than ability =
underachievement
But there are an equal number of “overachievers”
Status scores (IQ, PR) show good stability
But one must keep getting better to retain that IQ
Between 9 – 17 r(True IQ) = .75.
60% in top 3% at 9 NOT in top 3% at age 17
Fluid abilities invested in experience to produce
particular constellations of crystallized abilities?
Only for very young children
Thereafter, crystallized abilities -> fluid
Level 1. Nominalism (Most people here)
Ability Achievement
Level 2. Oh, Oh – there’s more overlap than
uniqueness here!
Its all ‘g’ (any indicant will do)
Its all just a product of experience
Preserve stage 1 beliefs –
Purge ability of visible achievement (e.g.
measure “process” or use only “nonverbal”
measures)
Ability Achievement
Level 3. Island kingdoms –Things get even
more complicated (most scholars of human abilities)
Effects of language, culture, and experience on
the development of ability (“All abilities are
developed” Anastasi)
Experience alters the structure of the brain
Mental processes do not exist independently of
knowledge.
Example of Flynn Effect
105
100
95
IQ on the 1995 Scale
90
85
80
75
70
1910 1920 1930 1940 1950 1960 1970 1980 1990 2000
Year
Low High
Fluid Crystallized
Basketball Football
Physical General Volleyball
physical Swimming
skills fitness
Field hockey Cycling
Wrestling
Fluid-Crystallized Continuum (1)
Fluid Crystallized
Science Social studies
achievement achievement Specific
Cognitive General fluid factual
ability (Gf) Math knowledge
abilities achievement Knowledge of
literature
Basketball Football
Physical General Volleyball
physical Swimming
skills fitness
Field hockey Cycling
Wrestling
Fluid-Crystallized Continuum (2)
Fluid Crystallized
Science Social studies
achievement achievement Specific
Cognitive General fluid factual
ability (Gf) Math knowledge
abilities achievement Knowledge of
literature
Basketball Football
Physical General Volleyball
physical Swimming
skills fitness
Field hockey Cycling
Wrestling
A common ability-achievement space
Level 4. Systems theories (A handful)
Form 5 – Hagen
No reading
Tests untimed (paced by teacher)
Mark directly in booklet
Multilevel Battery (gr. 3-12)
General Reasoning Ability
Tests timed
Separate Answer sheet
Common Directions
3 Separate Test
Batteries
(Not one)
Scores
Raw score = number correct
Scale score – USS
Within level - map number correct on to a scale
whose intervals are approximately the same size
Between levels – maps number correct on different
levels of the test on to a single, common,
developmental scale
USS Scale
D
etc
C
A
Relationships among Stanines, Percentile Ranks, and
Standard Age Scores
134 - 150
Composites
Composite scores
Partial VQ, VN, QN
Full – VQN or C [do NOT use for screening]
Primary Battery
V or (VQ) versus N
Multilevel Battery
V versus QN
Consequential Validity:
Score warnings
Age out of range
Age unusual for coded grade
Estimated test level
Level unusual for coded grade
Targeted score
Too few items attempted to score
Many items omitted (slow and accurate)
Extremely variable responses
Personal Confidence Intervals
Pattern of item responses aberrant?
Inconsistent across subtests within a battery?
Personal Standard Error of Measurement (PSEM)
SAS PR 1 25 50 75 99
V 120 89
Q 116 84
N 125 94
Score Profiles
CogAT 6 ‘ABC’ Profile system
Measuring the pattern
“A” profiles: Confidence bands overlap for all
three scores. Scores are at roughly the sAme level
SAS PR 1 25 50 75 99
V 120 89
Q 116 84
N 125 94
“B” Profiles
SAS PR 1 25 50 75 99
V 120 89
Q 116 84 N-
N 100 50
SAS PR 1 25 50 75 99
V 95 38
Q 92 31 N+
N 110 73
“C” Profile
SAS PR 1 25 50 75 99
V 120 89
V+ N-
Q 110 73
N 100 50
Extreme “C” Profile
SAS PR 1 25 50 75 99
V 120 89
Q 107 67
N 92 31
6 A
5 B (V+)
8 C (Q+ V-)
2 E (N+ V-)
CogAT6 Profile frequencies for students
in K-12 population
Percent in
K-12
Profile population
A 33
B 42
B+ ( 21)
B- (22)
E 7
B+ (4)
B- (3)
CogAT6 Profile frequencies for students in K-12
pop. and for students with two stanine scores of 9
Percent in Percent in
K-12 Stanine=9
Profile population group
A 33 37
B 42 27
B+ ( 21) ( 6)
B- (22) ( 21)
E 7 19 37%
B+ 4 ( 3)
B- 3 ( 16)
Comparing CogAT
with other tests
Reliability
Many estimates for a given test
Sources of error
Correlation versus standard error of
measurement (SEM)
Correlations depend on sample variability
Easily misinterpreted
SEM
Typical SD of distribution of test scores if
each student could be tested many times
Group-administered tests:
Inview
Otis-Lenon
NNAT
Conditional Standard Error of Meas.
20.00
15.00
SEM
USS Score
10.00
Raw Score
5.00
0.00
0 5 10 15 20 25 30 35 40 45 50 55 60 65
Number Correct
Conditional SEM's for CogAT6 Verbal USS scores, by test level
Gf Fluid Reasoning
Abilities
Carroll’s Three-Stratum Theory
of Human Abilities
100
95
IQ on the 1995 Scale
90
85
80
75
70
1910 1920 1930 1940 1950 1960 1970 1980 1990 2000
Year
20
Standard Deviation .
George (2001)
15
Naglieri &
Ronning (2000)
10
Bright Horizon
0
A B C D E F G
Test Level
True Versus Reported NAI Scores
by NNAT Test Level
350
V
O
Vocabulary Developmental Standard Score
C
300
A
B
99th %-tile
U 80th %-tile
L250 50th %-tile
A 20th %-tile
R 1st %-tile
Y
200
150
100
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade
Reading Vocab Across Grades
400
350
V
Vocabulary Developmental Standard Score
O
C 300
A
B 99th %-tile
U 80th %-tile
L 250 50th %-tile
20th %-tile
A
1st %-tile
R
Y
200
150
100
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade
Myths about adapting instruction
All students are pretty much alike
Every student is unique
Myths about adapting instruction
All students are pretty much alike
Every student is unique
Adaptations should be based on self-reported
learning styles
Myths about adapting instruction
All students are pretty much alike
Every student is unique
Adaptations should be based on self-reported
learning styles
If the method is right, the outcome will be good
Examples of correlations
Build on Strength
Focus on working memory
Scaffold wisely
Emphasize strategies
PR 1 25 50 75 99
V 67
Q 17
N 71
Profile 6E (Q-)
Primary uses of CogAT
To guide efforts to adapt instruction to the
needs and abilities of students
To provide an alternative measure of
cognitive development
To identify students whose predicted levels of
achievement differ markedly from their
observed levels of achievement
ITBS – CogAT correlation
High
CogAT
Low
Low High
ITBS
ITBS – CogAT correlation
High
CogAT
Low
Low High
ITBS
ITBS – CogAT correlation
High
CogAT
Low
Low High
ITBS
ITBS – CogAT correlation
ITBS only
Low
Low High
ITBS
Proportion of students identified by one
test also identified by the second test
Moderate Correlation
Unexpectedly
High Ach.
Expected
Level of Ach.
Achievement
B
Unexpectedly
Low Ach.
Ability
Predicting Ach vrs Flagging Ach-
Ability discrepancies
Who are the students (at any ach level) who are
most likely to improve if given new motivation
or instructional resources?
Reasoning Ability > Achievement
1. Underachievement
• poor effort, instruction, etc.
Visual Arts?
100
90
80
Mathematics Achievement
70
60
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Example
Example r = PR
r = .6 using .6
100
90
80
Mathematics Achievement
70
60
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Example
Example r = PR
r = .6 using .6
100
90
80
Mathematics Achievement
70
60
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Example r = .6
Imprecision of even high
correlations
Given r = .8
What is the likelihood that a student who scores
in 60-70th PR at Time 1 will scores in the 60-70th
PR at Time 2?
Lohman, D. F. (2003). Tables of
prediction efficiencies.
Lohman, D. F. (2003). Tables of
prediction efficiencies.
Lohman, D. F. (2003). Tables of
prediction efficiencies.
Proportion of students identified by
both tests
Figure 5. Plots of the effects of three rules: (a) high scores on test 1 and test 2; (b) high
scores on test 1 or test 2; and (c) high scores on the average of test 1 and test 2.
Screening tests
You administer a screening test to reduce the
number who must be administered the
admissions test
Assume a correlation of r = .6 between the two
tests
Assume students must score at the 95th PR or
higher on the admissions test
What cut score on the screening test will include
all of those who would meet this criterion?
Proportion of students in top X percent of screening test who
exceed the same or a more stringent cut score on follow up
test
r = .6
Admissions test
Top x % 5% 3% 1%
30% 0.80 0.84 0.91
25% 0.75 0.80 0.87
Screening Test
Top x % 5% 3% 1%
30% 0.80 0.84 0.91
25% 0.75 0.80 0.87
Screening Test
Grades 3 – 12
Average CogAT V and ITBS Reading Total
Average CogAT QN and ITBS Math Total
Or
Quant/NV
Ability
(ALL
Students)
Teacher Rating (Renzulli Scales) on
Learning ability, Motivation, or
Creativity
Quant/NV >80th PR
Ability
Teacher Rating (Renzulli Scales) on
Learning ability, Motivation, or
Creativity
http://faculty.education.uiowa.edu/dlohman
NCE Scores
Get from the publisher for CogAT
Table look up (Table 32 in CogAT Norms
Manual)
Convert PR’s to NCE scores
In Excel
NCE = NORMINV (PR/100, 50, 21.06)