Calculating a Generalized Kappa Statistic for Use With Multiple Raters

Calculations based on equations presented in Fleiss (Statistical Methods for Rates and Proportions , 1981, pp. 229-232)
(Copyright 2004, 2009 Jason King, Wallace Judd. All rights reserved. Available at http://www.ccitonline.org/jking/homepage/kappa1.xls)

Directions: Enter rating data in the Rater1, Rater2, Rater3, Rater4, and Rater5 worksheets, as needed.
Edit the shaded numbers below to indicate the number of raters, items and categories.
Enter # of raters (m):
Enter # of items (n):
# of categories (k):

5
100
6

=N_Raters
=N_Items

0.885

= Generalized Kappa for Rater 1, Rater n data

The table below summarizes the data entered in the corresponding spreadsheets. Highlighting indicates areas of disagreement.
Items
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

CAT1
0
0
0
0
0
0
0
4
0
0
0
0
0
0
0
0
0
0
0

CAT2
0
0
5
0
0
0
0
1
4
0
0
0
0
1
0
0
0
0
4

CAT3
5
0
0
1
0
0
0
0
1
0
5
0
0
3
0
0
5
4
0

CAT4
0
1
0
4
0
0
0
0
0
0
0
5
0
1
0
5
0
0
0

CAT5
0
4
0
0
0
5
5
0
0
5
0
0
5
0
0
0
0
1
1

CAT6
0
0
0
0
5
0
0
0
0
0
0
0
0
0
5
0
0
0
0

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
5
0
1
0
4
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0

0
0
5
0
0
0
0
0
0
1
5
1
0
0
0
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0
5
0
5
1
0
0

0
4
0
4
0
0
5
0
5
4
0
4
0
0
0
5
0
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
4
0
0

0
1
0
1
0
5
0
0
0
0
0
0
5
5
5
0
0
0
4
0
1
0
0
0
0
5
5
0
0
0
0
0
0
0
0
5

0
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
0
0
0
5
0
5
0
0
0
0
1
0
0
0
0
0
0

5
0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
0
5
0
5
0
0
0
0
0
0
0
0
5
4
0
5
0
0
5
0

56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

0
0
0
0
0
0
4
0
5
0
0
5
0
0
0
0
1
0
0
0
0
5
5
5
0
0
0
0
0
0
0
0
0
0
0
0

0
0
5
5
0
5
0
0
0
0
0
0
0
5
0
0
0
4
0
0
0
0
0
0
0
0
0
5
1
0
5
0
0
4
5
5

0
5
0
0
4
0
0
5
0
0
0
0
0
0
0
0
4
0
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
0
0

5
0
0
0
1
0
1
0
0
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
5
0
0
0
0
0
5
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0
5
0
0
5
0
0
0
0
5
0
0
0
0
0
0
0
4
5
0
5
0
1
0
0

0
0
0
0
0
0
0
0
0
5
5
0
0
0
0
0
0
1
5
5
0
0
0
0
0
5
0
0
0
0
0
0
0
0
0
0

92
93
94
95
96
97
98
99
100

5
5
0
0
0
0
0
0
0

0
0
0
0
4
5
0
0
0

0
0
0
5
1
0
0
0
4

0
0
0
0
0
0
0
0
1

0
0
5
0
0
0
0
0
0

0
0
0
0
0
0
5
5
0
Sum_x2

Total # of ratings
prop (p)
******************
BY CATEGORY
num = sum_xi*(m-xi)
den = nm(m-1)pq
gen kappa_cat1 =

59
20
0.118

101
44
0.202

97
54
0.194

81
40
0.162

77
24
0.154

85
8
0.170

1.000

1.000

1.000

1.000

1.000

1.000

CAT1
20
208.2
0.904

CAT2
44
322.4
0.864

CAT3
54
312.7
0.827

CAT4
40
271.5
0.853

CAT5
24
260.6
0.908

CAT6
8
282.2
0.972

******************
OVERALL
num = nm2 - sum_x2
den = nm(m-1)sum_pq
gen kappa =
(p(e))
(sum p3)

190
1657.6
0.885
0.171
0.030

SEFleiss1a
z=
p calc =

0.015
60.723
0.000000

SEFleiss2b
z=
p calc =

0.014
61.897
0.000000

2310
500
1.000

=Prod_Cats
=Sum_Cats
Row = Cat_Props

CILower =
CIUpper =

0.857
0.914

CILower =
CIUpper =

0.857
0.913

This approximate standard error formula is based on Fleiss (Psychological Bulletin, 1971, Vol. 76, 378-382)

This approximate standard error formula is based on Fleiss, Nee & Landis (Psychological Bulletin , 1979, Vol. 86, 974-977)

Rater 1 Classifications
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Cat
3
5
2
4
6
5
5
1
2
5
3
4
5
3
6
4
3
3
2
6
3
2
3
6
4
3
5
3
3
2
3
4
4
4
3
1
6
4
6
1

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

2
5
3
5
4
4
1
6
6
2
6
2
3
6
4
4
3
2
2
3
2
1
3
1
6
6
1
5
2
4
5
3
2
6
6
5
1
1
1
3
6
4
2

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

5
5
2
5
4
2
2
2
1
1
5
3
2
2
6
6
3

Rater 2 Classifications
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Cat
3
4
2
4
6
5
5
1
3
5
3
4
5
2
6
4
3
3
5
6
4
2
3
6
4
3
5
3
3
2
2
4
4
4
3
1
6
1
6
1

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

2
5
3
5
4
4
1
6
5
2
6
2
3
6
4
4
3
2
2
4
2
1
3
1
6
6
1
5
2
4
5
3
6
6
6
5
1
1
1
3
6
4
2

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

2
5
2
5
4
2
2
2
1
1
5
3
2
2
6
6
4

Rater 3 Classifications
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Cat
3
5
2
3
6
5
5
2
2
5
3
4
5
4
6
4
3
5
2
6
3
2
4
6
4
3
5
3
2
2
3
4
4
4
3
1
6
4
6
4

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

2
5
3
5
4
4
1
6
6
2
6
2
2
6
4
4
3
2
2
3
2
4
3
1
6
6
1
5
2
4
5
1
2
6
6
5
1
1
1
3
6
4
2

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

5
5
2
5
4
5
2
2
1
1
5
3
3
2
6
6
3

Rater 4 Classifications
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Cat
3
5
2
4
6
5
5
1
2
5
3
4
5
3
6
4
3
3
2
6
3
2
3
6
4
3
5
3
3
2
3
4
4
4
3
1
6
4
6
1

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

2
5
3
5
4
4
1
6
6
2
6
2
3
6
4
4
3
2
2
3
2
1
3
1
6
6
1
5
2
4
5
3
2
6
6
5
1
1
1
3
6
4
2

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

5
5
2
5
4
2
2
2
1
1
5
3
2
2
6
6
3

Rater 5 Classifications
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Cat
3
5
2
4
6
5
5
1
2
5
3
4
5
3
6
4
3
3
2
6
3
2
3
6
4
3
5
3
3
2
3
4
4
4
3
1
6
4
6
1

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

2
5
3
5
4
4
1
6
6
2
6
2
3
6
4
4
3
2
2
3
2
1
3
1
6
6
1
5
2
4
5
3
2
6
6
5
1
1
1
3
6
4
2

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

5
5
2
5
4
2
2
2
1
1
5
3
2
2
6
6
3