An astonishing act of statistical chutzpah in the Indiana schools grade-changing scandal.

By Jordan Ellenberg|Posted Friday, Aug. 2, 2013, at 12:27 PM http://www.slate.com/articles/news_and_politics/do_the_math/2013/08/tony_bennett_education_an_astonishing _act_of_statistical_chutzpah_in_the.single.html Florida Education Commissioner Tony Bennett resigned Thursday amid claims that, in his former position as superintendent of public instruction in Indiana, he manipulated the states system for evaluating school performance. Bennett, a Republican who created an A-to-F grading protocol for Indiana schools as a way to promote educational accountability, is accused of raising the mark for a school operated by a major GOP donor. Bennett calls that charge, which arose from emails among Bennetts staff obtained by the AP, malicious and unfounded and frankly so off base. He offered a different explanation for why the grades for Indianapolis charter school Christel House Academywhose mark soared from a C to an Aand 12 other schools were changed at the last minute. According to Bennett, he was just correcting a simple math mistake. Bennetts explanation is perfectly mathematically reasonable, and it would get him off the hook. The only problem is that the story hes telling appears to be totally false. Bennett told AEIs Rick Hess, As we were looking at the grades we were giving our schools, we realized that state law created an unfair penalty for schools that didn't have 11th and 12th grades. Statewide, there were 13 schools in question had unusual grade configurations. The data for grades 11 and 12 came in as zero. When we caught it, we fixed it. Bennetts stated rationale makes sense. Heres an analogy. Youre teaching a course with three exams, and each students overall exam grade for the class is computed as an average of her three individual exams (1/3) * (exam 1) + (1/3) * (exam 2) + (1/3) * (exam 3) But what happens if a student misses exam 3, with a justified absence, and the test cant be made up? Then we have whats called a missing data problemwe have to infer something about the students performance without the full complement of data we have for everybody else. One natural approach is to compute that students overall exam grade as the average of the exams she did take: (1/2) * (exam 1) + (1/2) * (exam 2). What you shouldnt do is give the student a zero for the exam and average it into her grade. Christel House had started out as a middle school and was adding one high school grade each year; for the 2011-12 school year, it served students through the 10th grade. High schools, according to Indiana statute, were to be graded on four metrics, averaged like so: 30% * (English scores) + 30% * (algebra scores) + 30% * (graduation rate) + 10% * (college and career readiness score) But, as Bennett said in an official statement, Christel House only served students in grades K-10, thus the graduation rate and college and career readiness measures could not be calculated because the school did not serve grades 11 and 12. So it faced a missing data problem, like the student who had to miss an exam.

Indeed, it wouldnt be fair to count those scores as zero! The obvious fix, just as with the student, is to grade such a school on the two scores that it does have: 50% * (English scores) + 50% * (Algebra scores) Does that mean Bennett really did get railroaded, and he was just fixing an obvious error? Nobecause the fixed version of the grade was what Indiana was already using before Bennett started tinkering with the gears. When you dig into the numbers, the story about the unfair zeroes looks like a complete fabrication. Christels ninth- and 10th-grade students got brutally low scores on the English and math tests, with only 70 percent passing English and just a third passing math. Heres how Jon Gubera, then the Education Departments chief accountability officer, described Christels performance in one of the emails released by the AP: OK, here is their breakdown ... They served grades K-10 in 2011-12 so they are a combined school but do not have any graduates. So their grade is a combination of the standard E/MS model and the HS 9&10 model which only counts ECA proficiency. E/MS results: 3.00 on E/LA (no growth bonuses) and 4.00 on math (bottom 25% bonus) = 3.50 points (B) HS results: 2.00 on E/LA (70% pass rate) and 0.00 on math (33% pass rate) = 1.00 points (D) Final Combined results: E/MS 3.50 x .76 (76% of school is in grades 3.8) = 2.66 + HS 1.00 x .24 (24% of school is in high school) = .24. Thus overall grade is 2.66 + .24 = 2.9 (C). Bottom line: their terrible 10th grade Algebra I results (33% passing) was the principal factor in earning a C grade." Christels grade 38 scores came to a 3.50, or a B, and their high school scores, thanks to the algebra fiasco, were a 1.00, or a D. The schools score is then an average of the grade 38 scores and the 910 scores, weighted according to the proportion of students in each group. Where are the zeroes for the graduation rates and readiness score that got averaged into Christels scorethe unfair penalty that Bennett claims he fixed? Theyre not there, because thats the part that Bennett, as far as I can tell, simply made up. Guberas computation wasnt satisfactory for Bennett, because it didnt give Christel an A, andjudging by the emails obtained by the APhe had already told Christel administrators they were getting an A. No mere numbers were going to make a liar out of him! Whats more, Bennett wrote in one of the emails, the low grade for Christel meant legislative leadership as well as our critics of A-F are going to use this against us to undo our accountability metrics through legislation. He went on: If you cant tell, I am more than a little miffed about this. I hope we come to the meeting today with solutions and not excuses and/or explanations for me to wiggle myself out of the repeated lies I have told over the past 6 months.

Bennetts staff swung loyally and swiftly into action; within a day they had found what they called a loophole in the state law that allowed them to change Christels grade. The statute clearly states that high schools without 12th-graders get a score made up half of the English score and half of the algebra score. Whats more, the law said schools that combined high school grades and lower grades should use a weighted average of the elementary/middle school measures and the four high school measures. Heres where Bennetts team found the loophole big enough to drive a charter school through. A normal person would do exactly what Chief Accountability Officer Jon Gubera didgive Christel the weighted average of its elementary/middle school score, according to the rules for elementary/middle schools, and its high school score, according to the rules for high schools. But Bennett had a better idea. Christel was, technically speaking, not a high school, so the statutory formula for the high school grades didnt apply. But it also didnt have all four high school measures, so, he argued, the rules for combined schools didnt apply either. There were just 13 schools in the state that had both middle school and high school grades but no seniors. For these schools, Bennett reasoned, the Indiana education poobahs should have a free hand to set the grades however they pleased. You can guess what happened next: Bennett ruled that the ninth- and 10th-graders in these schools didnt count at all. So it was that the offending algebra grades vanished in a puff of bureaucratic smoke. (Anne Hyslop at Ed Money Watch has an even more detailed accounting of the process, if you like watching sausage get made.) This was an act of astonishing statistical chutzpah. Suppose the syllabus for my math class said that the final grade would be determined by averaging the homework grade and the exam grade, and that the exam grade was itself the average of the grades on the three tests I gave. Now imagine a student gets a B on the homework, gets a D-minus on the first two tests, and misses the third. She then comes to me and says, Professor, your syllabus says the exam component of the grade is the average of my grade on the three testsbut I only took two tests, so that line of the syllabus doesnt apply to my special case, and the only fair thing is to drop the entire exam component and give me a B for the course. I would laugh her out of the office. Or maybe suggest that she apply for a job as a state superintendent of instruction. The saddest part is that Im guessing Bennett sincerely felt he was doing the right thing. In his mind, he knew Christel was a great school, so if the scores said otherwise, the scores had to be wrong. In this respect, ironically, he ends up echoing his policy opponents, adopting the position that a mechanistic testing and scoring procedure cant be allowed to override firsthand knowledge about teachers and schools. Saying this out loud wasnt an option, so any test scores that seemed to indicate learning problems at Christel had to be eliminated from the spreadsheet with extreme prejudice. (I attempted to reach both Tony Bennett and Jon Gubera and have not received a reply from either man. If I do hear back, I will add an update to this story.) Bennett told AEIs Hess, I'm a track and field guy. I run, I try to keep my weight down at about 190. Christel has been a track-performing school for a number of years. If I get on the scale one day, am doing everything the same, and am still wearing my same clothes and they fit, and the scale suddenly reads 215, I am going to question what's going on. But Christel wasnt doing everything the same. It didnt have a 10th grade before, and then it did, and whatever it was doing to teach those new 10th-graders math, it didnt lead them to pass algebra. Bennett could, and should, have faced up to that fact, given the school the C it earned, and delivered them some honest tough love: Youre still doing well at the stuff youve always done well, but you obviously havent succeeded at the new stuff youre trying to do. We believe you can do it, but until you do, your grades going to suffer. Isnt that what accountability in education means, if it still means anything at all?

Do the Math: Christel Houses Grade Doesnt Add Up

Author(s): Anne Hyslop Published: July 31, 2013 Issues: Data and Data Systems School Accountability School Choice Education Mel Horowitz: You mean to tell me that you argued your way from a C+ to an A-? Cher: Totally based on my powers of persuasion, you proud? Mel Horowitz: Honey, I couldn't be happier than if they were based on real grades. Turns out weve all been Clueless when it comes to Indianas A-F school grades. Former Indiana (and current Florida) schools chief Tony Bennett has been under fire for released emails that show he and officials at the Indiana Department of Education altered the grades for certain schools prior to the very-public release of the new accountability measures last fall. Whats particularly worrisome is that the change to the grading methodology wasnt so public. In fact, it was never announced. And from the emails obtained by AP reporter Tom LoBianco, its clear that Christel Houses initial grade set off a firestorm of panic at the IN DOE. In a press call and separate interview with AEIs Rick Hess, Bennett explained the matter by saying that Christel House Academy and a dozen other schools were unfairly penalized due to their unconventional grade configurations. Because they didnt serve students in grades 11 or 12, these schools were missing key data elements for the high school calculation namely, graduation rates and college readiness indicators, which typically count for 40 percent of the high school model. In Bennetts words: The backstory is simple here, Rick. In our first run of the new school calculations in Indiana, we turned up an anomaly in the results. As we were looking at the grades we were giving our schools, we realized that state law created an unfair penalty for schools that didn't have 11th and 12th grades. Statewide, there were 13 schools in question had unusual grade configurations. The data for grades 11 and 12 came in as zero. When we caught it, we fixed it. That's what this is all about. Because Christel House was a K-10 school, the systems essentially counted the other two grades as zeroes. That brought the school's score down from an "A" to a "C". Turns out its not quite that simple. The state has several variations of its grading rubric to apply to different school situations and set-ups. The basic models are 1) elementary and/or middle school grades and 2) high school grades. Then, there is a combined model for schools that have students in grades preK-8 and grades 9-12 like Christel House, which served students through 10th grade in 2011-12. The grade point averages for the 38 portion of the school and the 9-12 portion of the school are weighted according to the percentage of enrolled students in each grade span to arrive at one final, combined grade. (The final scale: 3.51 4.00 points = A; 3.00 3.50 points = B; 2.00 2.99 points = C; 1.00 1.99 points = D; 0.00 0.99 points = F) Within the two basic models (ES/MS and HS), there are also deviations for special circumstances. Typically high school grades are calculated with a 60% weight on proficiency in end-of-course exams in Algebra I and English 10 (with potential bonus points for increases in proficiency rates from grades 8-10 and grades 10-12), 30% weight on graduation rates, and 10% weight on college readiness indicators. But some high schools are given special consideration: small schools, HS feeder schools (grade 9 only), 9-10 schools, and 11-12 schools.

In the 9-10 model, proficiency rates make up the entire school grade, split evenly between Algebra I and English 10, and the bonus points do not apply. Confused yet? Bear with me. Christel House should have been evaluated using a mixture of two of the models: the 9-10 model and the combined ES/MS + HS model. Except they werent. Because Christel House wouldnt have gotten an A that way. In fact, one of the released emails walks through the calculation (using preliminary, rather than final, achievement data). Under this method, Christel House earned a C grade, a HUGE problem for us according to officials. And it set off the panic within the Indiana Department of Education at 2:30 in the morning on September 13. However, state officials soon that same day, in fact came upon a solution. Or in their words, a loophole, in the combined model calculation. Heres the original definition (as written in one of the emails): (j) A schools grade shall be determined by: (i) Multiplying the average of the ELA and Math points for the EMS grades by the percentage of all students (ii) Multiplying the sum of the four weighted scores for the high school by the percentage of students. Those three bold words contain the loophole Will Krebs, then Director of Policy and Research, found later that day dubbed option one. Because Christel House didnt have four weighted scores for its high school, the argument was that the combined school methodology was invalid. Without graduation rates and college readiness indicators, the school only had two of the four weighted components. Jon Gubera, Chief Accountability Officer, signed off on this option the following morning writing, Option one works. This would eliminate the HS points and ensure Christel House receives at least a B. So what does that mean, exactly? In truth, Christel House was never evaluated on its poor high school performance. Instead, all of the high school data were thrown out a little detail Bennett failed to mention. Christel Houses A is based on the ES/MS model only. As you can see below, Christel Houses grade was clearly inflated. The initial data run showed the school with a C grade. Using the combined methodology sans loophole with its final performance data, however, the school would have actually earned a B. Yet the school still received an A from the state and was treated as only having elementary and middle school grades. Further, there is no indication anywhere on the states school report card that Christel Houses grade fails to reflect the schools poor high school math performance.

According to the Indianapolis Star, Bennett refused to allow two regular public schools facing state takeover to use a similar "loophole" a year earlier. In both cases, poor middle school performance (where the school had recently expanded) penalized the high school. If their grades could not be separated, why was Bennett so eager to make an exception for Christel House?

These kinds of shenanigans are unacceptable and have chipped away at public faith in the legitimacy of school accountability systems over the last 10+ years of No Child Left Behind. Christel Houses grade is simply more false advertising from states and local districts that have a long history of finding loopholes in accountability systems and exploiting them. In fact, Indiana officials questioned whether using the loophole in this case would encourage other schools to adopt a grade 6-10 model to avoid accountability. Gubera replied: Not in the immediate if we dont advertise this everywhere. This just illustrates the problem. Christel House is an A school but only for its elementary and middle school program. Yet that isnt the story Bennett and his staff are telling. This grade inflation is particularly unfortunate in Indiana, where parents and families have a greater degree of school choice than in most states and rely on information like A-F grades to determine where to enroll their children. The thing is, Tony Bennett knows this: This kind of system has to make sense for the end user, in this case, the family Back in Indiana, we were trying to build a new system. It's an interesting parallel. My recommendation to the Florida board was, "If your system doesn't fully make sense, then how do you defend it?" If the results come out suspect, then, in the end, you can really question the integrity of the system. Commissioner Bennett, Christel Houses inflated grade is suspect, and Im questioning the integrity of the system. Accountability systems even those required from the U.S. Department of Education can be done right, but Tony Bennett unfortunately just made it that much harder to make the case for them. Note: To see option 1 in action for yourself, check out the attached speadsheet from Indiana's Office of Accountability. Christel House Academy appears on the Elementary/Middle School tab, but not on the High School or Combined School tabs.