The Cato Institute’s Andrew Coulson, who prepared the graph that I posted a few days ago, has responded that I’m wrong in saying that school reform can produce only marginal improvements in test scores, citing Ben Chavis’s accomplishments with some charter schools in California as proof to the contrary.
A couple of things that we aren’t arguing about: First, I’d send my own children to the American Indian Public Charter School that Ben Chavis ran. The descriptions I’ve read (I can’t put my fingers on the full-scale profile I read, despite Googling) convince me that he’s a great principal and ran a fine school. Second, I was referring to marginal test score improvements on a large scale, not what is possible in a single terrible school. But only a few percent of the nation’s students attend the worst-of-the-worst schools, and national or even state-wide test score results are driven by the huge majority of students who attend schools in the normal range, where improvements are modest.
However, I do disagree with Coulson in an important respect, and I might as well lay it out: I have yet to see a single, solitary example of a school that produced dramatic improvements in test scores that stood up to scholarly examination. They always have evaporated—not completely, but down to small puddles. Here are the questions that a scholarly examination must ask:
1) Were the tests conducted by the same people who reported the results?
2) Were the students tested representative of the entire student population (or were certain kids mysteriously absent that day)?
3) Are the results compared to those of a legitimate control group?
4) Were there practice effects from teaching to the test?
5) Has attrition been taken into account? (The failure to include the subsequent performance of the kids who dropped out of the program or school is usually the single most decisive artifact of inadequate evaluations.)
6) Was there a test for fadeout two or three years after the exit test? (Fadeout of initial results has been universal when such tests have been conducted.)
My prediction: the miraculous results of Ben Chavis will also evaporate down to marginal improvements when all of those questions have been answered about his schools. I’m not saying that he did a bad job or didn’t produce better education for his students, or that he’s undeserving of our applause. But, just as TINSTAAFL, TINSTAMTSI.
Okay, Andrew—and Rick Hess, Jay Greene, and Paul Peterson, and my other friends and colleagues who think I’m a grinch on this: give me the citation that proves me wrong.