Society and Culture

Don’t Make the Impossible the Enemy of the Good

A recap for those who haven’t been following the saga: Andrew Coulson at Cato has been arguing that the right schools can transform students, with kids from the inner city suddenly performing at suburban levels, and I’ve been saying bah, humbug.

Now Andrew, with Ben Chavis’s blessing, has thrown down the gauntlet, challenging me to see if my pessimism about the dramatic claims made for American Indian Public Charter School in Oakland, run by Ben Chavis, stands up in the face of the evidence.

Okay, Andrew, here’s what I could do:

Option 1. Next time I’m in the Bay area, I could visit American Indian School. I’m pretty sure I know what I would find—the kind of principal and school that led me to say in our original exchange that I would send my own children to American Indian School. I have no doubt that Ben Chavis is giving his students a better educational experience than they would get at any other public school in Oakland. That would take a couple of days of my time, and I’m sure I would enjoy Ben Chavis’s company.

Option 2. I could be principal investigator of an evaluation designed to answer this question: How much effect does Ben Chavis’s school have on quantitative measures of academic ability? This would take at least six months of my time, over an elapsed research schedule of a year or so, and I would require a research support staff to do it and funding (even if I contributed my own time pro bono) in six figures.

Why can’t I spend a few days in the school’s offices, take a look at the school’s records and test scores, and get an answer quicker? Because it doesn’t work that way. If you go back to the criteria for convincing evidence I listed in an earlier post, it’s obvious why. Meeting those criteria requires time-consuming data collection of existing records, verification of information through sources independent of the institution being evaluated, investigation of the self-selection factors at work in recruitment, follow-up to find out what happened to students who dropped out of the program, the administration of additional tests to answer questions that California’s tests don’t answer (e.g., tests of the level of cognitive ability that the students bring with them to the school, and follow-up tests for students who have been out of the program for a few years), and sophisticated data analysis.

Here’s the problem: If I do Option 1, I won’t be in a position to say yea or nay about the claims that you make for Ben Chavis’s accomplishments. If I do Option 2, it is possible that I will vindicate your high expectations, which would be great news and confound my own pessimism. I would be delighted to trumpet those findings from the rooftops. But suppose it turns out that the effects of American Indian School on the quantitative measures are of the magnitude that characterize the literature on successful programs—on the order of 0.15–0.25 standard deviations on the exit test, diminishing in two-year or three-year follow-up tests to near zero. I would have gone to great lengths to discredit claims for a school that I will continue to think is an excellent school. Ben Chavis’s school should not be judged a failure because he fails to do what no individual school in the history of education has ever done—in Andrew’s touchingly naïve phrase, shift the bell curve dramatically to the right.

I would be glad to serve on a panel that designs the evaluation I describe for Option 2. But I believe in my heart of hearts that the evaluation will end up making life more difficult for American Indian School, not help it, and producing such an evaluation report is not the way that I want to spend a significant chunk of my remaining professional life.

Comments are closed.