It’s Not Camelot, ‘It’s Only a Model’

In Monty Python's Quest for the Holy Grail, the assembled knights look in awe upon the imposing walls of "Camelot"... until someone points out that "it's only a model."

I feel I'm watching a remake of Quest every time I read another blog post about the economics paper "Anti-Lemons" by MacLeod and Urquiola.

Matt Yglesias reproduced its abstract last month, saying "I would have to pay $5 to read the whole paper, but the abstract conveniently supports political positions I like, so I’ll talk about it some more." That, needless to say, isn't the sort of talk that calls for a thoughtful response.

But now that Megan McArdle has picked up the thread from a second Yglesias post, read the paper, and it given it a favorable verdict, it's time to point out that "it's only a model" -- and not a very good one at that.

"Anti Lemons" is not an empirical study. Instead it presents a series of abstract mathematical models with arbitrary assumptions. The final model purports to demonstrate the authors' conclusion that "For-profit entry turns out to be feasible, despite these assumptions, as long as private schools can cream skim the highest ability students from the public system."

What are the authors' assumptions?

i) individuals differ only with respect to innate ability

ii) all schools are equally productive

iii) for-profit schools must operate unsubsidized

The first two of these assumptions are nonsense and the third contravenes the whole point of a school choice program (whether tax credits or vouchers), which is to subsidize access to private schooling for those who could not otherwise afford it.

As if these problems were not enough, the model also incorrectly assumes that when academic selectivity is permitted, every private school will not only select students based on academic entrance tests, but that they will all use the same test. Like the others, this assumption is out of touch with reality. When I analyzed survey data for Arizona private schools in 2006, I found that nearly half of all private schools were not academically selective. Only a third actually administered an academic admissions test of any kind. The only admissions criteria applied by a majority of schools were measures of student and parent desire to attend the school and students’ and parents’ willingness to abide by its code of conduct.

So the MacLeod and Urquiola model has precious little to do with reality. It tells us nothing about the real world or about tax credit or voucher programs or proposals. In fact, it seems to serve no productive purpose whatsoever, unless one considers it productive to give left-wing bloggers a study abstract to talk about that "conveniently supports political positions [they] like."

Though MacLeod and Urquiola briefly discuss a modified model that relaxes the proscription against subsidization of private schools, its other erroneous assumptions remain and so it produces a result that is, not surprisingly, completely at odds with the reality established by the large body of empirical findings in this field.

Last year, I reviewed the worldwide literature comparing public and private schools (65 studies reporting 156 different statistical findings) and found that the statistically significant findings favor private schools by a margin of roughly 8 to 1. More importantly, when we focus more precisely and compare truly market-like school systems to monopolies such as U.S. public schooling, the statistically significant results favor markets by a margin of nearly 15 to 1 (and they greatly outnumber the insignificant findings as well). It is thus the least regulated private schools that show the most consistent advantage.

MacLeod and Urquiola mischaracterize that research literature as follows: "there is no consistent evidence that introducing choice substantially improves learning, or that private schools have higher value added than public ones." The sources they cite to back up their mischaracterization are both incomplete and imprecise, failing to look at a large swath of the research and failing to distinguish among various forms of "choice" with fundamentally different features.

So, no, the "Anti-Lemons" study is not the Camelot it is cracked up to be by recent rhapsodic blog posts. It's not even a good model.

[Should anyone want to interject Hsieh and Urquiola's 2006 empirical study of the highly regulated Chilean voucher system at this point, I've already offered my thoughts on it here.]