Yesterday, the New York Times ran a front-page story purporting to show that "betting big" on charters has produced "chaos" and a "glut of schools competing for some of the nation’s poorest students." (One wonders how many of those low-income families are upset that they have "too many" options.). However, the article's central claim about charter school performance rests on a distorted reading of the data.

The piece claims that "half the charters perform only as well, or worse than, Detroit’s traditional public schools." This is a distortion of the research from Stanford University's Center for Research on Education Outcomes (CREDO). Although the article actually cites this research -- noting that it is "considered the gold standard of measurement by charter school supporters across the country" -- it only does so to show that one particular charter chain in Detroit is low performing. (For the record, the "gold standard" is actually a random-assignment study. CREDO used a matching approach, which is more like a silver standard. But I digress.) The NYT article fails to mention that the same study found that "on average, charter students in Michigan gain an additional two months of learning in reading and math over their [traditional public school] counterparts. The charter students in Detroit gain over three months per year more than their counterparts at traditional public schools."

As shown in this table from page 44 of the CREDO report, nearly half of Detroit's charter schools outperformed the city's traditional district schools in reading and math scores, while only one percent of charter schools performed worse in reading and only seven percent performed worse in math.

Grouping the very few underperforming charters with the approximately half of schools that perform at roughly the same level as the district schools distorts the picture. It's just as fair to say that more than nine out of ten Detroit charters performed as well or better than their district school counterparts. The most accurate description would note that about half of Detroit's charters outperform their district school counterparts, about half perform roughly the same, and a very small number underperform.

According to CREDO's 2015 nationwide study, 60 percent of charter schools outperform their district school competition in math and 51 percent outperform the district schools in reading. By contrast, the district schools outperform only 8 percent and 4 percent of Detroit charters in math and reading, respectively. The following two charts from pages 29 and 31 of the report show comparisons of charter school performance in various cities against "the alternative schooling options their students face" (i.e., the nearby district schools to which students would otherwise be assigned). 

In other words, the best available research on Detroit's charter school sector shows almost exactly the opposite of what the NYT piece portrayed. Indeed, as Professor Jay P. Greene of the University of Arkansas noted: 

To claim that half the charters perform the same or worse than traditional public schools is a grotesque distortion of the study’s findings. [...] [I]f the reporter cites that research to demonstrate that one charter management organization has sub-par performance, it is journalistic malpractice not to mention the positive overall results.  And those positive overall results contradict the very foundation of the entire article.

The NYT reporter, Kate Zernike, took to Twitter to defend her reporting against Greene’s takedown, citing data from Excellent Schools Detroit. However, as Greene explained, those data do not allow for direct comparisons. Zernike is right that the data show the citywide averages in each sector, but looking at the averages is misleading.

The charter schools tend to be mission-based schools that open in the toughest areas and serve the most at-risk students. Comparing city-wide averages fails to take that into account. It would be like comparing the New England Patriots against a championship high school team and concluding that the teenagers are superior athletes because they scored more touchdowns per game.  

The appropriate comparison is between the charters and the district schools that serve the same or similar student populations. That is what the CREDO study attempted to do by matching students with similar characteristics and initial test scores in each sector, then tracking and comparing them.

Zernike is still claiming that the CREDO study "does not consider Detroit['s charter sector] stellar," even though both the 2013 CREDO study of Michigan's charter sector and the 2015 CREDO study of charters nationwide found that, on average, Detroit's charter schools outperformed the district schools that their students would otherwise have attended. Indeed, one even called Detroit's charter sector "a model to other communities."

Zernike is simply wrong.

For an even more detailed critique of the article, read Tom Gantert of the Mackinac Center for Public Policy here.