What Fordham Can’t Say, But Does Anyway

Yesterday, the Thomas B. Fordham Institute released a report suggesting that the No Child Left Behind Act has encouraged schools to focus on the lowest-performing students and neglect the highest-performing. This is not an unreasonable hypothesis: National Assessment of Educational Progress data suggest it could be true, though the results are mixed and Tom Loveless, the author of the report’s NAEP analysis (the report also includes some interesting survey results), makes it clear that it is impossible to say what, if any, test-score changes have been caused by NCLB.

Unfortunately, the spin put on the “good news” in the report by Fordham president Chester Finn and vice president Mike Petrilli is not nearly as measured as Loveless’ caveat about NCLB. On National Review Online today, Finn and Petrilli write with total certainty that government-driven “standards and accountability” regimes have produced gains for low-performers.

“NCLB and state-level efforts to impose standards and accountability on the schools are plainly boosting the kids who need it most — surely a good thing,” they pronounce.

Rising achievement surely is a good thing. That government standards and accountability produced it, however, is far from sure.

First, compare the period that contains NCLB, which was passed in 2002, to score changes in the period preceding it. In reading, the lowest 10 percent of 4th grade performers saw a much bigger increase in scores immediately before 2002 than after, and 8th graders saw their scores drop under NCLB. In math, we have to start with 2003, the earliest testing year within the NCLB timeframe. Again, for the lowest performers, in both 4th and 8th grades scores increased faster in the period right before NCLB — 2000 to 2003 — than after.

Loveless notes in the report that it is impossible to be sure what effect NCLB had on math in the 2000 to 2003 period — where the fastest gains are seen — since NCLB was passed in 2002. He’s right. However, in light of long delays in issuing NCLB regulations, and the unlikelihood of a huge jump in just one year of NCLB, it is more reasonable not to ascribe improvements to the law than to give it credit. More importantly, one definitely cannot say, as Finn and Petrilli nonetheless do, that the law “plainly” has something to do with rising low-achiever scores.

To be fair, Finn and Petrilli say NCLB and “state-level efforts” — not just NCLB — boosted those scores. On what basis do they split credit?

In his analysis, Loveless examined states' NAEP score changes for the highest and lowest performers, controlling for whether or not states had their own standards and accountability regimes before NCLB. Unfortunately, the report doesn’t list which are considered “accountability” and which “non-accountability” states, so it is impossible to search for other common characteristics — charter schools, private-school-choice programs, increasingly affluent populations, new curricula — that could have driven states’ performances. Even more damaging to Finn and Petrilli’s pronouncement, the data the report does make available simple cannot support their all-too-firm-sounding conclusion.

For one thing, for the four subject-grade combinations presented, only between 34 and 37 states are analyzed, leaving out one-third of the country. More important, while in three of the four subject-grade combinations the lowest performers in states with accountability regimes did see greater score increases than low perfomers in states without them, when you only have four comparisons you simply cannot declare uncontestable victory, much less when only three of the comparisons support your conclusion. Change one, and you’ve got a coin flip. Unfortunately, that didn’t stop Petrilli and Finn.

In the end, Fordham’s new report doesn’t tell us anything definitive about the effect of NCLB or any other standards and accountability regimes. It offers some reason to believe that NCLB might help low scorers and leave high scorers behind — and it's well worth reading just for that — but it provides nothing close to proof. It also suggests that standards and accountability regimes might help raise low-performers’ scores, but again has far too many holes and far too little information to support what Finn and Petrilli declare: that government-imposed standards and accountability “plainly” help low-achieving kids.