Evidence‐​based for Thee, But Not for Me

One of the things that strikes me as curious about supporters of the No Child Left Behind Act is that they talk regularly about “evidence” and having everything be “research-based,” yet they often ignore or distort evidence in order to portray NCLB as a success. Case in point, an op-ed in today’s New York Times by the Brookings Institution’s Tom Loveless and the Fordham Foundation’s Michael Petrilli.

Truth be told, the piece doesn’t lionize NCLB, criticizing the law for encouraging schools to neglect high-performing students because its primary goal is to improve the performance of low achievers. Fair enough. The problem is, Loveless and Petrilli assert with great confidence that the law is definitely doing the job it was intended to do. “It is clear,” they write, “that No Child Left Behind is helping low-achieving students.”

As you shall see in a moment, that is an utterly unsustainable assertion according to the best available evidence we have: results from the National Assessment of Educational Progress, which carries no consequences for schools or states and, hence, is subject to very little gaming. Ironically, Loveless and Petrilli make their indefensible pronouncement while criticizing a study for failing to use NAEP in reaching its own conclusions about NCLB.

So what’s wrong with stating that NCLB is clearly helping low-achieving students? Let me count the ways (as I have done before):

1. Numerous reforms, ranging from class-size reduction, to school choice, to new nutritional standards, have been occurring at the same time as NCLB. It is impossible to isolate which achievement changes are attributable to NCLB, and which to myriad other reforms
2. As you will see in a moment, few NAEP score intervals start cleanly at the beginning of NCLB – which is itself a difficult thing to pinpoint – making it impossible to definitively attribute trends to the law
3. When we look at gains on NAEP in many periods before NCLB, they were greater on a per-year basis than during NCLB. That means other things going on in education before NCLB were working just as well or better than things since the law’s enactment.

So let’s go to the scores. Below I have reproduced score trends for both the long-term and regular NAEP mathematics and reading exams. (The former is supposed to be an unchanging test and the latter subject to revision, though in practice both have been pretty consistent measures.) I have posted the per-year score increase or decreases above the segments that include NCLB (but that might also include years without NCLB). I have also posted score increases in pre-NCLB segments that saw greater improvements than segments including NCLB. (Note that on 8^th-grade reading I didn’t highlight pre-NCLB segments with smaller score decreases than seen under NCLB. I didn’t want to celebrate backward movement in any era.)

For context, NCLB was signed into law in January 2002 but it took at least a year to get all the regulations written and more than that for the law to be fully implemented. As a result, I’ll leave it to the reader to decide whether 2002, 2003, or even 2004 should be the law’s starting point, noting only that this problem alone makes it impossible to say that NCLB clearly caused anything. In addition, notice that some of the biggest gains under NCLB are in periods that also include many non-NCLB years, making it impossible to confidently attribute those gains to NCLB.

Please note that I calculated per-year changes based on having data collected in the same way from start to end. So some lines are dashed and others solid (denoting changes in how some students were counted); I calculated changes based on start and end points for the type of line used for the period. I also rounded to one decimal point to save space. Finally, I apologize if this is hard to read—I’m no computer graphics wizard—and would direct you to NAEP’s website to check out the data for yourself.

4th Grade Regular Math

8th Grade Regular Math

4th Grade Regular Reading

8th Grade Regular Reading

Age 9 Long-term Math

Age 13 Long-term Math

Age 17 Long-term Math

Age 9 Long-term Reading

Age 13 Long-term Reading

Age 17 Long-term Reading

So what does the data show us? First, that there were numerous periods that didn’t include NCLB that saw greater or equal growth for low-achieving students as periods with NCLB. That means much of what we were doing before NCLB was apparently more effective than what we’ve been doing under NCLB, though it is impossible to tell from the data what any of those things are. In addition, it is notable that those periods with the greatest gains that include NCLB are typically the ones that also include non-NCLB years, such as 2000 to 2003 for 4^th and 8^th-grade math. That means there is inescapable doubt about what caused the gains in those periods most favorable to NCLB. And, let’s not forget, 4^th -grade reading saw a downward trend from 2002 to 2003, and 8^th-grade reading dropped from 2002-2005. That suggests that NCLB was actually decreasing scores for low-achievers, and one would have to acknowledge that if one were also inclined to give NCLB credit for all gains.

And so, the evidence is absolutely clear in one regard, but in the opposite direction of what Loveless and Petrilli suggest: One thing you definitely cannot say about NCLB is that it has clearly helped low achievers. And yet, they said it anyway!