The latest results of the Nation's Report Card for history, geography, and civics are out, and as usual they are depressing. The exam, formally known as the National Assessment of Educational Progress, is administered to a representative sample of U.S. students to give a snapshot of student performance in a variety of subjects nationwide. Education Week reports:
The nation's eighth graders have made no academic progress in U.S. history, geography or civics since 2010, according to the latest test results from the National Assessment of Educational Progress (NAEP).
Fewer than one-third of students scored proficient or better on any of the tests and only 3 percent or fewer scored at the advanced level in any of the three subjects.
However, Chad Aldeman of Bellwether Education Partners argues that saying students "have made no academic progress" is "the wrong way to look at it" because of something called Simpson's Paradox (which has nothing to do with the voice of Principal Skinner and Mr. Burns turing down a $14 million contract):
Because NAEP takes a representative sample, it’s also vulnerable to something called Simpson’s Paradox, a mathematical paradox in which the composition of a group can create a misleading overall trend. As the United States population has become more diverse, a representative sample picks up more and more minority students, who tend to score lower overall than white students. That tends to make our overall scores appear flat, even as all of the groups that make up the overall score improve markedly.
Recent NAEP results in history, geography, and civics illustrate this trend once again. Education Week reported that scores were “flat” from 2010 to 2014. That’s mostly true—the scores were all higher than in 2010 but didn’t meet the standard for statistical significance. But scores are up over longer periods of time. Here are the gains since 2001 on geography (* signifies statistically significant):
• All students: +1
• White students: +4*
• Black students: +7*
• Hispanic students: +9*
• Students with disabilities: +8*
• English Language Learners: +7... A few things jump out from these longer-term results. First, overall scores are up a little bit, but particular groups of students are making big gains. One rule of thumb suggests that 10–15 points on the NAEP translates into one grade level. Applying that here, scores for most groups of students have improved by roughly a full grade level over the last 15 years or so. Second, achievement gaps are closing as lower-performing groups are catching up to higher-performing ones. Third, Simpson’s Paradox makes the overall scores look relatively “flat.” Don’t let that mislead you. Although we might wish for faster progress, American achievement scores are rising.
However, other education policy experts have cast a gimlet eye on this interpretation. Jay P. Greene of the University of Arkansas argues, "It is not appropriate to explain away the lack of aggregate progress in academic achievement by referencing Simpson’s Paradox and dis-aggregating results by racial/ethnic group." In a blog post on the mis-use of Simpson's Paradox a few years ago, Greene wrote:
The unstated argument behind the use of Simpson’s Paradox to explain the lack of educational progress [is that] minority students are more difficult to educate and we have more of them, so holding steady is really a gain.
The problem with this is that it only considers one dimension by which students may be more or less difficult to educate —race. And it assumes that race has the same educational implications over time. Unless one believes that minority students are more challenging because they are genetically different [which, Greene notes, he does not think Aldeman believes], we have to think about race/ethnicity differently over time as the host of social and economic factors that race represents changes. Being African-American in 1975 is very different from being African-American in 2008. (Was a black president even imaginable back then?) So, the challenges associated with educating minority students three decades ago were almost certainly different from the challenges today.
If we want to see whether students are more difficult to educate over time, we’d have to consider more than just how many minority students we have. We’d have to consider a large set of social and economic variables, many of which are associated with race. Greg Forster and I did this in a report for the Manhattan Institute in which we tracked changes in 16 variables that are generally held to be related to the challenges that students bring to school. We found that 10 of those 16 factors have improved, so that we would expect students generally to be less difficult to educate.
In addition, my Cato colleague Andrew Coulson—creator of the infamous chart below—expressed skepticism about using the main NAEP assessment (which changes between test administrations) to analyze long-term trends. Instead, he points to the NAEP's "Long-Term Trend" series that was designed just for that purpose. Unlike the main NAEP assessment, the Long-Term Trend remains relatively unchanged over time. Sadly, the results are bleak:
It is true that both black and Hispanic students now score higher than they did in the early 1970s, and the difference isn’t negligible as it is with the overall aggregate trend. ... The trends for white students, who still make up the majority of test takers, are only marginally better than the overall trends. Whites gained four points in each of reading and math, and lost six points in science. The overall picture for whites is thus also essentially a flat line, and it is their performance that is chiefly responsible for the stagnation in the overall average scores, not the increasing participation of historically lower-scoring groups.
In short, there is little evidence that Simpson's Paradox explains flat U.S. student performance in the last few decades.