Even strong proponents of private school choice programs often disagree on who ought to have access. Many people that view private school choice as a means for social justice argue that programs should be targeted to the least-advantaged members of society. Alternatively, I have claimed that a universally accessible program could benefit the least-advantaged in society more than anyone because of amplified market entry. Other education scholars argue that program access should be determined based on the empirical evidence on student test scores. While we gain important information from scientific experiments, we should not make program access decisions based on them.

Experiments Can Only Tell Us about Groups

As a social scientist performing quantitative analyses on school choice programs across the United States, I have realized one thing that is particularly frustrating. Even the strongest quantitative scientific experiments do not tell us much about the individual children that we are studying. Since our statistical results rely on the law of large numbers, there is no way around this issue. We must group the people we are studying together in order to calculate a statistically significant treatment effect.

Sure, we can perform subgroup analyses to determine if there are heterogeneous effects for different types of individuals. Nevertheless, these subgroup analyses suffer from the same systemic flaw; they rely on grouping people to calculate average effects. While subgroup analyses give us important information about groups of people, they are often erroneously used by decision-makers to determine which specific children in society ought to have access to school choice programs. A large positive result for advantaged members of society and an insignificant result for the disadvantaged may lead one to solely support access for the advantaged.

The problem with this decision is that it assumes all members within the subgroup will respond to the treatment in the same way. This is far from true. An average overall result of “zero” for disadvantaged children likely means that the program worked for some of them and did not work for others. Why prevent disadvantaged students from accessing a program simply because they looked like those that did not benefit previously?

Negative Impacts in DC and Louisiana

Let’s examine the most recent experimental studies of the Louisiana Scholarship Program and the DC Opportunity Scholarship Program. The evaluations found overall negative impacts on student test scores in early years. Consequently, many school choice critics have called for choice program elimination. It is naïve to make this conclusion based on studies that cannot tell us anything about individual children. Although the average result is negative for test scores, it is positive for many children that have access to the voucher program. Mathematically, even with a negative average outcome, the program could be positive for more children than it is negative.

Most importantly, we must realize that these results only pertain to test scores. All of these children likely benefit in areas that are not captured by standardized tests. After all, individual families are choosing these private schools for reasons that are unique to their own children. Moreover, several evaluations have shown a notable disconnect between test scores and vital long-term outcomes such as graduation rates, criminal activity, and income.

A Truly Ironic Phenomenon

I find my experience as a social scientist quite ironic: the very people calling for decentralization within our society’s system of schooling use empirical results like central planners, placing people into overly simplistic groups and deciding who ought to have access to an enhanced education and better life. We should accept that even gold-standard scientific analyses only have the power to detect effects for groups of people, not individuals. Instead, we should allow individuals to use empirical results to make decisions about how they are going to improve their own lives.