Even strong proponents of private school choice programs often disagree on who ought to have access. Many people that view private school choice as a means for social justice argue that programs should be targeted to the least-advantaged members of society. Alternatively, I have claimed that a universally accessible program could benefit the least-advantaged in society more than anyone because of amplified market entry. Other education scholars argue that program access should be determined based on the empirical evidence on student test scores. While we gain important information from scientific experiments, we should not make program access decisions based on them.
Experiments Can Only Tell Us about Groups
As a social scientist performing quantitative analyses on school choice programs across the United States, I have realized one thing that is particularly frustrating. Even the strongest quantitative scientific experiments do not tell us much about the individual children that we are studying. Since our statistical results rely on the law of large numbers, there is no way around this issue. We must group the people we are studying together in order to calculate a statistically significant treatment effect.
Sure, we can perform subgroup analyses to determine if there are heterogeneous effects for different types of individuals. Nevertheless, these subgroup analyses suffer from the same systemic flaw; they rely on grouping people to calculate average effects. While subgroup analyses give us important information about groups of people, they are often erroneously used by decision-makers to determine which specific children in society ought to have access to school choice programs. A large positive result for advantaged members of society and an insignificant result for the disadvantaged may lead one to solely support access for the advantaged.
The problem with this decision is that it assumes all members within the subgroup will respond to the treatment in the same way. This is far from true. An average overall result of “zero” for disadvantaged children likely means that the program worked for some of them and did not work for others. Why prevent disadvantaged students from accessing a program simply because they looked like those that did not benefit previously?