Tag: performance

Highlights of the New PISA International Test Results

The latest (2012) PISA results are out! PISA is a test of fairly basic, practical skills given to 15-year-olds around the world. Here are some of the highlights:

  • U.S. performance is essentially flat across subjects since 2003
  • Finland’s performance has declined substantially since 2003
  • Korea is continuing to improve, solidifiying its position as one of the highest performing nations
  • Already the highest-performing Latin American country, Chile has continued to improve, leaving the regional average further behind.

The U.S. story needs little elaboration. Neither the structure nor the content of American schooling has changed in educationally meaningful ways since 2003. We still have 50 state education monopolies, with a growing but still realtively small homogenizing federal presence.

The “Replicate Finland!” bandwagon was always misguided. It is simply not sensible to take a nation’s performance on a single test, in isolation, as evidence for the merits (or demerits) of its national education policies. There are too many other factors that affect outcomes, and there are too many important outcomes for a single test to measure. For those who nevertheless championed Finland as a model, the latest PISA results are a bit awkward (see, for instance, the book: The Smartest Kids in the World).

Though the Chilean student protests of 2011 and 2012 focused on the desire for free, universal college, the leaders of that movement also harshly criticized that nation’s universal K-12 private school choice program. About 60 percent of children in Chile attend private schools, most of them fully or substantially funded by the national government. One of the most famous protest leaders, Camila Vallejo, was recently elected to the Chilean congress as a member of the Communist party. The influence of Vallejo and her compatriots has shifted public sentiment against crucial aspects of the nation’s private school choice program, despite the fact that private schools themselves remain extremely popular with parents. It is quite possible that, in the coming years, Chile will unravel the very policies that have made it one of the fastest improving countries in the world and the top performer in Latin America.

The NEA has called for higher U.S. teachers’ salaries based on the PISA results, arguing that some of the top performing countries pay their teachers more relative to people in other careers. This is self-serving and scientifically dubious. The NEA presents no evidence for a causal link between overall teacher salaries and student performance, just a bit of random cherry picking that ignores countless confounding factors. To find the real link between average salaries and performance, we can look at domestic U.S. research on the subject. Hanushek and Rivkin, for instance, find that “overall salary increases for teachers would be both expensive and ineffective.” Not surprisingly, a recent review of Ohio’s data on teacher “value-added” and teacher pay finds an inverse relationship:

in Cleveland… teachers deemed “Least Effective” by the new state evaluation system earned, on average, about $3,000 more than the teachers deemed “Most Effective.”

There’s some evidence that tying teacher pay to student performance helps to improve learning, but that’s about it.

Finally, it’s important to remember that PISA is a test of everyday “literacy” in the three subjects it covers (math, reading, and science). If you want to know how well students are learning the specific academic content needed for continuing study at the college level, PISA isn’t your best choice. For that, take a look at TIMSS.

Here’s Where Better Schools HAVE Scaled Up…

Earlier this summer, I released a study comparing the performance of California’s charter school networks with the amount of philanthropic grant funding they have received. The purpose was to find out if this model for replicating excellence was consistently effective. The answer, regrettably, was no.

But a new study we are releasing today finds that there is at least one place where better schools HAVE consistently scaled-up: Chile. Thanks to that nation’s public and private school choice program, chains of private schools have arisen, and they not only outperform the public schools, they also outperform the independent “mom-and-pop” private schools.

For anyone interested in replicating educational excellence, this study by a team of Chilean scholars is worth a look.

Could You Modify It ‘To Stop Students From Becoming This Advanced?’

The free Web tutoring service “Khan Academy” has gotten much well-deserved attention, including a feature story in the current issue of Wired. That story includes a quote that literally took my breath away:

Even if Khan is truly liberating students to advance at their own pace, it’s not clear that the schools will be able to cope. The very concept of grade levels implies groups of students moving along together at an even pace. So what happens when, using Khan Academy, you wind up with a kid in fifth grade who has mastered high school trigonometry and physics—but is still functioning like a regular 10-year-old when it comes to writing, history, and social studies? Khan’s programmer, Ben Kamens, has heard from teachers who’ve seen Khan Academy presentations and loved the idea but wondered whether they could modify it “to stop students from becoming this advanced.”

This attitude is a natural outgrowth of our decision to operate education as a monopoly. In a competitive marketplace, educators have incentives to serve each individual child to the best of their ability, because each child can easily be enrolled elsewhere if they fail to do so. That is why the for-profit Asian tutoring industry groups students by performance, not by age. There are “grades,” but they do not depend on when a student was born, only on what she knows and is able to do.

But why should a monopolist bother doing that? It’s easier just to feed children through the system on a uniform conveyor belt based on when they were born.

Fordham Institute Reviews ‘The Other Lottery’

Gerilyn Slicker, of the Fordham Institute, offers a brief review of my study of charter school philanthropy in the latest “Education Gadfly” mailing, including the following observation:

Note, though, that this analysis is not without fault. The report doesn’t break down spending by pupil (only reporting aggregate grant-giving), nor does it account for student growth over time or for how long the charter networks have been operational.

All three of these concerns are worth raising, and the first two of them were actually addressed in the paper itself. The aggregate vs. per-pupil grant funding question is discussed in endnote 15:

Note that total grant funding, rather than grant funding per pupil, is the correct measure. That is because enrollment is endogenous—it is a product, in part, of earlier grant funding. So, controlling for enrollment (which dividing by enrollment would do) would control away some of the very characteristics we are trying to measure: the charter network’s ability to attract funding.

Student growth over time, as noted on page 5, cannot be measured using the California Standards Tests, because it reports results as averages of subgroups of students at the classroom level, not as individual student scores. And since the CST is the only source that has broad-based performance data for all charter and traditional public schools in the state, it is the only dataset that can be used to measure the performance of all California charter school networks. Fortunately, good controls for both student factors and school-wide peer effects are available, and the study’s results are consistent with earlier research, where it overlaps with that research.

The final concern, network age, is not one that I directly addressed in the study. There are a couple of reasons to expect it would not have much of an impact on the findings, however. First, a cursory look at the age of some of the top networks shows no particular pattern. American Indian and KIPP are both a decade old, and rank #1 and #7, respectively, out of 68 networks. Oakland Charter Academies and Rocketship are just a couple of years old, and rank #2 and #4, respectively. Similarly, some of low performing networks are brand new, while others, like GreenDot (ranked 42nd), are also over a decade old.

Second, in Appendix E, I show that network size and network academic performance are not significantly linked to one another. And since network age and network enrollment are likely to be strongly positively correlated with one another, it would be surprising if network age were correlated with performance when enrollment is not. That said, I’d probably include network age as a control in future, if I repeat the study, just to be on the safe side.

Ranking the Charter School Networks

Much of the response to the study I released last week has focused on the relative academic performance rankings of California’s charter school networks. That wasn’t the point of the study, which focuses on whether or not philanthropy + charter schooling can replace venture capital and competitive markets as a mechanism for scaling-up the best education services. Rather than try to fight the tide, I thought I’d just share the relevant rankings in an easy-to-link form, and once the debate about them dies down we can return to the larger policy point.

With that in mind, the first table below lists the top 15 charter school networks in terms of performance on the California Standards Tests, adjusted for student factors and peer effects. For comparison, two non-charter schools are included: the academically selective elite public prep schools Gretchen Whitney and Lowell–both of which feature in most lists of the top public schools in the country. There are 68 networks with the necessary data, but the lowest grant rank is 61 because eight of the networks received no philanthropic funding at all.

Next is a list of the charter networks that philanthropists have invested-in most heavily, with a view to replicating their models. Notice the minimal overlap? I repeat this comparison in the study with Advanced Placement test performance, and find the same pattern (it’s just slightly worse).

Every one of the above networks received substantially more grant funding individually than the top three highest achieving networks… combined.