The practice of policy analysis is often forward-looking. It attempts to answer questions such as: “How will the Affordable Care Act affect the health of the uninsured?” “How will a carbon tax affect fuel consumption and greenhouse gas emissions?” “Will charter schools provide a better education in communities wracked by poverty?” In other words, policy analysis attempts to answer some of the most vexing questions in the policymaking realm. We hope that those charged with answering these questions—often economists—are experts in their fields and that we can trust their answers.

Nowhere is this truer than in the world of regulatory policy. Over the past several decades we have put a great deal of hope in policy analysis, specifically cost-benefit analysis, as a means of improving regulatory decisions. Cost-benefit analysis is required before decisions are made on economically significant regulations. Fundamentally, these analyses are predictions about how regulations will affect individual decisions and the general welfare.

However, recent literature—most prominently the work of Wharton School management professor Philip Tetlock—has raised doubts about the reliability of experts’ predictions. Tetlock has shown that experts often do little better than laymen in predicting the future. Since reliable prediction is the fundamental goal of policy analysis, the conclusions of Tetlock and others are concerning.

Experts and Predictions

The role of experts in setting policy in a democratic society has been debated since the time of the ancient Greeks. Some worry that supposed experts merely impose their own preferences on society, thereby undermining democracy. This worry is heard across the political spectrum, both from those who assume government bureaucrats are obsessed with their agencies’ missions and intent on over-regulating industry, and from those who believe that regulators are “captured” by industry experts and therefore prefer to under-regulate.

What unifies these perspectives is that each implicitly assumes that experts are imbued with knowledge that makes their assessments of policy implications more accurate than those of non-experts. In a series of experiments and observations, Tetlock has cast doubt on this assumption. Additionally, he finds that simple algorithms—such as “Assume that current trends will continue”—typically outperform experts’ predictions.

Hedgehogs and foxes / There are differences within the expert community. Adopting Isaiah Berlin’s typology, Tetlock divides experts into “foxes” and “hedgehogs.” The key distinction between the two is that hedgehogs focus on one thing that they know very well and try to use it to answer all possible questions. Foxes on the other hand are generalists, knowing many things and being naturally skeptical of grand theories. Foxes are better Bayesians (they update their predictions on the basis of new information) and they are less subject to hindsight bias (coming up with rationalizations for previous mistakes). As a result, foxes—while far from exceptional in making predictions—regularly outperformed hedgehogs in Tetlock’s experiments.

In addition to Tetlock, several other scholars and writers have produced significant recent work on the nature of prediction. Statistician Nate Silver, formerly of the New York Times and now of ESPN’s FiveThirtyEight website, made a name for himself predicting election results. He has used that success to look at the nature of prediction generally. Like Tetlock, Silver has found that in many fields, expert predictions have been found to be inaccurate and that the predictors who are most confident are also the most likely to be incorrect. He echoes Tetlock’s praise of foxes and also emphasizes the importance of those making predictions acting as Bayesians.

Dan Gardner, a senior fellow at the University of Ottawa’s Graduate School of Public and International Affairs and co-author with Tetlock of the book Superforecasting (see “Of Hedgehogs, Foxes, and Superforecasting,” Fall 2016), shares many of these conclusions in his study of experts. He argues that better decisionmaking will require greater humility among experts and greater skepticism among their audiences.

Despite the many failures of prediction, Gardner and Silver both note that we can find some characteristics of those who make better predictions. Silver highlights the successful advances in weather prediction and Gardner praises George Soros as a “classic fox” who foresaw the economic crisis of 2008 among other correct predictions. This raises the question, do those who are charged with making predictions about policy have the characteristics of foxes, and if not, how can we construct an analytical process that uses these conclusions?

Cost–Benefit Analysis and Predicting Regulatory Outcomes

The role of analysis in U.S. policymaking has long been controversial. The broader the claims of the analysis, and the more that analysis advocates claim it is comprehensive and the best way to answer policy questions, the greater the controversy. In few arenas has this controversy been as heated and as relevant to actual policy decisions as in the use of cost-benefit analysis to evaluate regulatory proposals.

After initial, cautious steps in the Nixon, Ford, and Carter administrations, cost-benefit analysis became a formal part of U.S. regulatory decisionmaking in 1981 with President Ronald Reagan’s issuance of Executive Order 12291. Economists argued that cost-benefit analysis would lead to regulations that achieved the goals of regulatory statutes at lower costs. Later, economists studying implementation contended that cost-benefit analysis would bring a broader perspective to regulatory decisions both because it was inherently broader than the mission-driven preferences of those who set policies within regulatory agencies, and because it assisted the president (who was responsible to the entire electorate) in his control of regulatory agencies.

While its implementation in the context of regulatory policymaking has changed over the years, cost-benefit analysis has clearly become a permanent part of the regulatory process. It has arguably made regulatory policymaking more transparent by forcing agencies to lay bare their estimates of the consequences of their decisions. But at the end of the day, the more important question is whether the cost-benefit analyses have been correct: i.e., has the addition of economists to the regulatory process led to better predictions of policy effects?

How accurate have the predictions of the costs and benefits of regulations been? The data on this are limited because there is little mandate for government agencies to retrospectively analyze the costs and benefits of their regulations. Nonetheless, academics have attempted to answer the question.

In a 2005 report to Congress on the costs and benefits of regulations, the Office of Management and Budget summarized some of this literature. It reported that of 47 analyses studied, 11 were roughly accurate, 22 overestimated the cost-benefit ratio, and 14 underestimated it. This is not a great record in terms of accuracy, but at least it does not show an overt bias toward over- or underestimation. Like many approaches to prediction, however, the record of cost-benefit analysis is far from stellar.

Cost-benefit analysis arguably has made regulatory policymaking more transparent by forcing agencies to lay bare their estimates of the costs of their actions. But have the analyses been correct?

Does this mediocre record mean that the economists conducting cost-benefit analysis are hedgehogs? Both Silver and Gardner are skeptical about the predictive power of economists. Gardner writes:

Economists, in particular, are treated with the reverence the ancient Greeks gave the Oracle of Delphi. But unlike the notoriously vague pronouncements that once issued from Delphi, economists’ predictions are concrete and precise. Their accuracy can be checked. And anyone who does that will quickly conclude that economists make lousy soothsayers.

Those who support and those who oppose economists having a role in policymaking implicitly take different positions on whether the economists are hedgehogs or foxes. Those in favor of cost-benefit analysis as a requirement for regulatory decisionmaking tout its commitment to comprehensively examining all aspects of a decision. By requiring agencies to quantify all of the effects of their decisions, the resulting prediction is likely to be more accurate. By putting this quantification in the hands of economists, advocates of cost-benefit analysis are arguing that this is the best way to ensure that regulatory policy predictions are made by foxes.

On the other hand, critics of cost-benefit analysis have focused to a large extent on those aspects of a policy decision that are systematically ignored by economists. They point out that which cannot be quantified is automatically given less weight in a cost-benefit analysis. And, these critics maintain, there is much that cannot be given its proper weight in the realm of regulatory policy. In other words, economists have a very particular view of the world and as a result focus inordinately on the efficiency of policy decisions—that is, they are hedgehogs.

At this point, after three and a half decades of strident arguments, it seems like the question of whether economists are hedgehogs or foxes will not be resolved. I suggest that, instead of continuing the debate, we consider what the regulatory policy process would look like without economists.

Behavioral Public Choice

Instead of thinking about whether economists are hedgehogs or foxes, we can look at who else would make the predictions about policy effects if economists and cost-benefit analysis were left out. These predictors would likely be agency personnel with expertise in the subject matter, and their superiors.

The academic literature has studied the motivations of decisionmaking agency personnel, and a review of the literature’s conclusions does not generate confidence in the likely accuracy of their pre-analysis predictions. Anthony Downs, in his classic 1967 book Inside Bureaucracy, characterized agency personnel in one of five categories: climbers, conservers, advocates, zealots, and statesmen. Of those five types, only statesmen can be seen as possible foxes. The rest, to lesser or greater degrees, reflect varieties of hedgehogs.

Other studies tend to confirm that agency decisionmakers are, on the whole, best characterized as hedgehogs, although the studies reach that conclusion for different reasons. As James Q. Wilson persuasively argued, agency personnel tend to be dedicated to their missions; that is, few people go to work at the U.S. Environmental Protection Agency who don’t believe in the cause of environmentalism to the exclusion of other policy concerns. William Niskanen, in his 1971 book Bureaucracy and Representative Government, viewed agency officials as less motivated by mission and more by the goal of procuring generous budgets. Finally, there is a long history of arguments that agency officials are routinely captured by outside interests. These characterizations make agency officials sound much more like hedgehogs than foxes.

Recent work in the new field of behavioral public choice (an outgrowth of behavioral economics) casts further doubt on the predictive abilities of agency personnel. Government employees are subject to the same behavioral biases as the public. In fact, some have argued that because they are not disciplined by the market, public servants are more likely than others to make errors because of heuristics and biases. Additionally, as Wilson theorized, scholars of behavioral policy have hypothesized that government regulators will be biased in favor of government intervention in the market. As Gary Lucas and Slavisa Tasic put it in a 2015 law review article, “Bureaucrats who believe strongly in their agency’s mission and other experts who are focused upon the particular problems addressed in their respective fields sometimes ignore relevant information and competing interests.”

Jeffrey Rachlinski and Cynthia Farina argue that institutional design may be one remedy to a hedgehog-dominated policy process. They believe that current checks upon expert biases in government, including public participation in agency decisions and judicial review, are sufficient.

But the cure may not be sufficient. Concerns about agency bias persist decades after judicial review, executive branch deliberation, and public participation have been routine. Greater inclusion of economists in this world is another possible solution—even if the economists themselves are hedgehogs rooted in a commitment to economic efficiency. This view has been articulated by former Office of Information and Regulatory Affairs administrator Sally Katzen, who has touted cost-benefit analysis as a necessary corrective to agency biases. In an ideal world, we would have foxes working for the government and predicting policy effects. In the absence of foxes (or even in the absence of a hiring process that can identify foxes), we should have hedgehogs with competing views.

Recent work by Russell Mills and Christopher Carrigan supports this idea. They found that when agencies incorporate input from individuals from diverse professional backgrounds, their regulations are less complex and more comprehensible to the public. I would never argue that economists should be the only individuals making decisions within the government. But ensuring that economists are part of the team—even if they are blinkered by a particular world view—can act as a check upon the limited world view of others already involved in the process.

Conclusion

Making accurate predictions about the future is hard. More than that, recent research has shown that many experts—the people we trust most—are little better than laypeople at making predictions. The effect of this research on the policymaking process has not been sufficiently explored. This gap is important because the fundamental point of policy analysis is to accurately predict the effects of proposed policies. Furthermore, these predictions are often entrusted within the federal bureaucracy to experts: individuals who have studied policy analysis broadly or have expertise in a particular field.

One area of policy where there has been considerable debate about the role of experts (albeit not couched in that language) is regulatory policy. Tetlock’s findings are disturbing because predicting the effect of regulations depends on experts. Supporters of cost-benefit analysis claim that economists have asserted the foxlike abilities of economists (again not couching it in this language) to bring a broader perspective to regulatory decisions. Their critics have said that economists have a narrow perspective and therefore are inherently biased against decisions that protect public health; in other words, economists are the worst kind of experts: hedgehogs.

As discussed above, determining whether economists are hedgehogs or foxes is difficult, if not impossible. Their record on predicting the effects of regulatory policies is mixed at best, suggesting that they do suffer from the problems of experts generally. It is tempting to look at these data and say that economists are hedgehogs and should be excluded from regulatory decisionmaking.

But their exclusion from the regulatory decisionmaking process would not improve regulatory decisions. It would leave regulatory policy in the hands of those who are even more clearly hedgehogs: experts in a particular area of policy who have dedicated their lives to the idea that government intervention in a policy area is the appropriate solution. Even if economists are little better, and biased in the opposite direction, their inclusion in regulatory decisionmaking has served to balance the influence of these experts.

The requirement for cost-benefit analysis has only rarely been framed this way, but it may provide a model for improving government decisionmaking more broadly. Economists may or may not have the qualities we want to see in experts. But we may be able to design policy analytic systems that mimic these qualities. Absent a fox, the best way to make policy might be to ensure that hedgehogs with many different biases are involved in the decisionmaking process.

Readings

  • “A Reality Check on an Empirical Study: Comments on ‘Inside the Administrative State,’ ” by Sally Katzen. Michigan Law Review 105(7): 1497–1510 (2007).
  • “Behavioral Public Choice: The Behavioral Paradox of Government Policy,” by W. Kip Viscusi and Ted Gayer. Harvard Journal of Law & Public Policy 38(3): 973‑1007 (2015).
  • “Behavioral Public Choice and the Law,” by Gary M. Lucas Jr. and Slavisa Tasic. West Virginia Law Review 118(1): 199–266 (2015).
  • “Cognitive Psychology and Optimal Government Design,” by Jeffrey J. Rachlinski and Cynthia R. Farina. Cornell Law Review 87(2): 549–615 (2001).
  • Expert Political Judgment: How Good Is It? How Can We Know? by Philip Tetlock. Princeton University Press, 2005.
  • Future Babble: Why Expert Predictions Fail—and Why We Believe Them Anyway, by Dan Gardner. McClelland & Stewart, 2010.
  • “Structuring Rulemaking: The Effects of Organization on Rule Detail and Timeliness,” by Russell Mills and Christopher Carrigan. Presented at the American Political Science Association annual meeting, 2016.
  • The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t, by Nate Silver. Penguin, 2012.