Fail Often, Fail Well

Most new ideas, products, and businesses fail. Failure is not just ubiquitous and painful, but it also is an essential source of learning for individuals and organizations. But are we using it the right way? Do we embrace it as an unavoidable, and indeed valuable, part of life, or do we try to avoid it at all cost?

Megan McArdle — who hardly needs to be introduced to the Umlaut readership — says with tongue in cheek that she was prompted to write her forthcoming book about failure by the simple principle of “write what you know.” The Up Side of Down is often an unusually personal book, replete with anecdotes as well as with solid references to research coming from various fields of social science.

Megan shows the common thread of psychological and institutional biases going through situations in which people confront failure. The differences in how we handle mistakes, errors, and failures, she argues, go a long way to explain the success of individuals, companies, and societies.

Failure should lead to a corrective adjustment. An idealized example of such adjustment can be found in the classic experiments by Vernon Smith, in which the subjects, placed in a stylized competitive marketplace, quickly learn when bidding too much or too little does not deliver the desired outcome. These experimental markets very quickly converge to equilibria and produce efficient outcomes.

The world falls short of this ideal. Even in the lab, once researchers moved their subjects to experimental settings with fuzzier or undefined rules, the experiments yielded a variety of outcomes, many of them quite disheartening.

In short, when not subjected to the instant discipline provided for example by the competitive market, people and organizations are prone to ignore their mistakes and compound them collectively by trying to avoid failure at all cost, not recognizing it, or drawing wrong lessons from it.

At their core, the factors driving our inability to process failure are psychological. Our reluctance to disregard sunk costs in making our decisions is one culprit, says Megan:

Another reason? Denial, or “normalcy bias.” It usually takes people a long time to internalize that things are going wrong. Even during disasters — such as 9/11, Katrina, or air crashes — many have been reported to have a strong urge to pretend that things are normal rather than quickly adjusting to the new reality.

Being around other people who share from the same set of priors can reinforce the problem. Herd behavior leads to bad decisions in groups of like‐​minded people and in organizations. In part, this is because individuals assume that others have good reasons for their beliefs. Worse yet, collective mistakes lock even those who know better in a prisoner’s dilemma. In most organizations, exposing the errors of your numerous colleagues rarely brings about large pay‐​offs.

Finally, as Hayek tried to explain at great length, people have an awkward relationship with complex systems. When things go awry, our instinctive reaction is to assign blame and punishment. Sometimes these are in order, but frequently that way of thinking is misleading. Unintentionally, it can lead to a stigmatization of failure as something that has to be avoided. But, as one of the subtitles of the book reads, “the opposite of failure is not safety: it’s nothing.”

While human psychology is key, its effects are magnified by institutional design. In settings with misaligned incentives, failure will not fuel the process of creative destruction but will breed organizational dysfunction and possibly cascade into catastrophes. Although this is not primarily a book about public policy, it is not a stretch to suggest that the political process — with its many stakeholders, ill‐​defined goals, high transaction costs, misaligned incentives and weak feedback mechanisms — will be particularly bad at handling failure.