Anyone who follows the  stress tests conducted by the Fed and various European banking authorities can’t help poking fun at them.  After all, it’s hard to repress a chuckle when, time and again, a bank passes one of these tests with flying colors only to end up failing not long afterwards.  Whether it’s Iceland in 2008, Ireland in 2010, or Cyprus in 2013, the story is the same: all three national banking systems collapsed shortly after being signed off as safe following regulatory stress tests.

When putting banks to such a test, a central bank or other banking authority starts by imagining  one or more “stressful” scenarios to which banks might be exposed, uses a bunch of models to determine how those scenarios will affect the banks’ capital adequacy (that is, their ratio of capital to assets), and then passes or fails banks depending on whether their capital remains adequate at the end of the test.

The Bank of England reported the results of its first set of annual stress tests in December.  Its message was a reassuring one: the UK banks are safe.  Unfortunately, there’s no good reason to trust that assessment than there was to trust the others, for the Bank of England’s stress tests are also deeply flawed, in ways that, besides obscuring the significant vulnerability of the UK banking system, actually tend to accentuate it.

For starters, the tests are based on a “risk-weighted asset” metric, which depends on models that underestimate banks’ risks.  Worse still, these models are eminently gameable, and banks have every incentive to game them, since doing so can reduce their capital requirements and allow them to distribute greater false profits.

Abundant research—from the Bank of England itself, among other authorities—has convincingly established that lower risk-weighted asset scores do not necessarily mean lower risk.  In fact, the risks are often greater; they just happen to be among those that are invisible to the risk measure.  We saw precisely this problem in 2008–2009, when UK banks appeared to be well capitalized using risk-weighted asset metrics, but actually turned out to be massively undercapitalized when the financial crisis hit.

These points alone ought to be enough to discredit the Bank of England’s stress tests.   Still, there are a plenty of other problems  to consider.

First, the Bank’s stress tests are based on just a single scenario.  This approach cannot possibly give us confidence that the banking system is safe against all the other possible scenarios that were not considered.  Indeed, the Bank acknowledged the need to consider multiple scenarios in one of its preliminary discussion papers, but then inexplicably ignored its own advice and opted for a single scenario in its final report.

Second, the Bank describes its stress tests as ‘extremely tough,’ but in reality its scenario is a mild one: GDP growth falls to -3.2 percent before bouncing back, inflation rises to peak at 6.5 percent, long term gilts peak just below 6 percent, and unemployment hits 12 percent.  This is not particularly stressful by historical standards, and also pales in comparison to the Eurozone’s recent (and on-going) strife.  The Bank of England scenario also has only a mild impact on bank capital and profitability—and if a stress scenario doesn’t actually stress the banking system, what is the point of the exercise?

Third, the Bank uses a very low “pass” standard—a 4.5 percent minimum ratio of capital to risk-weighted assets.  This is lower than the 5.5 percent ratio that the European Central Bank used in its widely discredited 2014 stress tests, and is well below the capital requirements coming through under Basel III.  Had the Bank carried out a test using these Basel minimums, the UK banking system would have failed.  Same exercise, higher safety standard, opposite result.

Fourth, the Bank also failed to carry out any tests based on leverage—the ratio of capital to total, unweighted assets—which offers a much less gameable measure of a bank’s financial health.  Even an undemanding such test, based on the Bank’s required minimum leverage ratio of 3 percent, would have revealed how weak the UK banking system really was.  Of course, many experts recommend a minimum leverage ratio of 15 percent or more, at least five times larger than the leverage test that the Bank failed to conduct—or, at least, to report.  Had the Bank based its stress tests on this measure, December’s comforting financial headlines would have been very different indeed.

Fifth, stress tests impose a single view of risk on the banking system—one based on the same flawed models, with the same blind spots.  This creates systemic instability: if the stress test’s view of risk turns out to be wrong, all the banks subject to it are likely to run into trouble at the same time.

Finally, stress tests have all the credibility of a Soviet election.  Even if the central bank discovers there are major problems in the banking system, it will be loathe to publicly admit to them.  Doing so would undermine confidence and also lead to awkward questions about the central bank’s own supervisory competence.

The bottom line is this: official stress tests are like a lookout who has trouble seeing big, white icebergs.  If you wouldn’t travel on a liner with such a lookout, you shouldn’t trust a banking system based on some stress test that detects every sort of risk—except the sort that’s about to sink it.

(This post is based on Kevin Dowd’s recent in-depth report for the Adam Smith Institute, “No Stress: The Flaws in the Bank of England’s Stress Testing Programme,” in which he critically assesses the stress tests conducted by the Bank of England and other central banks.)