Over at the New York Times, reporter Scott Shane announces the beginning of a running dialogue about how to strike “the proper balance between liberty and security” more than a decade after the terror attacks of 9/11.  I want to suggest, however, that framing the question the way Shane does, in terms of optimizing the “trade-off” between these competing values, begs the crucial question: Has there been a trade-off? Have all the billions of dollars and intrusive new surveillance powers granted our intelligence agencies in recent years actually made us any safer? Shane presents the choice we face in a way that simply assumes, without argument, that they have:

The next president might reach one of two very different conclusions: to continue its record of success, the government should keep doing everything it is doing, and Americans should accept that the trade-offs of the national security state are permanent. Or: the terrorism emergency that began with 9/11 has eased, the threat has diminished, so the security bureaucracy should shrink accordingly and the pendulum should swing back in favor of civil liberties and individual privacy.

But surely there’s a third possible conclusion—and one with the virtue of being far better supported by the available evidence: That much of the expansion of the national security state has not involved any “trade-off” at all because it has not meaningfully increased our security; that the absence of major terror attacks since 9/11 is not remotely the same thing as a “record of success” for the “security bureaucracy,” and that to the extent a genuine “record of success” does exist, it has very little to do with the most controversial and prominent “War on Terror” measures.

As my colleague John Mueller observes, there has been essentially no serious effort to do any serious cost benefit analysis of the costly security measures imposed since 2001—probably in part because it seems so obvious that most would fail any such test. The mere fact that we haven’t seen a repeat of 9/11 hardly tells us much: We hadn’t had one in the decades before we started inflating the powers and budgets of the intelligence agencies either. Inferring a causal relationship from the absence of an already rare phenomenon is pure magical thinking—the national security equivalent of trumpeting the lack of volcano eruptions since the last virgin sacrifice.  Moreover, there have been multiple attacks over the past decade that the newly expanded security bureaucracy clearly failed to detect or preempt. The NSA’s vast surveillance apparatus didn’t stop shoe bomber Richard Reid or undie bomber Umar Farouk Abdulmutallab: Alert passengers did that. Attempted Times Square bomber Faisal Shazahd was foiled by his own incompetence and an alert street vendor.

Of course, our intelligence agencies have had concrete counterterrorism successes, but that in itself doesn’t tell us anything about the causes of those successes. If, as the saying goes, “everything changed” after 9/11,  we need to know which changes contributed to those successes and which didn’t. There little evidence that the intelligence failures leading up to the 2001 attacks were fundamentally about insufficient power or resources. Rather, as intelligence scholar Amy Zegart has convincingly argued, they were the result of organizational and cultural problems within the agencies which had been identified again and again and again over the years by expert panels and blue ribbon reports whose reform recommendations somehow never wound up getting implemented. To the extent some of those problems finally started to be addressed after 9/11, it would be a much more plausible explanation of any subsequent performance improvements.

What about the government’s own claims that some of these radical new surveillance powers have been vital, life-saving intelligence tools? That would support the “trade-off” thesis, but since it’s bad politics to announce that you’ve violated people’s rights frivolously, we should want to check those claims against some independent analysis. In the case of the warrantless surveillance program authorized by President Bush—predecessor to the programmatic surveillance now conducted under FISA’s section 702—we’ve actually got just that: An unclassified report by the Inspectors General of the intelligence community. And what did they find? A lot of officials making vague statements to the effect that the program was “one useful tool in the toolbox,” but not much in the way of concrete achievements—plots foiled or terrorists captured—that could be attributed to the program we’d been told for years was absolutely critical. Instead—as Scott Shane’s own reporting had told us!—it seems to have wasted the time of a lot of FBI agents chasing down all the dead-end leads it generated.

Approaching it from the other direction, we can look at the known cases of people charged with terror-related crimes. Do we find a string of plots foiled thanks to sophisticated surveillance methods that would have been unavailable under pre-2001 laws? We do not. Mostly we find human intelligence and tips from alert members of the community playing the critical role.

Benjamin Franklin’s wise aphorism notwithstanding, there are cases where we do face genuine trade-offs between liberty and security, and sometimes—as when we allow homes to be searched pursuant to a judicial warrant supported by probable cause—those are even trade-offs worth making. But many measures that make us more secure don’t affect liberty at all: The single reform that probably did the most to guarantee we wouldn’t see a repeat of 9/11 was the simple decision to reinforce and lock cockpit doors. And the easiest thing of all to do is is implement “security theater” that diminishes our privacy to create the appearance of “doing something” without actually making us any safer. Sometimes we face hard questions about the trade-off between liberty and security—but we shouldn’t even begin to consider which trade-offs are worth making until we’ve seen some solid evidence that the trade-off is real. For most “war on terror” measures, the evidence just isn’t there.