Andrew Exum and the Philosophy of Science

Andrew Exum suggests a "manifesto...for those using quantitative analysis to study war/Hippocratic Oath for Quantitative Analysis in Security Studies" here.  I think there are two different critiques lurking in there, but his presentation of his list muddles them together.  The first critique is mostly about the importance of modesty in social science, but the second seems quite like an assault on the very idea of social science.

First, let me put my cards on the table.  I am not a quant or a formal modeler.  (These two approaches are different, but Exum seems to lump them together.)  I have a rudimentary statistics background, and could identify supremely egregious errors in both quantitative and formal model papers if I were locked in a room and threatened with violence.  I am no partisan of either faction.  But I think Exum's views are probably common in DC, so this could work as a forum for discussing part of what I think is wrong with the DC policy debate.

Take, to start, Exum's suggested pledge that "War is a human endeavor. I recognize that it is a phenomenon that does not conform to neat mathematical equations," and set it in the context of another one: "I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do. "

The first claim is about modesty: social science is not the same as physical science.  It is harder to conduct controlled experiments in social science, for a variety of practical/political and moral/ethical reasons.  (The war in Iraq may be an exception.)  If what Exum is getting at here is a claim like "quantitative scholars can be arrogant and oversell their research," then Amen.  But his second claim lionizes squad leaders in the 10th Mountain Division as superior in knowledge to social science researchers.  I find this juxtaposition very odd, and I think it's basically a rejection of social scientific principles in general.  (It also seems to carry with it an implicit claim that military operations cannot be subject to scrutiny by non-military overseers.  As a helpful reviewer of this post wrote, "It's the equivalent of saying that we should just do whatever teacher's unions want in K-12 education policy, or that the guys who run meatpacking plants are qualified to offer opinions about food safety.")

It just isn't true that inducing inferences from anecdotal experience produces better explanations/predictions than do people who have larger universes of cases and can control for various factors.  Exum seems to support an approach to theory-building in which one directly observes facts and then induces theory based on those observed facts.  To put it mildly, this is a peculiar view of the philosophy of science.  So what starts as a lament about the arrogance of various factions of social scientists becomes a larger criticism of social science itself.

That is because while Exum is explicitly focusing on quantitative researchers or formal modelers or both, qualitative research is subject to the same criticism he is offering.  If first-hand observation of facts leads to sounder understandings of subject matter than does clear theorizing and fair-minded examination of larger samples of data, then social science itself is cast into doubt.  If that's not where he's going with this, I'd like to hear more about where he is going.

The practical problem with his call for theoretical and analytic modesty is that it cuts against the incentives researchers face.  Existing scholarship consists of very ambitious theories that are promised to hold lots of explanatory power.  Given that is the nature of the debate, a paper that says outright "my theory is pretty good, but I identify lots of important cases where it won't hold together and I don't know why" would have little chance of publication.  I think this is an important point, and many theorists will tell you over drinks the limits of their theories, but the incentive structure is such that one can't sell a theory in that way.