I have written a little about the utility of quantitative analysis in the field of security studies here and here. Last week, though, I finished Wall Street Journal reporter Scott Patterson’s book on how quantitative hedge funds — as opposed to “fundamental” investors like Warren Buffett — contributed to the Wall Street collapse of 2008. Patterson ends his book with the efforts of some quants to get their analysis to abide by a code of conduct. The resulting manifesto — written by Paul Wilmott and Emanuel Derman — can be read here. There are some useful passages, highlighted below, which address the uncomfortable reality that elegant mathmatical formulae don’t always describe messy human endeavors like the behavior of the markets — or war, for that matter.
Financial theory has tried hard to emulate the style and elegance of physics in order to discover its own laws. But markets are made of people, who are influenced by events, by their ephemeral feelings about events and by their expectations of other people’s feelings. The truth is that there are no fundamental laws in finance. And even if there were, there is no way to run repeatable experiments to verify them. …
The Modelers’ Hippocratic Oath
~ I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
~ I will never sacrifice reality for elegance without explaining why I have done so.
~ Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
~ I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.
I found the humility in this manifesto to be really refreshing. What might a similar manifesto look like for those using quantitative analysis to study war? And should the U.S. graduate programs in political science (and subsets of the field, like international relations and security studies) pushing their students toward quantitative analysis be more up-front about the explanatory limits of such analysis? Anyway, borrowing liberally (read: plagiarizing) from Wilmott and Derman, here is what I think a Hippocratic Oath for Quantitative Analysis in Security Studies should look like:
- War is a human endeavor. I recognize that it is a phenomenon that does not conform to neat mathematical equations.
- I will use quantitative analysis in conjunction with theory and qualitative analysis to describe what I see as phenomena in war and peace. I will be honest about the limits of both my theory and my analysis.
- In war and peace, the variables are infinite, and not everything can be measured or assigned a numerical value.
- I will not use numbers to signify what are fundamentally qualitative assessments without acknowledging to my reader that I have done so in order to satisfy a departmental requirement, gain tenure, or get published in the APSR. Or because I have been in graduate school for so long that I have forgotten how to effectively write in prose.
- I recognize there are no mathematical equations in Vom Kriege and that it is nonetheless unlikely that my legacy will transcend that of Clausewitz.
- I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do.
Stephen Walt in Foreign Policy:
Wise words indeed. I’d just add that Nobel prize-winning economist and strategic guru Thomas Schelling offered a similar warning in The Strategy of Conflict, cautioning against any tendency “to treat the subject of strategy as thohttp://aroundthesphere.wordpress.com/wp-admin/post-new.phpugh it were, or should be, solely a branch of mathematics.”
That’s not to say that various types of mathematical analysis aren’t useful, whether one is talking about operations research, basic statistics, game theory, or whatever. But it’s just a tool, and ought to be used in conjunction with other methods and with an appropriate degree of humility.
Drew Conway at Zero Intelligence Agents:
Yesterday, Andrew Exum—a person who admits his own ignorance of the current state-of-the-art in political science literature—presented his “manifesto” on the quantitative analysis of conflict. While Exum’s bonafides in counterinsurgency and military strategy go without saying, given that he knows almost nothing about quantitative analysis I found this manifest rather disingenuous. Furthermore, since he has referred to me as a quantitative “hired assassin,” I felt an additional duty to respond.
To be fair, Exum has recently praised the work of contemporary quantitative analysis of conflict by scholars such as Lyall, Berman and Shapiro, all three of which whom are well deserving of praise. As such, it is peculiar that Exum would feel the need to present this manifesto after having to first be told by others about this work (due presumedly to his own admitted ignorance); and second, actually liking it. To be sure, such an endeavor is useful, as it is clear that both subtle formal models and sophisticated statistical analyses can be manipulated and misinterpreted to present dangerous falsehoods; however, Exum’s attempt to undervalue the contributions of this work with respect to policy is equally dangerous.
Justin Logan at Cato:
First, let me put my cards on the table. I am not a quant or a formal modeler. (These two approaches are different, but Exum seems to lump them together.) I have a rudimentary statistics background, and could identify supremely egregious errors in both quantitative and formal model papers if I were locked in a room and threatened with violence. I am no partisan of either faction. But I think Exum’s views are probably common in DC, so this could work as a forum for discussing part of what I think is wrong with the DC policy debate.
Take, to start, Exum’s suggested pledge that “War is a human endeavor. I recognize that it is a phenomenon that does not conform to neat mathematical equations,” and set it in the context of another one: “I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do. ”
The first claim is about modesty: social science is not the same as physical science. It is harder to conduct controlled experiments in social science, for a variety of practical/political and moral/ethical reasons. (The war in Iraq may be an exception.) If what Exum is getting at here is a claim like “quantitative scholars can be arrogant and oversell their research,” then Amen. But his second claim lionizes squad leaders in the 10th Mountain Division as superior in knowledge to social science researchers. I find this juxtaposition very odd, and I think it’s basically a rejection of social scientific principles in general. (It also seems to carry with it an implicit claim that military operations cannot be subject to scrutiny by non-military overseers. As a helpful reviewer of this post wrote, “It’s the equivalent of saying that we should just do whatever teacher’s unions want in K-12 education policy, or that the guys who run meatpacking plants are qualified to offer opinions about food safety.”)
It just isn’t true that inducing inferences from anecdotal experience produces better explanations/predictions than do people who have larger universes of cases and can control for various factors. Exum seems to support an approach to theory-building in which one directly observes facts and then induces theory based on those observed facts. To put it mildly, this is a peculiar view of the philosophy of science. So what starts as a lament about the arrogance of various factions of social scientists becomes a larger criticism of social science itself.
In my opinion, this is the most important lesson that the social sciences have to offer to policy makers – be careful about selection bias. Policy debates in Washington DC are rife with selection effects, with advocates highlighting convenient cases for a particular policy argument and hiding inconvenient ones. I’m co-teaching a big MA intro course on IR theory and international affairs practice with a practitioner this semester. If I can get this one single point across to my students, so that they really understand it, I think I’ll have given them good value for money.
1 This is not my area of the social sciences, so I can’t speak ex cathedra or anything like it, but the case seems to me to be a strong one on its face.
let’s go through Exum’s rules, shall we?
War is a human endeavor. I recognize that it is a phenomenon that does not conform to neat mathematical equations.
I will use quantitative analysis in conjunction with theory and qualitative analysis to describe what I see as phenomena in war and peace. I will be honest about the limits of both my theory and my analysis
Of course. Good job nailing the compulsories so far.
In war and peace, the variables are infinite, and not everything can be measured or assigned a numerical value
Um… the variables are infinite on just about every dimension of life. No operationalization, econometric equation or formal model is going to completely capture reality. I guarantee you, however, that no qualitiative analysis will perfectly capture reality either (I will further note that qualitative scholars often fool themselves into believing this is not the case, which gets them into all sorts of trouble — but some quant jockeys commit this sin as well). This doesn’t mean you give up on explanation — it just means you acknowledge the limitations of your approach.
I will not use numbers to signify what are fundamentally qualitative assessments without acknowledging to my reader that I have done so in order to satisfy a departmental requirement, gain tenure, or get published in the APSR. Or because I have been in graduate school for so long that I have forgotten how to effectively write in prose.
Yeah, this is where Exum’s manifesto departs from the land of common sense and enters the world of unadulterated horses**t. First, I’ve occasionally used this kind of data, and I sure as hell didn’t do it to get tenure — I did it because I thought it was a good way to test my explanation. Second, whether someone can write clear and crisp prose has nothing to do with whether they use quantitative methods or not. That Exum seems not to know this is the first sign that we’re dealing with some very muddled thinking.
I recognize there are no mathematical equations in Vom Kriege and that it is nonetheless unlikely that my legacy will transcend that of Clausewitz.
Um… I could provide the undisputed, univerally-hailed-by-all explanation for why the United States invaded Iraq in 2003 and my legacy wouldn’t transcend Clausewitz. Or Thucydides. But that’s a really high bar to set.
Just to turn things around, there are plenty of mathematical equations in Strategy of Conflict and it is nevertheless likely that Exum’s — or your — legacy will never transcend that of Thomas Schelling.
I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do.
I think there is some truth to this statement. It is also a fair statement, however, that very few graduate students in security studies have ever served a day in uniform yet probably know more about the causes of war than those squad leaders do.
Much to my amusement, this post on the utlity of quantitative analysis caused quite a stir in the international relations blogosphere. I don’t know if folks in security studies just don’t have a sense of humor or if it’s true what Kissinger said about how university politics are vicious precisely because the stakes are so small. But what I think happened is that Stephen Walt read my post, chuckled, and his chuckling did two things: 1) it brought a lot of people to this site who were not aware that the posts on this blog are meant to be light and irreverent, and 2) it opened up an old fault line in security studies between traditionalists like Walt who aren’t so impressed by quantitative analysis and the Young Turks and political economists who have pushed to make it ascendent in political science departments across the United States. I have about as much interest getting involved in these scholarly disputes as I do catching the Ebola virus. But I did find some of the reaction pretty amusing. Like the fact that Hein Goemans, a brilliant scholar at the University of Rochester, was writing comments on my blog at 5:17 on a Friday afternoon. (Hein, buddy, it’s happy hour. Put down the TI-89, get off the internets and go drink a beer.) Or the fact that Cranky Dan Drezner was left in a cursing, sputtering rage over at his Foreign Policy blog. (I was particularly hurt that Drezner didn’t see the humor in my post, as I have always found his willingness to hold forth on the peoples and politics of the Arabic-speaking world and Iran without any time spent in the region or training in its languages to be hilarious.)
All this shit is academic jargon that I don’t know or care about, but I do love a good pwning, and Andrew Exum delivers
Shorter Exum: “the posts on this blog are meant to be light and irreverent…. I am sorry that folks got their proverbial panties in a twist about a post that was meant to be funny.” He then outsourced a more substantive response to Scott Wedman, who said eminently reasonable things.
According to Spencer Ackerman, Exum also pwned me.
Some are dissatisfied with this response. As for me… meh. If Exum’s original post really was intended as a humorous lark, then so be it. I apologize for misinterpreting and overreacting – though I gotta say, the bulk of his recent posts aren’t exactly overflowing with wit.
I’m not quite sure what to say, other than that this isn’t much of a response. Note, though, that he obliquely makes the same argument he made last week, criticizing Dan Drezner’s “willingness to hold forth on the peoples and politics of the Arabic-speaking world and Iran without any time spent in the region or training in its languages.”
Richard Pipes made a similar argument when he argued that despite his lack of expertise in nuclear weapons or security studies he was qualified to lead the Team B project because of his “deep knowledge of the Russian soul.” And we all remember how that turned out.
UPDATE: Heather Hurlburt and Dan Drezner at Bloggingheads