Are we reaching the limits of hypothetical approaches to international politics?

I just returned from the American Political Science Association (APSA) annual meeting in Boston. I met up with some friends, connected with new people, provided some thoughts on academia to a grad student who reached out to me. My co-authors and I presented a paper from our new project on the role of religion in the UN.

And I actually went to panels that weren’t mine. As anyone who attends conferences knows, there is a temptation to just show up for your panel and then socialize or play tourist the rest of the time. And I’ll admit I did miss a few panels I meant to attend, especially the 8:00 AM ones. But I made it to enough to see what junior scholars in my area (religion, foreign policy, political violence) are working on.

A lot of the papers were survey experiments. One particularly interesting panel looked at support for political violence. Three of the four papers were survey experiments. For people just entering academia, survey experiments seem to be the method of choice. This caused me to become concerned.

The growing dominance of survey experiments

First, for those who are outside of academia or who don’t follow methods, I should explain survey experiments. According to Huber and Graham, they are: “an experiment whose randomly assigned treatment is administered within the context of a survey.” They usually take the form of vignettes provided to survey respondents that vary in some important way: details of the incident discussed, the individuals involved, or subtle framing of the issue. Respondents are randomly assigned the treatment or the control, and the difference in the outcome can be said to be the result of the treatment.

I guess I shouldn’t be surprised at their popularity. Experimental approaches to international politics—the political science sub-fields of international relations and comparative politics—were starting to become “the thing” when I was finishing graduate school, building off of general methodological interest in design- rather than model-based inference. They let you answer questions that were difficult to get at with observational data. And, as with all methods trends, it became a self-reinforcing cycle: growing interest in experiments meant schools with PhD programs wanted faculty members who could teach these methods to their students, thus helping them get jobs.

As more and more of international politics becomes survey experiments, more and more of our research becomes hypothetical

Of course there are issues. It’s an artificial setting. There are questions about external validity, and aggregation up from changes in opinion to changes in behavior. There were major ethical issues with field experiments—a related method—in which people would actually manipulate real-world policies to see the results. I remember articles in which authors got governments to randomly provide services to different districts, and one in which foreign aid was randomly provided and denied to different regions. It just never felt right to me, especially as the subjects of the experiments had no chance to consent.

I’m not going to rant about my own regression-based methods training seeming to become obsolete. I refuse to accept that I’m old enough to rant about changes in the discipline. And there are plenty of issues with regressions and observational data. Also, the approach has gotten better. The first survey experiment I saw presented involved asking college students about war, i.e. major issues of external validity. What I saw presented at APSA, by contrast, mixed field work with survey experiments, as scholars were running the experiments with people experiencing the situation they were studying. I also haven’t seen as many of the ethically-iffy field experiments, although maybe they’re still out there.

My worry is that the many incentives to use survey experiments may lead them to be the norm for how we study the world. Anything that isn’t a survey experiment may be dismissed as invalid. I’ve seen some of this in my own professional experience. I worked at a research organization that specialized in public opinion. Polling is great, but also has a lot of issues. However, because their focus was on public opinion surveys, anything that was not a survey was seen as suspect, limiting the questions that could or would be asked. A development like that at the scale of all international politics (or at least its quantitative wing) would be a problem.

Hypothetical questions

Why would it be a problem? If we are open about the limitations of survey experiments—as any good scholar should be with their methods—what is the issue with our field gravitating towards them at the expense of other methods?

The concern, to me, is that this risks turning international politics into a fundamentally different approach to the world. Observational data looks at something happening and attempts to find patterns in it. The randomized control trials survey experiments aspire to apply a treatment—such as medical interventions—to a real-world population to study the effects. The psychological studies on which survey experiments are based provide some kind of cue or prompt to study how the brain reacts.

Survey experiments in international politics are none of these. They are not actually studying real political events. They are not changing facts on the ground (and when experimental approaches did it was unethical). And psychological studies are meant to study changes in how people think; survey experiments in international politics claim such shifts affect political behavior.

But the use of this method needs to be accompanied by reflection on how hypothetical we want our science to become.

As more and more of international politics research becomes survey experiments, more and more of our research becomes hypothetical. We are not studying or explaining anything that happened. We are developing hypothetical scenarios and looking at how people respond to those scenarios.

And it would be one thing if these survey experiments were leading to an accumulation of knowledge, circumscribed as it may be. Based on what I’ve seeing, however, I worry that’s not the case. Most of the findings I saw were null. This has also been the case for manuscripts using survey experiments that I have reviewed in journals. It’s possible that survey experiments have already picked the low-hanging fruit, and there will be diminishing returns moving forwards.

Now, much of this involved complicated subjects: interfaith dialogue, mobilization towards political violence, etc. It may be easier to find results in areas with more visible and discrete behaviors, like voting. But that’s also an issue, as complications with the studies may lead scholars to avoid “tricky” but important areas.

What to do

I’m not saying we should stop doing survey experiments. They bring a lot of benefits to the sub-fields. But we need to be careful with their use. Embedding them in a broader research program can be helpful. So take the findings of a survey experiment and apply them to an observational study. Some book-lengths studies have done this.

Another way forward is an additional trend I noticed at APSA. A lot of papers, especially by junior scholars, used natural language processing and automated content analysis to turn unsystematized content into analyzable data. This is another area that’s been in development for awhile, and I remember learning an R program in graduate school. The computational tools and skills have improved, opening up a great opportunities for research that previously were limited to not-great proxies.

I should admit my current work makes use of this, so maybe I’m a bit biased. And there are issues: the data it produces is “messy” as my APSA discussant put it, and it does require advanced training not available to everyone.

So keep working on survey experiments. Even if this is a fad that takes over international politics, it’s happened before. But the use of this method needs to be accompanied by reflection on how hypothetical we want our science to become.