By Dave Nussbaum
Simine Vazire (@siminevazire) is at it again — stirring up the blogosphere with provocative and insightful posts. If you’re not already following her blog, sometimes i’m wrong, you definitely should. Most recently she posted some musings on counterintuitive resarch findings that have already provoked interesting responses fromSanjay Srivastava (@hardsci) and Brent Donnellan. In the spirit of my self-proclaimed mission for this blog, which is to promote more discussion and engagement, I thought I’d present a quick sample of the three posts, and encourage you to go read them in their entirety. Let’s start with Simine’s original post, where she first lays out the argument for holding counterintuitive findings, then presents its foil:
the counterargument, as far as i can tell, goes something like this: counterintuitive findings do more to advance our knowledge because they force us to rethink our old theories. in kuhnian language,* these are the anomalies that lead to a crisis in the paradigm, and sometimes force scientific revolutions. findings that are intuitive, or fit with the existing paradigm, can add more bricks to the wall but will never lead to huge leaps in knowledge. sounds good. i agree that counterintuitive findings can be breakthroughs. but if they cause so much upheaval, isn’t that all the more reason to hold them to a higher evidentiary standard? if they have the potential to undo entire paradigms, don’t we want to be super sure about them?
Next, Sanjay Srivastava wonders what we really mean by counterintuitive and whether it’s often used merely as a ploy:
What I suspect that means is that “counterintuitive” is often just a rhetorical strategy for writing introduction sections and marketing our work. No matter how your results turned out, you can convince your audience that they once thought the opposite. Because chances are very good that they did. A skilled writer can exploit the same mechanisms that lead to hindsight bias to set people up, and then surprise! show them that the results went the other way. I would not claim that this describes all instances of counterintuitive, but I think it describes a lot of them. As Simine points out, many people in psychology say that counterintuitive findings are more valuable — so clearly there is an incentive to frame things that way. (Counterintuitive framing is also a great way to sell a lot of books.)
Finally, Brent Donnelan weighs in on the issue of how much evidence is needed for counterintuitive claims and lists some of the things that cause him to raise an eyebrow: 1) Large effect sizes coupled with small sample sizes, 2) Conceptual replications but no direct replications, and 3) Breathless press releases. Not only should you go read these posts, don’t miss the comment sections either, where there are more interesting points being made. My thoughts on counterintuitive findings are very much in line with points that Roger Giner-Sorolla makes in his comment on Simine’s blog. I think the meaning of counterintuitive has evolved over time and the new version, as Sanjay points out, is often rhetorical framing — outcomes that nobody would have expected, but largely because nobody had any reason to expect them, including the researchers themselves. That said, the “original” meaning of counterintuitive was rightly considered very valuable and it continues to be. Roger uses the same example that jumped to my mind — the Festinger and Carlsmith classic dissonance study with the boring peg-turning task. The dominant theory at the time, as well as popular intuition, would have openly scoffed at the suggestion that paying someone one dollar would lead them to report finding the task more interesting than paying them twenty dollars. But the counterintuitive finding was well grounded in theory. It overturned what people had previously thought for well articulated set of reasons. That sort of counterintuitive finding is extremely valuable. Running a study with 14 participants per condition that finds that Scorpios are less introverted when sitting than when standing may confound my intuitions, but not in a particularly valuable way. Cindy Pickett expressed a similar sentiment on Facebook:
I think we need to refine the term “counterintuitive” in a way that takes into account the different ways in which the term is used. There are some findings that are deemed unbelievable because it’s impossible to even fathom what mechanisms or series of psychological events could possibly lead to the outcome (e.g., ESP). Other findings are labeled “counterintuitive” because they too were unexpected but these findings are often accompanied by an “Aha!” moment where it makes sense why the outcome was obtained (e.g., cognitive dissonance).
Of course this doesn’t resolve the question of how much evidence is needed to accept a counterintuitive finding. On that front, I really like the way Sanjay put it in his talk at this year’s SPSP (although he was speaking more broadly, not about counterintuitive findings in particular): lower the bar for accepting papers, raise the bar for accepting conclusions. Feel free to comment here, or on any of the other blogs directly, or on facebook. Have another topic that you’d like to see discussed? Email me at SPSPblog@gmail.com and let me know.