Resolving the replication crisis in social psychology? A new proposal
By: Brian D. Earp
In the last few years a “replication crisis” has triggered soul searching in psychology – and in particular social psychology. Among other issues, the crisis was sparked when several widely cited findings in the field appeared to falter before replication attempts carried out by independent researchers (for an overview of the crisis, see my recent paper with David Trafimow, “Replication, Falsification, and the Crisis of Confidence in Social Psychology”).
What was surprising about this turn of events was not only that the findings (apparently) failed to replicate—though note that “failed” replications can be difficult to interpret—but also that independent attempts at direct replication had (a) been carried out in the first place, and (b) actually been published in professional journals.
This was surprising because both (a) and (b) are infrequent occurrences in psychology (as well as in other disciplines), notwithstanding the fact that independent replication of major findings is supposed to be one of the hallmarks of a progressive science. Jim Everett and I mention some of the reasons for this in in a forthcoming paper in Frontiers in Psychology:
(1) independent, direct replications of others’ findings can be time-consuming for the replicating researcher;
(2) they are likely to take energy and resources directly away from other projects that reflect one’s own original thinking;
(3) they are generally harder to publish (in large part because they are viewed as being unoriginal);
(4) even if they are published, they are likely to be seen as “bricklaying” exercises, rather than as major contributions to the field; and accordingly:
(5) they bring less recognition and reward, and even basic career security, to their authors.
Reflecting on these considerations, we suggest that the ongoing “crisis” of replication should be be interpreted as a disciplinary social dilemma, with the problem facing early-career researchers being especially acute.
Social dilemmas are situations in which collective interests are at odds with private interests. They have two fundamental characteristics: first, that each individual receives a higher payoff for defecting from what is in the collective interest than for cooperating; and second, that all individuals are better off if they all cooperate than if they all defect.
See how this applies to the replication crisis. While it is in everyone’s interest that high-quality, direct replications of key studies in the field are conducted (so that we can know what degree of confidence to place in previous findings from the literature), it is not typically in any particular researcher’s interest to spend her time conducting such replications—for the reasons outlined above.
This is especially the case for younger researchers. Not only do they have less career security to begin with, they also have a limited window of time in which to find it if they are to continue working as scientists at all. The problem is that time spent on a replication study is time lost on original research. It really is a zero sum game.
So how can this dilemma be resolved? In our new paper, Jim and I suggest that what is needed is a structural solution that can address the “collective action” nature of the problem. Here is our proposal, in a nutshell:
“As a condition of receiving their PhD from any accredited institution, graduate students in psychology should be required to conduct, write up, and submit for publication a high-quality replication attempt of at least one key finding from the literature, focusing on the area of their doctoral research.”
Because this would be a requirement for all students in accredited programs, it would not put any individual student at a disadvantage for doing her part to contribute to the resolution of the crisis. It would also “remove the untenable burden from early-career researchers (post PhD) to spend time on replication studies during a critical period of their career development, in a context in which, in any event, such a burden could not be uniformly enforced. Moreover, since all such researchers would already have contributed something meaningful toward the resolution of the replication crisis, they could feel free to spend their precious research hours on whatever projects they deemed to be most advantageous either personally or professionally.”
There is of course the niggling question of how this would actually happen. For starters, the idea would have to be floated at the right level of decision-making authority, such that the new requirement could be instituted as a matter of policy for all accredited programs simultaneously. (This is because the problem with social dilemmas—in general—is that they can’t be resolved in a piecemeal fashion: each individual who “sticks her neck out” to do the right thing typically pays a penalty for doing so.) But before that could happen, the basic soundness of the idea would have to be established. And that means subjecting it to critical scrutiny.
So, with my colleague and co-author Jim Everett, I humbly submit for your consideration our paper, “A Tragedy of the (Academic) Commons: Interpreting the Replication Crisis in Psychology as a Social Dilemma for Early-Career Researchers.” We’re eager to hear your thoughts and feedback.
Everett, J. A. C.**, & Earp, B. D.** (forthcoming). A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers in Psychology. Available online head of print at https://www.academia.edu/14162686/A_tragedy_of_the_academic_commons_Interpreting_the_replication_crisis_in_psychology_as_a_social_dilemma_for_early-career_researchers ** Co-first authors.
Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6(621), 1-11. Available at http://journal.frontiersin.org/article/10.3389/fpsyg.2015.00621/abstract
Earp, B. D., Everett, J. A. C., Madva, E. N., & Hamlin, J. K. (2014). Out, damned spot: Can the “Macbeth Effect” be replicated? Basic and Applied Social Psychology, 36(1), 91-98. Available at http://www.tandfonline.com/doi/abs/10.1080/01973533.2013.856792#.Va68RWCGjds
** Note: this post is partially adapted from a similar piece by my co-author Jim Everett, originally published at the Practical Ethics blog, hosted by the University of Oxford. See here http://blog.practicalethics.ox.ac.uk/2015/07/could-phd-students-solve-the-replication-crisis-in-psychology/.