Skip to main page content
U.S. flag

An official website of the United States government

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Dec 17;116(51):25535-25545.
doi: 10.1073/pnas.1910951116. Epub 2019 Nov 25.

Replicator degrees of freedom allow publication of misleading failures to replicate

Christopher J Bryan  1 , David S Yeager  2 , Joseph M O'Brien  2
Affiliations

Replicator degrees of freedom allow publication of misleading failures to replicate

Christopher J Bryan et al. Proc Natl Acad Sci U S A. .

Erratum in

Abstract

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that "replicator degrees of freedom" make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate that we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field's standard remedies, including preregistration. Implications for how the field could develop more effective methodological standards for replication are discussed.

Keywords: null hacking; p-hacking; replication crisis; reproducibility; researcher degrees of freedom.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interest.

Figures

Fig. 1.
Fig. 1.
Plot of the effect size estimates from 1,200 models in the stage-2 specification curve, which includes the 270 specifications from the stage-1 specification curve and 930 models that represent analytical choices made by the replicating authors. A–F represent subsets of study days. The triangular points along the bottom of each panel indicate specifications using the exact set of 95 covariates used by Gerber et al. (35).
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Camerer C. F., et al. ., Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018). - PubMed
    1. Harris C. R., Coburn N., Rohrer D., Pashler H., Two failures to replicate high-performance-goal priming effects. PLoS One 8, e72467 (2013). - PMC - PubMed
    1. Nelson L. D., Simmons J., Simonsohn U., Psychology’s renaissance. Annu. Rev. Psychol. 69, 511–534 (2018). - PubMed
    1. O. S. Collaboration; Open Science Collaboration , PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). - PubMed
    1. Ranehill E., et al. ., Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychol. Sci. 26, 653–656 (2015). - PubMed

Publication types

LinkOut - more resources