A manifesto for reproducible science

doi:10.1038/s41562-016-0021

. 2017 Jan 10;1(1):0021.

doi: 10.1038/s41562-016-0021.

A manifesto for reproducible science

Marcus R Munafò^{1

2}, Brian A Nosek^{3

4}, Dorothy V M Bishop⁵, Katherine S Button⁶, Christopher D Chambers⁷, Nathalie Percie du Sert⁸, Uri Simonsohn⁹, Eric-Jan Wagenmakers¹⁰, Jennifer J Ware¹¹, John P A Ioannidis^{12

13

14}

Affiliations

¹ MRC Integrative Epidemiology Unit, University of Bristol, Bristol BS8 2BN, UK.
² UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK.
³ Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA.
⁴ Center for Open Science, Charlottesville, Virginia 22903, USA.
⁵ Department of Experimental Psychology, University of Oxford, 9 South Parks Road, Oxford OX1 3UD, UK.
⁶ Department of Psychology, University of Bath, Bath BS2 7AY, UK.
⁷ Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff CF24 4HQ, UK.
⁸ National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), London NW1 2BE, UK.
⁹ The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA.
¹⁰ Department of Psychology, University of Amsterdam, Amsterdam 1018 WT, Netherlands.
¹¹ CHDI Management/CHDI Foundation, New York, New York 10001, USA.
¹² Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford 94304, California, USA.
¹³ Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford 94305, California, USA.
¹⁴ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford 94305, California, USA.

PMID: 33954258
PMCID: PMC7610724
DOI: 10.1038/s41562-016-0021

A manifesto for reproducible science

Marcus R Munafò et al. Nat Hum Behav. 2017.

. 2017 Jan 10;1(1):0021.

doi: 10.1038/s41562-016-0021.

Authors

Affiliations

¹ MRC Integrative Epidemiology Unit, University of Bristol, Bristol BS8 2BN, UK.
² UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK.
³ Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA.
⁴ Center for Open Science, Charlottesville, Virginia 22903, USA.
⁵ Department of Experimental Psychology, University of Oxford, 9 South Parks Road, Oxford OX1 3UD, UK.
⁶ Department of Psychology, University of Bath, Bath BS2 7AY, UK.
⁷ Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff CF24 4HQ, UK.
⁸ National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), London NW1 2BE, UK.
⁹ The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA.
¹⁰ Department of Psychology, University of Amsterdam, Amsterdam 1018 WT, Netherlands.
¹¹ CHDI Management/CHDI Foundation, New York, New York 10001, USA.
¹² Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford 94304, California, USA.
¹³ Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford 94305, California, USA.
¹⁴ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford 94305, California, USA.

PMID: 33954258
PMCID: PMC7610724
DOI: 10.1038/s41562-016-0021

Abstract

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

PubMed Disclaimer

Conflict of interest statement

Competing interests M.R.M, together with C.D.C and D.V.M.B., has received funding from the BBSRC (grant number BB/N019660/1) to convene a workshop on advanced methods for reproducible science, and is chair of the CHDI Foundation Independent Statistical Standing Committee. B.A.N. is executive director of the non-profit Center for Open Science with a mission to increase openness, integrity and reproducibility of research. N.P.d.S. leads the NC3Rs programme of work on experimental design, which developed the ARRIVE guidelines and Experimental Design Assistant. J.J.W. is director, experimental design, at CHDI Management/CHDI Foundation, a non-profit biomedical research organization exclusively dedicated to developing therapeutics for Huntington’s disease. The other authors declare no competing interests.

Figures

**Figure 1. Threats to reproducible science.**
An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication, hypothesizing after the results are known (HARKing), poor study design, low statistical power, analytical flexibility, P-hacking, publication bias and lack of data sharing. Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.

**Figure 2. The impact of introducing badges for data sharing.**
In January 2014, the journal *Psychological Science* (PSCI) introduced badges for articles with open data. Immediately afterwards, the proportion of articles with open data increased steeply, and by October 2015, 38% of articles in *Psychological Science* had open data. For comparison journals (*Clinical Psychological Science* (CPS), *Developmental Psychology* (DP), *Journal of Experimental Psychology: Learning, Memory and Cognition* (JEPLMC) and *Journal of Personality and Social Psychology* (JPSP)) the proportion of articles with open data remained uniformly low. Figure adapted from ref. , PLoS.

See this image and copyright information in PMC

Cited by

Biomedical researchers' perspectives on the reproducibility of research.
Cobey KD, Ebrahimzadeh S, Page MJ, Thibault RT, Nguyen PY, Abu-Dalfa F, Moher D. Cobey KD, et al. PLoS Biol. 2024 Nov 5;22(11):e3002870. doi: 10.1371/journal.pbio.3002870. eCollection 2024 Nov. PLoS Biol. 2024. PMID: 39499707 Free PMC article.
Bayesian evaluation of behavior change interventions: a brief introduction and a practical example.
Heino MTJ, Vuorre M, Hankonen N. Heino MTJ, et al. Health Psychol Behav Med. 2018 Apr 11;6(1):49-78. doi: 10.1080/21642850.2018.1428102. Health Psychol Behav Med. 2018. PMID: 34040821 Free PMC article.
Nine quick tips for open meta-analyses.
Moreau D, Wiebels K. Moreau D, et al. PLoS Comput Biol. 2024 Jul 25;20(7):e1012252. doi: 10.1371/journal.pcbi.1012252. eCollection 2024 Jul. PLoS Comput Biol. 2024. PMID: 39052540 Free PMC article.
A focus shift in the evaluation of misinformation interventions.
Tay LQ, Lewandowsky S, Hurlstone MJ, Kurz T, Ecker UKH. Tay LQ, et al. Harv Kennedy Sch Misinformation Rev. 2023 Oct;4(5):mr-2020-124. doi: 10.37016/mr-2020-124. Epub 2023 Oct 5. Harv Kennedy Sch Misinformation Rev. 2023. PMID: 39301480 Free PMC article.
Plea for a Simple But Radical Change in Scientific Publication: To Improve Openness, Reliability, and Reproducibility, Let's Deposit and Validate Our Results before Writing Articles.
Girault JA. Girault JA. eNeuro. 2022 Sep 15;9(5):ENEURO.0318-22.2022. doi: 10.1523/ENEURO.0318-22.2022. Print 2022 Sep-Oct. eNeuro. 2022. PMID: 36109162 Free PMC article.

See all "Cited by" articles

References

1. Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed
1. Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed
1. Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed
1. John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed
1. Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- H1 Connect

[1] Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed

[2] Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed

[3] Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed

[4] Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed

[5] Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed

[6] Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed

[7] John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed

[8] John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed

[9] Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

[10] Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A manifesto for reproducible science

Affiliations

A manifesto for reproducible science

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources