A manifesto for reproducible science

doi:10.1038/s41562-016-0021

. 2017 Jan 10;1(1):0021.

doi: 10.1038/s41562-016-0021.

A manifesto for reproducible science

Marcus R Munafò^{1

2}, Brian A Nosek^{3

4}, Dorothy V M Bishop⁵, Katherine S Button⁶, Christopher D Chambers⁷, Nathalie Percie du Sert⁸, Uri Simonsohn⁹, Eric-Jan Wagenmakers¹⁰, Jennifer J Ware¹¹, John P A Ioannidis^{12

13

14}

Affiliations

¹ MRC Integrative Epidemiology Unit, University of Bristol, Bristol BS8 2BN, UK.
² UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK.
³ Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA.
⁴ Center for Open Science, Charlottesville, Virginia 22903, USA.
⁵ Department of Experimental Psychology, University of Oxford, 9 South Parks Road, Oxford OX1 3UD, UK.
⁶ Department of Psychology, University of Bath, Bath BS2 7AY, UK.
⁷ Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff CF24 4HQ, UK.
⁸ National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), London NW1 2BE, UK.
⁹ The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA.
¹⁰ Department of Psychology, University of Amsterdam, Amsterdam 1018 WT, Netherlands.
¹¹ CHDI Management/CHDI Foundation, New York, New York 10001, USA.
¹² Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford 94304, California, USA.
¹³ Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford 94305, California, USA.
¹⁴ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford 94305, California, USA.

PMID: 33954258
PMCID: PMC7610724
DOI: 10.1038/s41562-016-0021

A manifesto for reproducible science

Marcus R Munafò et al. Nat Hum Behav. 2017.

. 2017 Jan 10;1(1):0021.

doi: 10.1038/s41562-016-0021.

Authors

Affiliations

¹ MRC Integrative Epidemiology Unit, University of Bristol, Bristol BS8 2BN, UK.
² UK Centre for Tobacco and Alcohol Studies, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK.
³ Department of Psychology, University of Virginia, Charlottesville, Virginia 22904, USA.
⁴ Center for Open Science, Charlottesville, Virginia 22903, USA.
⁵ Department of Experimental Psychology, University of Oxford, 9 South Parks Road, Oxford OX1 3UD, UK.
⁶ Department of Psychology, University of Bath, Bath BS2 7AY, UK.
⁷ Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff CF24 4HQ, UK.
⁸ National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), London NW1 2BE, UK.
⁹ The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA.
¹⁰ Department of Psychology, University of Amsterdam, Amsterdam 1018 WT, Netherlands.
¹¹ CHDI Management/CHDI Foundation, New York, New York 10001, USA.
¹² Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford 94304, California, USA.
¹³ Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford 94305, California, USA.
¹⁴ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford 94305, California, USA.

PMID: 33954258
PMCID: PMC7610724
DOI: 10.1038/s41562-016-0021

Abstract

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

PubMed Disclaimer

Conflict of interest statement

Competing interests M.R.M, together with C.D.C and D.V.M.B., has received funding from the BBSRC (grant number BB/N019660/1) to convene a workshop on advanced methods for reproducible science, and is chair of the CHDI Foundation Independent Statistical Standing Committee. B.A.N. is executive director of the non-profit Center for Open Science with a mission to increase openness, integrity and reproducibility of research. N.P.d.S. leads the NC3Rs programme of work on experimental design, which developed the ARRIVE guidelines and Experimental Design Assistant. J.J.W. is director, experimental design, at CHDI Management/CHDI Foundation, a non-profit biomedical research organization exclusively dedicated to developing therapeutics for Huntington’s disease. The other authors declare no competing interests.

Figures

**Figure 1. Threats to reproducible science.**
An idealized version of the hypothetico-deductive model of the scientific method is shown. Various potential threats to this model exist (indicated in red), including lack of replication, hypothesizing after the results are known (HARKing), poor study design, low statistical power, analytical flexibility, P-hacking, publication bias and lack of data sharing. Together these will serve to undermine the robustness of published research, and may also impact on the ability of science to self-correct.

**Figure 2. The impact of introducing badges for data sharing.**
In January 2014, the journal *Psychological Science* (PSCI) introduced badges for articles with open data. Immediately afterwards, the proportion of articles with open data increased steeply, and by October 2015, 38% of articles in *Psychological Science* had open data. For comparison journals (*Clinical Psychological Science* (CPS), *Developmental Psychology* (DP), *Journal of Experimental Psychology: Learning, Memory and Cognition* (JEPLMC) and *Journal of Personality and Social Psychology* (JPSP)) the proportion of articles with open data remained uniformly low. Figure adapted from ref. , PLoS.

See this image and copyright information in PMC

Cited by

Towards open, reliable, and transparent ecology and evolutionary biology.
O'Dea RE, Parker TH, Chee YE, Culina A, Drobniak SM, Duncan DH, Fidler F, Gould E, Ihle M, Kelly CD, Lagisz M, Roche DG, Sánchez-Tójar A, Wilkinson DP, Wintle BC, Nakagawa S. O'Dea RE, et al. BMC Biol. 2021 Apr 9;19(1):68. doi: 10.1186/s12915-021-01006-3. BMC Biol. 2021. PMID: 33836762 Free PMC article.
Self-supervised machine learning for live cell imagery segmentation.
Robitaille MC, Byers JM, Christodoulides JA, Raphael MP. Robitaille MC, et al. Commun Biol. 2022 Nov 2;5(1):1162. doi: 10.1038/s42003-022-04117-x. Commun Biol. 2022. PMID: 36323790 Free PMC article.
Facilitating bioinformatics reproducibility with QIIME 2 Provenance Replay.
Keefe CR, Dillon MR, Gehret E, Herman C, Jewell M, Wood CV, Bolyen E, Caporaso JG. Keefe CR, et al. PLoS Comput Biol. 2023 Nov 27;19(11):e1011676. doi: 10.1371/journal.pcbi.1011676. eCollection 2023 Nov. PLoS Comput Biol. 2023. PMID: 38011287 Free PMC article.
How to Conduct Responsible Research: A Guide for Graduate Students.
Antes AL, Maggi LB Jr. Antes AL, et al. Curr Protoc. 2021 Mar;1(3):e87. doi: 10.1002/cpz1.87. Curr Protoc. 2021. PMID: 33740292 Free PMC article.
Improving transparency and scientific rigor in academic publishing.
Prager EM, Chambers KE, Plotkin JL, McArthur DL, Bandrowski AE, Bansal N, Martone ME, Bergstrom HC, Bespalov A, Graf C. Prager EM, et al. Cancer Rep (Hoboken). 2019 Feb;2(1):e1150. doi: 10.1002/cnr2.1150. Epub 2018 Dec 2. Cancer Rep (Hoboken). 2019. PMID: 32721132 Free PMC article.

See all "Cited by" articles

References

1. Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed
1. Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed
1. Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed
1. John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed
1. Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- H1 Connect

[1] Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed

[2] Ioannidis John P. A. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed

[3] Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed

[4] Button KS. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed

[5] Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed

[6] Fanelli Daniele. “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE. 2010;5(4):e10068. doi: 10.1371/journal.pone.0010068. - DOI - PMC - PubMed

[7] John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed

[8] John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 2012;23:524–532. doi: 10.1177/0956797611430953. - DOI - PubMed

[9] Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

[10] Makel MC, Plucker JA, Hegarty B. Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 2012;7:537–542. doi: 10.1177/1745691612460688. - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A manifesto for reproducible science

Affiliations

A manifesto for reproducible science

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources