Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature
- PMID: 11106885
- DOI: 10.1016/s0895-4356(00)00242-0
Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature
Abstract
Publication and selection biases in meta-analysis are more likely to affect small studies, which also tend to be of lower methodological quality. This may lead to "small-study effects," where the smaller studies in a meta-analysis show larger treatment effects. Small-study effects may also arise because of between-trial heterogeneity. Statistical tests for small-study effects have been proposed, but their validity has been questioned. A set of typical meta-analyses containing 5, 10, 20, and 30 trials was defined based on the characteristics of 78 published meta-analyses identified in a hand search of eight journals from 1993 to 1997. Simulations were performed to assess the power of a weighted regression method and a rank correlation test in the presence of no bias, moderate bias or severe bias. We based evidence of small-study effects on P < 0.1. The power to detect bias increased with increasing numbers of trials. The rank correlation test was less powerful than the regression method. For example, assuming a control group event rate of 20% and no treatment effect, moderate bias was detected with the regression test in 13.7%, 23.5%, 40.1% and 51.6% of meta-analyses with 5, 10, 20 and 30 trials. The corresponding figures for the correlation test were 8.5%, 14.7%, 20.4% and 26.0%, respectively. Severe bias was detected with the regression method in 23.5%, 56.1%, 88.3% and 95.9% of meta-analyses with 5, 10, 20 and 30 trials, as compared to 11.9%, 31.1%, 45.3% and 65.4% with the correlation test. Similar results were obtained in simulations incorporating moderate treatment effects. However the regression method gave false-positive rates which were too high in some situations (large treatment effects, or few events per trial, or all trials of similar sizes). Using the regression method, evidence of small-study effects was present in 21 (26.9%) of the 78 published meta-analyses. Tests for small-study effects should routinely be performed in meta-analysis. Their power is however limited, particularly for moderate amounts of bias or meta-analyses based on a small number of small studies. When evidence of small-study effects is found, careful consideration should be given to possible explanations for these in the reporting of the meta-analysis.
Comment in
-
Association between maternal age at conception and risk of idiopathic clubfoot.Acta Orthop. 2016 Jun;87(3):291-5. doi: 10.3109/17453674.2016.1153359. Epub 2016 Feb 22. Acta Orthop. 2016. PMID: 26901038 Free PMC article.
Similar articles
-
A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints.Stat Med. 2006 Oct 30;25(20):3443-57. doi: 10.1002/sim.2380. Stat Med. 2006. PMID: 16345038
-
Arcsine test for publication bias in meta-analyses with binary outcomes.Stat Med. 2008 Feb 28;27(5):746-63. doi: 10.1002/sim.2971. Stat Med. 2008. PMID: 17592831
-
Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes.Stat Med. 2002 Sep 15;21(17):2465-77. doi: 10.1002/sim.1224. Stat Med. 2002. PMID: 12205693
-
Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task Force [Internet].Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr. Report No.: 14-05207-EF-1. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr. Report No.: 14-05207-EF-1. PMID: 24783270 Free Books & Documents. Review.
-
Behavioral and Pharmacotherapy Weight Loss Interventions to Prevent Obesity-Related Morbidity and Mortality in Adults: An Updated Systematic Review for the U.S. Preventive Services Task Force [Internet].Rockville (MD): Agency for Healthcare Research and Quality (US); 2018 Sep. Report No.: 18-05239-EF-1. Rockville (MD): Agency for Healthcare Research and Quality (US); 2018 Sep. Report No.: 18-05239-EF-1. PMID: 30354042 Free Books & Documents. Review.
Cited by
-
Integrating antiretroviral therapy into antenatal care and maternal and child health settings: a systematic review and meta-analysis.Bull World Health Organ. 2013 Jan 1;91(1):46-56. doi: 10.2471/BLT.12.107003. Epub 2012 Nov 28. Bull World Health Organ. 2013. PMID: 23397350 Free PMC article. Review.
-
A Systematic Review and Meta-Analysis Protocol on How Best to Use Non-Pharmacologic Therapies to Manage Chronic Low Back Pain and Associated Depression.J Pain Res. 2022 Nov 4;15:3509-3521. doi: 10.2147/JPR.S380058. eCollection 2022. J Pain Res. 2022. PMID: 36394058 Free PMC article.
-
Relationship between dominance hierarchy steepness and rank-relatedness of benefits in primates.Behav Ecol. 2024 Aug 13;35(5):arae066. doi: 10.1093/beheco/arae066. eCollection 2024 Sep-Oct. Behav Ecol. 2024. PMID: 39193469 Free PMC article.
-
The Mechanisms for the Association of Cancer and Esophageal Dysmotility Disorders.Med Sci (Basel). 2021 May 21;9(2):32. doi: 10.3390/medsci9020032. Med Sci (Basel). 2021. PMID: 34064058 Free PMC article. Review.
-
Artesunate Plus Amodiaquine (AS+AQ) Versus Artemether -Lumefantrine (AL) for the Treatment of Uncomplicated Plasmodium Falciparum Malaria in Sub-Saharan Africa-A Meta-Analysis.Afr J Infect Dis. 2010;4(2):20-8. doi: 10.4314/ajid.v4i2.55149. Afr J Infect Dis. 2010. PMID: 23878697 Free PMC article.
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
Medical