The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses

Abstract

Variance between studies in a meta-analysis will exist. This heterogeneity may be of clinical, methodological or statistical origin. The last of these is quantified by the I(2) -statistic. We investigated, using simulated studies, the accuracy of I(2) in the assessment of heterogeneity and the effects of heterogeneity on the predictive value of meta-analyses. The relevance of quantifying I(2) was determined according to the likely presence of heterogeneity between studies (low, high, or unknown) and the calculated I(2) (low or high). The findings were illustrated by published meta-analyses of selective digestive decontamination and weaning protocols. As expected, I(2) increases and the likelihood of drawing correct inferences from a meta-analysis decreases with increasing heterogeneity. With low levels of heterogeneity, I(2) does not appear to be predictive of the accuracy of the meta-analysis result. With high levels of heterogeneity, even meta-analyses with low I(2) -values have low predictive values. Most commonly, the level of heterogeneity in a meta-analysis will be unknown. In these scenarios, I(2) determination may help to identify estimates with low predictive values (high I(2) ). In this situation, the results of a meta-analysis will be unreliable. With low I(2) -values and unknown levels of heterogeneity, predictive values of pooled estimates may range extensively, and findings should be interpreted with caution. In conclusion, quantifying statistical heterogeneity through I(2) -statistics is only helpful when the amount of clinical heterogeneity is unknown and I(2) is high. Objective methods to quantify the levels of clinical and methodological heterogeneity are urgently needed to allow reliable determination of the accuracy of meta-analyses.