An official website of the United States government
Abstract Meta‐analysis (MA), a powerful tool for synthesizing reported results, is influential in ecology. While ecologists have long been well‐informed on the potential problems associated with nonindependence in experimental work (e.g., pseudoreplication), they have, until recently, largely neglected this issue in MA. However, results used in MAs are likely much more similar when they come from the same locality, system, or laboratory. A simple and common form of nonindependence in MA arises when multiple data points, that is, observed effect sizes, come from the same paper. We obtained original data from 20 published MAs, reconstructed the published analyses, and then, for 14 that had not accounted for a paper effect, used three approaches to evaluate whether within‐paper nonindependence was a problem. First, we found that “nonsense” explanatory variables added to the original analyses were statistically significant (p < 0.05) far more often than the expected 5% (25%–50% for four nonsense variables). For example, the number of vowels in the first author's name had a significant effect 50% of the time. Second, we found that an added dummy variable, which was randomly assigned at one of two levels, was statistically significant an average of 38% of the time, far exceeding the expected 5%. Even after including a random paper effect in the analyses, there was still an excess of significant results, suggesting that the within‐paper nonindependence was more complex than modeled with the random paper effect. Third, we repeated the original MAs that did not include random paper effects (n = 14 MAs) but added a random paper effect to each revised analysis. In 12 out of the 14 MAs, an added random effect was statistically significant (indicating group nonindependence that was not accounted for in the original analyses), and often the original inferences were substantially altered. Further, incorporating random paper effects was not a sufficient solution to nonindependence. Thus, problems resulting from nonindependence are often substantial, and accounting for the problem will likely require careful consideration of the details of the potential dependence among observed effect sizes. MAs that do not properly account for this problem may reach unwarranted conclusions.
more »
« less
An assessment of statistical methods for nonindependent data in ecological meta‐analyses
Abstract In ecological meta‐analyses, nonindependence among observed effect sizes from the same source paper is common. If not accounted for, nonindependence can seriously undermine inferences. We compared the performance of four meta‐analysis methods that attempt to address such nonindependence and the standard random‐effect model that ignores nonindependence. We simulated data with various types of within‐paper nonindependence, and assessed the standard deviation of the estimated mean effect size and Type I error rate of each method. Although all four methods performed substantially better than the standard random‐effects model that assumes independence, there were differences in performance among the methods. A two‐step method that first summarizes the multiple observed effect sizes per paper using a weighted mean and then analyzes the reduced data in a standard random‐effects model, and a robust variance estimation method performed consistently well. A hierarchical model with both random paper and study effects gave precise estimates but had a higher Type I error rates, possibly reflecting limitations of currently available meta‐analysis software. Overall, we advocate the use of the two‐step method with a weighted paper mean and the robust variance estimation method as reliable ways to handle within‐paper nonindependence in ecological meta‐analyses.
more »
« less
- PAR ID:
- 10453175
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Ecology
- Volume:
- 101
- Issue:
- 12
- ISSN:
- 0012-9658
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Despite the wide application of meta‐analysis in ecology, some of the traditional methods used for meta‐analysis may not perform well given the type of data characteristic of ecological meta‐analyses.We reviewed published meta‐analyses on the ecological impacts of global climate change, evaluating the number of replicates used in the primary studies (ni) and the number of studies or records (k) that were aggregated to calculate a mean effect size. We used the results of the review in a simulation experiment to assess the performance of conventional frequentist and Bayesian meta‐analysis methods for estimating a mean effect size and its uncertainty interval.Our literature review showed thatniandkwere highly variable, distributions were right‐skewed and were generally small (medianni = 5, mediank = 44). Our simulations show that the choice of method for calculating uncertainty intervals was critical for obtaining appropriate coverage (close to the nominal value of 0.95). Whenkwas low (<40), 95% coverage was achieved by a confidence interval (CI) based on thetdistribution that uses an adjusted standard error (the Hartung–Knapp–Sidik–Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap orzdistribution CIs had lower coverage. Despite the importance of the method to calculate the uncertainty interval, 39% of the meta‐analyses reviewed did not report the method used, and of the 61% that did, 94% used a potentially problematic method, which may be a consequence of software defaults.In general, for a simple random‐effects meta‐analysis, the performance of the best frequentist and Bayesian methods was similar for the same combinations of factors (kand mean replication), though the Bayesian approach had higher than nominal (>95%) coverage for the mean effect whenkwas very low (k < 15). Our literature review suggests that many meta‐analyses that usedzdistribution or bootstrapping CIs may have overestimated the statistical significance of their results when the number of studies was low; more appropriate methods need to be adopted in ecological meta‐analyses.more » « less
-
This protocol is for a study that examines personalized adaptive learning (PAL) of undergraduate students with a focus on mathematics outcomes. This protocol addresses two meta-analytic studies: 1) Examination of PAL within the population of undergraduate students enrolled in mathematics courses broadly (across all mathematics courses with the type of course as a moderator); and 2) Specifically in college algebra, which is a STEM pipeline mathematics course. Undergraduate mathematics courses can be roadblocks to college degree attainment. PAL offers the opportunity to develop mathematical skills in a personalized way, while enabling instructors to identify students’ struggle areas and take remedial action.Partnering with academic librarians, databases likely to index relevant studies were identified, the search strategy was refined, and keywords and subject indexing terms were identified. Titles and abstracts were independently screened by two reviewers. Full-text review is in progress. Retained studies will have data extracted that will allow the results to be meta-analyzed.The standardized mean difference will be the main effect size computed. Multilevel meta-analysis will be used to address the hierarchical data structure. Robust variance estimation will be used to estimate standard errors and hypothesis tests, addressing dependencies of effect sizes within studies. Prediction intervals for effect sizes, for study-average effect sizes, and heterogeneity statistics will be reported. Subgroup analyses will evaluate the relationship between moderators and the magnitude of the effect size. Cochrane’s criteria for assessing the risk of bias in studies with a separate control group will be applied. Reporting bias will be assessed visually with funnel plots.more » « less
-
We examined the effect of estimation methods, maximum likelihood (ML), unweighted least squares (ULS), and diagonally weighted least squares (DWLS), on three population SEM (structural equation modeling) fit indices: the root mean square error of approximation (RMSEA), the comparative fit index (CFI), and the standardized root mean square residual (SRMR). We considered different types and levels of misspecification in factor analysis models: misspecified dimensionality, omitting cross-loadings, and ignoring residual correlations. Estimation methods had substantial impacts on the RMSEA and CFI so that different cutoff values need to be employed for different estimators. In contrast, SRMR is robust to the method used to estimate the model parameters. The same criterion can be applied at the population level when using the SRMR to evaluate model fit, regardless of the choice of estimation method.more » « less
-
Longitudinal clinical trials for which recurrent events endpoints are of interest are commonly subject to missing event data. Primary analyses in such trials are often performed assuming events are missing at random, and sensitivity analyses are necessary to assess robustness of primary analysis conclusions to missing data assumptions. Control‐based imputation is an attractive approach in superiority trials for imposing conservative assumptions on how data may be missing not at random. A popular approach to implementing control‐based assumptions for recurrent events is multiple imputation (MI), but Rubin's variance estimator is often biased for the true sampling variability of the point estimator in the control‐based setting. We propose distributional imputation (DI) with corresponding wild bootstrap variance estimation procedure for control‐based sensitivity analyses of recurrent events. We apply control‐based DI to a type I diabetes trial. In the application and simulation studies, DI produced more reasonable standard error estimates than MI with Rubin's combining rules in control‐based sensitivity analyses of recurrent events.more » « less
