The purposes of this study included conducting a meta-analysis and reviewing the study reporting quality of math interventions implemented in informal learning environments (e.g., the home) by children’s caregivers. This meta-analysis included 25 preschool to third-grade math interventions with 83 effect sizes that yielded a statistically significant summary effect (g = 0.26, 95% CI [0.07, 0.45) on children’s math achievement. Significant moderators of the treatment effect included the intensity of caregiver training and type of outcome measure. There were larger average effects for interventions with caregiver training that included follow-up support and for outcomes that were comprehensive early numeracy measures. Studies met 58.0% of reporting quality indicators, and analyses revealed that quality of reporting has improved in recent years. The results of this study offer several recommendations for researchers and practitioners, particularly given the growing evidence base of math interventions conducted in informal learning environments.
Quantitatively summarizing results from a collection of primary studies with meta-analysis can help answer ecological questions and identify knowledge gaps. The accuracy of the answers depends on the quality of the meta-analysis. We reviewed the literature assessing the quality of ecological meta-analyses to evaluate current practices and highlight areas that need improvement. From each of the 18 review papers that evaluated the quality of meta-analyses, we calculated the percentage of meta-analyses that met criteria related to specific steps taken in the meta-analysis process (i.e., execution) and the clarity with which those steps were articulated (i.e., reporting). We also re-evaluated all the meta-analyses available from Pappalardo et al. [1] to extract new information on ten additional criteria and to assess how the meta-analyses recognized and addressed non-independence. In general, we observed better performance for criteria related to reporting than for criteria related to execution; however, there was a wide variation among criteria and meta-analyses. Meta-analyses had low compliance with regard to correcting for phylogenetic non-independence, exploring temporal trends in effect sizes, and conducting a multifactorial analysis of moderators (i.e., explanatory variables). In addition, although most meta-analyses included multiple effect sizes per study, only 66% acknowledged some type of non-independence. The types of non-independence reported were most often related to the design of the original experiment (e.g., the use of a shared control) than to other sources (e.g., phylogeny). We suggest that providing specific training and encouraging authors to follow the PRISMA EcoEvo checklist recently developed by O’Dea et al. [2] can improve the quality of ecological meta-analyses.
more » « less- Award ID(s):
- 1655426
- PAR ID:
- 10479615
- Editor(s):
- Silva, Daniel de
- Publisher / Repository:
- PlosOne
- Date Published:
- Journal Name:
- PLOS ONE
- Volume:
- 18
- Issue:
- 10
- ISSN:
- 1932-6203
- Page Range / eLocation ID:
- e0292606
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
-
more » « less
Abstract In ecological meta‐analyses, nonindependence among observed effect sizes from the same source paper is common. If not accounted for, nonindependence can seriously undermine inferences. We compared the performance of four meta‐analysis methods that attempt to address such nonindependence and the standard random‐effect model that ignores nonindependence. We simulated data with various types of within‐paper nonindependence, and assessed the standard deviation of the estimated mean effect size and Type I error rate of each method. Although all four methods performed substantially better than the standard random‐effects model that assumes independence, there were differences in performance among the methods. A two‐step method that first summarizes the multiple observed effect sizes per paper using a weighted mean and then analyzes the reduced data in a standard random‐effects model, and a robust variance estimation method performed consistently well. A hierarchical model with both random paper and study effects gave precise estimates but had a higher Type I error rates, possibly reflecting limitations of currently available meta‐analysis software. Overall, we advocate the use of the two‐step method with a weighted paper mean and the robust variance estimation method as reliable ways to handle within‐paper nonindependence in ecological meta‐analyses.
-
null (Ed.)Impairments related to figurative language understanding have been considered to be one of the diagnostic and defining features of autism. Metaphor comprehension and production in autism spectrum disorder (ASD) as compared to typically developing (TD) individuals have been investigated for around thirty years, generally showing an overall advantage for TD groups. We present a preregistered systematic review and meta-analysis including a total of 15 studies that fulfilled our set of inclusion criteria (notably, ASD and TD groups matched in chronological age and verbal- or full-scale IQ). Along with accuracy, we also analyzed group differences in reaction time in the studies that reported them. The results revealed a medium-to-large group difference favoring TD over ASD groups based on accuracy measures, as well as a similar overall advantage for TD groups based on reaction times. There was reliable heterogeneity in effect sizes for group differences in accuracy, which was mostly explained by the effect of verbal intelligence, with differences in metaphor processing being smaller for participants with better verbal skills. Some of the variation in effect sizes may also be attributed to differences in types of metaphor processing tasks. We also evaluated the quality of the studies included in the meta-analysis, and the evidence relating to the potential presence of publication bias.more » « less
-
more » « less
This protocol is for a study that examines personalized adaptive learning (PAL) of undergraduate students with a focus on mathematics outcomes. This protocol addresses two meta-analytic studies: 1) Examination of PAL within the population of undergraduate students enrolled in mathematics courses broadly (across all mathematics courses with the type of course as a moderator); and 2) Specifically in college algebra, which is a STEM pipeline mathematics course. Undergraduate mathematics courses can be roadblocks to college degree attainment. PAL offers the opportunity to develop mathematical skills in a personalized way, while enabling instructors to identify students’ struggle areas and take remedial action.Partnering with academic librarians, databases likely to index relevant studies were identified, the search strategy was refined, and keywords and subject indexing terms were identified. Titles and abstracts were independently screened by two reviewers. Full-text review is in progress. Retained studies will have data extracted that will allow the results to be meta-analyzed.The standardized mean difference will be the main effect size computed. Multilevel meta-analysis will be used to address the hierarchical data structure. Robust variance estimation will be used to estimate standard errors and hypothesis tests, addressing dependencies of effect sizes within studies. Prediction intervals for effect sizes, for study-average effect sizes, and heterogeneity statistics will be reported. Subgroup analyses will evaluate the relationship between moderators and the magnitude of the effect size. Cochrane’s criteria for assessing the risk of bias in studies with a separate control group will be applied. Reporting bias will be assessed visually with funnel plots.
-
more » « less
We consider the setting of an aggregate data meta-analysis of a continuous outcome of interest. When the distribution of the outcome is skewed, it is often the case that some primary studies report the sample mean and standard deviation of the outcome and other studies report the sample median along with the first and third quartiles and/or minimum and maximum values. To perform meta-analysis in this context, a number of approaches have recently been developed to impute the sample mean and standard deviation from studies reporting medians. Then, standard meta-analytic approaches with inverse-variance weighting are applied based on the (imputed) study-specific sample means and standard deviations. In this article, we illustrate how this common practice can severely underestimate the within-study standard errors, which results in poor coverage for the pooled mean in common effect meta-analyses and overestimation of between-study heterogeneity in random effects meta-analyses. We propose a straightforward bootstrap approach to estimate the standard errors of the imputed sample means. Our simulation study illustrates how the proposed approach can improve the estimation of the within-study standard errors and consequently improve coverage for the pooled mean in common effect meta-analyses and estimation of between-study heterogeneity in random effects meta-analyses. Moreover, we apply the proposed approach in a meta-analysis to identify risk factors of a severe course of COVID-19.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1371/journal.pone.0292606
