An official website of the United States government
Abstract In ecological meta‐analyses, nonindependence among observed effect sizes from the same source paper is common. If not accounted for, nonindependence can seriously undermine inferences. We compared the performance of four meta‐analysis methods that attempt to address such nonindependence and the standard random‐effect model that ignores nonindependence. We simulated data with various types of within‐paper nonindependence, and assessed the standard deviation of the estimated mean effect size and Type I error rate of each method. Although all four methods performed substantially better than the standard random‐effects model that assumes independence, there were differences in performance among the methods. A two‐step method that first summarizes the multiple observed effect sizes per paper using a weighted mean and then analyzes the reduced data in a standard random‐effects model, and a robust variance estimation method performed consistently well. A hierarchical model with both random paper and study effects gave precise estimates but had a higher Type I error rates, possibly reflecting limitations of currently available meta‐analysis software. Overall, we advocate the use of the two‐step method with a weighted paper mean and the robust variance estimation method as reliable ways to handle within‐paper nonindependence in ecological meta‐analyses.
more »
« less
Ecological meta‐analyses often produce unwarranted results
Abstract Meta‐analysis (MA), a powerful tool for synthesizing reported results, is influential in ecology. While ecologists have long been well‐informed on the potential problems associated with nonindependence in experimental work (e.g., pseudoreplication), they have, until recently, largely neglected this issue in MA. However, results used in MAs are likely much more similar when they come from the same locality, system, or laboratory. A simple and common form of nonindependence in MA arises when multiple data points, that is, observed effect sizes, come from the same paper. We obtained original data from 20 published MAs, reconstructed the published analyses, and then, for 14 that had not accounted for a paper effect, used three approaches to evaluate whether within‐paper nonindependence was a problem. First, we found that “nonsense” explanatory variables added to the original analyses were statistically significant (p < 0.05) far more often than the expected 5% (25%–50% for four nonsense variables). For example, the number of vowels in the first author's name had a significant effect 50% of the time. Second, we found that an added dummy variable, which was randomly assigned at one of two levels, was statistically significant an average of 38% of the time, far exceeding the expected 5%. Even after including a random paper effect in the analyses, there was still an excess of significant results, suggesting that the within‐paper nonindependence was more complex than modeled with the random paper effect. Third, we repeated the original MAs that did not include random paper effects (n = 14 MAs) but added a random paper effect to each revised analysis. In 12 out of the 14 MAs, an added random effect was statistically significant (indicating group nonindependence that was not accounted for in the original analyses), and often the original inferences were substantially altered. Further, incorporating random paper effects was not a sufficient solution to nonindependence. Thus, problems resulting from nonindependence are often substantial, and accounting for the problem will likely require careful consideration of the details of the potential dependence among observed effect sizes. MAs that do not properly account for this problem may reach unwarranted conclusions.
more »
« less
- PAR ID:
- 10654125
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Ecology
- Volume:
- 106
- Issue:
- 12
- ISSN:
- 0012-9658
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
IntroductionCommunity Engaged Learning (CEL) is recognized for its positive impact on student development in higher education. This meta-analysis examined the effects of CEL on academic, personal, social, and citizenship outcomes among college students. MethodsStudies were identified through PsycINFO, PsycArticles, and ERIC, and were included if they met the following criteria: peer-reviewed English-language publications from 2017 to 2024, alignment with widely accepted definitions of CEL, inclusion of a control group, and sufficient data to calculate effect sizes. Random-effects models were used to estimate Hedges's g, a standardized measure of effect size, for each outcome domain. ResultsOur results showed that CEL had a statistically significant, small to medium effect on academic outcomes (Hedges'sg= 0.344, 95% CI [0.190, 0.497],p< 0.001) and social outcomes (Hedges'sg= 0.371, 95% CI [0.167, 0.575],p< 0.001). The effect on citizenship outcomes was small but significant (Hedges'sg= 0.220, 95% CI [0.096, 0.344],p= 0.001). For personal outcomes, the effect was moderate (Hedges'sg= 0.694, 95% CI [−0.089, 1.477]) but not statistically significant (p= 0.082). The substantial variability observed across studies suggests that differences in CEL implementation, program focus, and student populations may influence outcomes. ConclusionOverall, our findings highlight CEL as an impactful pedagogy that contributes to academic success, personal growth, and civic engagement. Further research may explore the long-term impacts of CEL and identify specific program components that enhance its effectiveness.more » « less
-
The file drawer problem—often operationalized in terms of statistically significant results being published and statistically insignificant not being published—is widely documented in the social sciences. We extend Franco’s et al. [Science345, 1502–1505(2014)] seminal study of the file drawer problem in survey experiments submitted to the Time-sharing Experiments for the Social Sciences (TESS) data collection program. We examine projects begun after Franco et al. The updated period coincides with the contemporary open science movement. We find evidence of the problem, stemming from scholars opting to not write up insignificant results. However, that tendency is substantially smaller than it was in the prior decade. This suggests increased recognition of the importance of null results, even if the problem remains in the domain of survey experiments.more » « less
-
Cyber-physical systems (CPS) designed in simulators, often consisting of multiple interacting agents (e.g., in multi-agent formations), behave differently in the real-world. We would like to verify these systems during runtime when they are deployed. Thus, we propose robust predictive runtime verification (RPRV) algorithms for: (1) general stochastic CPS under signal temporal logic (STL) tasks, and (2) stochastic multi-agent systems (MAS) under spatio-temporal logic tasks. The RPRV problem presents the following challenges: (1) there may not be sufficient data on the behavior of the deployed CPS, (2) predictive models based on design phase system trajectories may encounter distribution shift during real-world deployment, and (3) the algorithms need to scale to the complexity of MAS and be applicable to spatio-temporal logic tasks. To address these challenges, we assume knowledge of an upper bound on the statistical distance (in terms of anf-divergence) between the trajectory distributions of the system at deployment and design time. We are motivated by our prior work where we proposed an accurate and an interpretable RPRV algorithm for general CPS, which we here extend to the MAS setting and spatio-temporal logic tasks. Specifically, we use a learned predictive model to estimate the system behavior at runtime androbust conformal predictionto obtain probabilistic guarantees by accounting for distribution shifts. Building on our prior work, we perform robust conformal prediction over the robust semantics of spatio-temporal reach and escape logic (STREL) to obtain centralized RPRV algorithms for MAS. We empirically validate our results in a drone swarm simulator, where we show the scalability of our RPRV algorithms to MAS and analyze the impact of different trajectory predictors on the verification result. To the best of our knowledge, these are the first statistically valid algorithms for MAS under distribution shift.more » « less
-
Team building activities are popular interventions during early stages of team development. At RIT, in the multidisciplinary capstone course with an average cohort size of around 350, the students on a particular capstone project team may not be mutually acquainted and thus may benefit from such team building activities. Prior literature has studied the effectiveness of various instructor-directed team building activities on student teams. However, our students are generally eager to spend class time working on their projects and often see in-class activities as a distraction rather than an important part of their growth. Instead, the student teams are now allowed to choose an intervention based on team consensus. In this paper, the relationship between attributes of the chosen intervention and student performance, as measured using a series of AACU VALUE rubrics, was studied using statistical measures. The analysis revealed a statistically significant effect of type of team building activity on teamwork, oral communication, and design & problem solving scores of individual students on the team. Also, a statistically significant effect of location of team building activity (on or off campus) on design & problem solving score was observed.more » « less
