An official website of the United States government
The file drawer problem—often operationalized in terms of statistically significant results being published and statistically insignificant not being published—is widely documented in the social sciences. We extend Franco’s et al. [Science345, 1502–1505(2014)] seminal study of the file drawer problem in survey experiments submitted to the Time-sharing Experiments for the Social Sciences (TESS) data collection program. We examine projects begun after Franco et al. The updated period coincides with the contemporary open science movement. We find evidence of the problem, stemming from scholars opting to not write up insignificant results. However, that tendency is substantially smaller than it was in the prior decade. This suggests increased recognition of the importance of null results, even if the problem remains in the domain of survey experiments.
more »
« less
Ecological meta‐analyses often produce unwarranted results
Abstract Meta‐analysis (MA), a powerful tool for synthesizing reported results, is influential in ecology. While ecologists have long been well‐informed on the potential problems associated with nonindependence in experimental work (e.g., pseudoreplication), they have, until recently, largely neglected this issue in MA. However, results used in MAs are likely much more similar when they come from the same locality, system, or laboratory. A simple and common form of nonindependence in MA arises when multiple data points, that is, observed effect sizes, come from the same paper. We obtained original data from 20 published MAs, reconstructed the published analyses, and then, for 14 that had not accounted for a paper effect, used three approaches to evaluate whether within‐paper nonindependence was a problem. First, we found that “nonsense” explanatory variables added to the original analyses were statistically significant (p < 0.05) far more often than the expected 5% (25%–50% for four nonsense variables). For example, the number of vowels in the first author's name had a significant effect 50% of the time. Second, we found that an added dummy variable, which was randomly assigned at one of two levels, was statistically significant an average of 38% of the time, far exceeding the expected 5%. Even after including a random paper effect in the analyses, there was still an excess of significant results, suggesting that the within‐paper nonindependence was more complex than modeled with the random paper effect. Third, we repeated the original MAs that did not include random paper effects (n = 14 MAs) but added a random paper effect to each revised analysis. In 12 out of the 14 MAs, an added random effect was statistically significant (indicating group nonindependence that was not accounted for in the original analyses), and often the original inferences were substantially altered. Further, incorporating random paper effects was not a sufficient solution to nonindependence. Thus, problems resulting from nonindependence are often substantial, and accounting for the problem will likely require careful consideration of the details of the potential dependence among observed effect sizes. MAs that do not properly account for this problem may reach unwarranted conclusions.
more »
« less
- PAR ID:
- 10654529
- Publisher / Repository:
- Ecological Society of America
- Date Published:
- Journal Name:
- Ecology
- Volume:
- 106
- Issue:
- 12
- ISSN:
- 0012-9658
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
IntroductionCommunity Engaged Learning (CEL) is recognized for its positive impact on student development in higher education. This meta-analysis examined the effects of CEL on academic, personal, social, and citizenship outcomes among college students. MethodsStudies were identified through PsycINFO, PsycArticles, and ERIC, and were included if they met the following criteria: peer-reviewed English-language publications from 2017 to 2024, alignment with widely accepted definitions of CEL, inclusion of a control group, and sufficient data to calculate effect sizes. Random-effects models were used to estimate Hedges's g, a standardized measure of effect size, for each outcome domain. ResultsOur results showed that CEL had a statistically significant, small to medium effect on academic outcomes (Hedges'sg= 0.344, 95% CI [0.190, 0.497],p< 0.001) and social outcomes (Hedges'sg= 0.371, 95% CI [0.167, 0.575],p< 0.001). The effect on citizenship outcomes was small but significant (Hedges'sg= 0.220, 95% CI [0.096, 0.344],p= 0.001). For personal outcomes, the effect was moderate (Hedges'sg= 0.694, 95% CI [−0.089, 1.477]) but not statistically significant (p= 0.082). The substantial variability observed across studies suggests that differences in CEL implementation, program focus, and student populations may influence outcomes. ConclusionOverall, our findings highlight CEL as an impactful pedagogy that contributes to academic success, personal growth, and civic engagement. Further research may explore the long-term impacts of CEL and identify specific program components that enhance its effectiveness.more » « less
-
We provide statistical measures and additional analyses showing that our original analyses were sound. We use a generalized linear mixed model to account for program-to-program differences with program as a random effect without stratifying with tier and found the GRE-P (Graduate Record Examination physics test) effect is not different from our previous findings, thereby alleviating concern of collider bias. Variance inflation factors for each variable were low, showing that multicollinearity was not a concern. We show that range restriction is not an issue for GRE-P or GRE-V (GRE verbal), and only a minor issue for GRE-Q (GRE quantitative). Last, we use statistical measures of model quality to show that our published models are better than or equivalent to several alternates.more » « less
-
Background: When unaddressed, contamination in child maltreatment research, in which some proportion of children recruited for a nonmaltreated comparison group are exposed to maltreatment, downwardly biases the significance and magnitude of effect size estimates. This study extends previous contamination research by investigating how a dual‐measurement strategy of detecting and controlling contamination impacts causal effect size estimates of child behavior problems. Methods: This study included 634 children from the LONGSCAN study with 63 cases of confirmed child maltreatment after age 8 and 571 cases without confirmed child maltreatment. Confirmed child maltreatment and internalizing and externalizing behaviors were recorded every 2 years between ages 4 and 16. Contamination in the nonmaltreated comparison group was identified and controlled by either a prospective self‐report assessment at ages 12, 14, and 16 or by a one‐time retrospective self‐report assessment at age 18. Synthetic control methods were used to establish causal effects and quantify the impact of contamination when it was not controlled, when it was controlled for by prospective self‐reports, and when it was controlled for by retrospective self‐reports. Results: Rates of contamination ranged from 62% to 67%. Without controlling for contamination, causal effect size estimates for internalizing behaviors were not statistically significant. Causal effects only became statistically significant after controlling contamination identified from either prospective or retrospective reports and effect sizes increased by between 17% and 54%. Controlling contamination had a smaller impact on effect size increases for externalizing behaviors but did produce a statistically significant overall effect, relative to the model ignoring contamination, when prospective methods were used. Conclusions: The presence of contamination in a nonmaltreated comparison group can underestimate the magnitude and statistical significance of causal effect size estimates, especially when investigating internalizing behavior problems. Addressing contamination can facilitate the replication of results across studies.more » « less
-
We present new Very Large Telescope Interferometer (VLTI)/GRAVITY near-infrared interferometric measurements of the angular size of the innermost hot dust continuum for 14 type 1 active galactic nuclei (AGNs). The angular sizes are resolved on scales of ∼0.7 mas and the inferred ring radii range from 0.028 to 1.33 pc, comparable to those reported previously and a factor of 10−20 smaller than the mid-infrared sizes in the literature. Combining our new data with previously published values, we compiled a sample of 25 AGNs with bolometric luminosity ranging from 1042to 1047erg s−1, with which we studied the radius-luminosity (R − L) relation for the hot dust structure. Our interferometric measurements of radius are offset by a factor of 2 from the equivalent relation derived through reverberation mapping. Using a simple model to explore the dust structure’s geometry, we conclude that this offset can be explained if the 2 μm emitting surface has a concave shape. Our data show that the slope of the relation is in line with the canonicalR ∝ L0.5when using an appropriately non-linear correction for bolometric luminosity. In contrast, using optical luminosity or applying a constant bolometric correction to it results in a significant deviation in the slope, suggesting a potential luminosity dependence on the spectral energy distribution. Over four orders of magnitude in luminosity, the intrinsic scatter around theR − Lrelation is 0.2 dex, suggesting a tight correlation between the innermost hot dust structure size and the AGN luminosity.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1002/ecy.70269
