An official website of the United States government
Researchers often apply moderation analyses to examine whether the effects of an intervention differ conditional on individual or cluster moderator variables such as gender, pretest, or school size. This study develops formulas for power analyses to detect moderator effects in two-level cluster randomized trials (CRTs) using linear models. We derive the formulas for estimating statistical power, minimum detectable effect size difference and 95% confidence intervals for cluster- and individual-level moderators. Our framework accommodates binary or continuous moderators, designs with or without covariates, and effects of individual-level moderators that vary randomly or nonrandomly across clusters. A small Monte Carlo simulation confirms the accuracy of our formulas. We also compare power between main effect analysis and moderation analysis, discuss the effects of mis-specification of the moderator slope (randomly vs. non-randomly varying), and conclude with directions for future research. We provide software for conducting a power analysis of moderator effects in CRTs.
more »
« less
Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects
Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects.
more »
« less
- Award ID(s):
- 1913563
- PAR ID:
- 10546802
- Publisher / Repository:
- DOI PREFIX: 10.3102
- Date Published:
- Journal Name:
- Journal of Educational and Behavioral Statistics
- Volume:
- 46
- Issue:
- 5
- ISSN:
- 1076-9986
- Format(s):
- Medium: X Size: p. 527-559
- Size(s):
- p. 527-559
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Cluster randomized trials (CRTs) are commonly used to evaluate the causal effects of educational interventions, where the entire clusters (e.g., schools) are randomly assigned to treatment or control conditions. This study introduces statistical methods for designing and analyzing two-level (e.g., students nested within schools) and three-level (e.g., students nested within classrooms nested within schools) CRTs. Specifically, we utilize hierarchical linear models (HLMs) to account for the dependency of the intervention participants within the same clusters, estimating the average treatment effects (ATEs) of educational interventions and other effects of interest (e.g., moderator and mediator effects). We demonstrate methods and tools for sample size planning and statistical power analysis. Additionally, we discuss common challenges and potential solutions in the design and analysis phases, including the effects of omitting one level of clustering, non-compliance, threats to external validity, and cost-effectiveness of the intervention. We conclude with some practical suggestions for CRT design and analysis, along with recommendations for further readings.more » « less
-
Identifying treatment effect modifiers (i.e., moderators) plays an essential role in improving treatment efficacy when substantial treatment heterogeneity exists. However, studies are often underpowered for detecting treatment effect modifiers, and exploratory analyses that examine one moderator per statistical model often yield spurious interactions. Therefore, in this work, we focus on creating an intuitive and readily implementable framework to facilitate the discovery of treatment effect modifiers and to make treatment recommendations for time-to-event outcomes. To minimize the impact of a misspecified main effect and avoid complex modeling, we construct the framework by matching the treated with the controls and modeling the conditional average treatment effect via regressing the difference in the observed outcomes of a matched pair on the averaged moderators. Inverse-probability-of-censoring weighting is used to handle censored observations. As matching is the foundation of the proposed methods, we explore different matching metrics and recommend the use of Mahalanobis distance when both continuous and categorical moderators are present. After matching, the proposed framework can be flexibly combined with popular variable selection and prediction methods such as linear regression, least absolute shrinkage and selection operator (Lasso), and random forest to create different combinations of potential moderators. The optimal combination is determined by the out-of-bag prediction error and the area under the receiver operating characteristic curve in making correct treatment recommendations. We compare the performance of various combined moderators through extensive simulations and the analysis of real trial data. Our approach can be easily implemented using existing R packages, resulting in a straightforward optimal combined moderator to make treatment recommendations.more » « less
-
Richard E. Mayer has made major contributions to Educational Psychology since the 1970s, including work on learning in mathematics, creativity, interest, measurement, problem solving, and especially multimedia learning, defined as learning from instructional material that includes information in both verbal and visual form. In a 2024 reflection, Mayer called for identifying boundary conditions—i.e., moderators of effects—of his multimedia design principles. In an effort to identify these, we meta-analyzed Mayer’s corpus of multimedia research. We searched Google Scholar, PsycINFO, and the Cambridge Handbook of Multimedia Learning 3rd Ed. for peer-reviewed articles on multimedia learning with Mayer as an author published 1990-2022 and located 92 articles reporting on 181 studies reporting on 591 separate effects. We coded for 9 moderators: multimedia design principle, multimedia type, age, academic domain, country/continent, treatment duration, dependent variable type, year, and authorship order. We analyzed the Hedge’s g effect sizes using a multilevel regression approach in the metafor package in R. The overall effect was g = 0.37, which was significantly moderated by all moderators, including a small decline in effect size per year. Mean effects by multimedia design principle were uneven, with the largest significant effects for removing seductive detail, modality principle, personalization, multimedia principle, sentence-level coherence, and self-explanation. Medium significant overall effects were found for the testing effect, scaffolding, cueing, and embodiment. Large, consistent effects were found for text + diagrams across factual, inferential, and transfer outcomes. Less-consistent effects were found for animation, games, and simulations, with smaller effects on factual learning and on average larger effects on inferential and transfer outcomes, but no significant effects for virtual reality. We identified two boundary conditions in tests of design principle x DV type interactions and Multimedia type x DV type interactions. We close by interpreting various findings in phases of Mayer’s work, characterized by collaborators and educational technologies. We also contextualize Mayer’s findings within recent meta-analyses of the larger published research on various design principles.more » « less
-
null (Ed.)Content moderation is a critical service performed by a variety of people on social media, protecting users from offensive or harmful content by reviewing and removing either the content or the perpetrator. These moderators fall into one of two categories: employees or volunteers. Prior research has suggested that there are differences in the effectiveness of these two types of moderators, with the more transparent user-based moderation being useful for educating users. However, direct comparisons between commercially-moderated and user-moderated platforms are rare, and apart from the difference in transparency, we still know little about what other disparities in user experience these two moderator types may create. To explore this, we conducted cross-platform surveys of over 900 users of commercially-moderated (Facebook, Instagram, Twitter, and YouTube) and user-moderated (Reddit and Twitch) social media platforms. Our results indicated that although user-moderated platforms did seem to be more transparent than commercially-moderated ones, this did not lead to user-moderated platforms being perceived as less toxic. In addition, commercially-moderated platform users want companies to take more responsibility for content moderation than they currently do, while user-moderated platform users want designated moderators and those who post on the site to take more responsibility. Across platforms, users seem to feel powerless and want to be taken care of when it comes to content moderation as opposed to engaging themselves.more » « less
