An official website of the United States government
Cluster randomized trials (CRTs) are commonly used to evaluate the causal effects of educational interventions, where the entire clusters (e.g., schools) are randomly assigned to treatment or control conditions. This study introduces statistical methods for designing and analyzing two-level (e.g., students nested within schools) and three-level (e.g., students nested within classrooms nested within schools) CRTs. Specifically, we utilize hierarchical linear models (HLMs) to account for the dependency of the intervention participants within the same clusters, estimating the average treatment effects (ATEs) of educational interventions and other effects of interest (e.g., moderator and mediator effects). We demonstrate methods and tools for sample size planning and statistical power analysis. Additionally, we discuss common challenges and potential solutions in the design and analysis phases, including the effects of omitting one level of clustering, non-compliance, threats to external validity, and cost-effectiveness of the intervention. We conclude with some practical suggestions for CRT design and analysis, along with recommendations for further readings.
more »
« less
Sample Size Planning in the Design of Two-Level Randomized Cost-Effectiveness Trials
This study introduces recent advances in statistical power analysis methods and tools for designing and analyzing randomized cost-effectiveness trials (RCETs) to evaluate the causal effects and costs of social work interventions. The article focuses on two-level designs, where, for example, students are nested within schools, with interventions applied either at the school level (cluster design) or student level (multisite design). We explore three statistical modeling strategies—random-effects, constant-effects, and fixed-effects models—to assess the cost-effectiveness of interventions, and we develop corresponding power analysis methods and tools. Power is influenced by effect size, sample sizes, and design parameters. We developed a user-friendly tool, PowerUp!-CEA, to aid researchers in planning RCETs. When designing RCETs, it is crucial to consider cost variance, its nested effects, and the covariance between effectiveness and cost data, as neglecting these factors may lead to underestimated power.
more »
« less
- Award ID(s):
- 2000705
- PAR ID:
- 10550378
- Publisher / Repository:
- SAGE Publications
- Date Published:
- Journal Name:
- Research on Social Work Practice
- Volume:
- 35
- Issue:
- 3
- ISSN:
- 1049-7315
- Format(s):
- Medium: X Size: p. 307-320
- Size(s):
- p. 307-320
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects.more » « less
-
Multilevel regression discontinuity designs have been increasingly used in education research to evaluate the effectiveness of policy and programs. It is common to ignore a level of nesting in a three-level data structure (students nested in classrooms/teachers nested in schools), whether unwittingly during data analysis or due to resource constraints during the planning phase. This study investigates the consequences of ignoring intermediate or top level in blocked three-level regression discontinuity designs (BIRD3; treatment is at level 1) during data analysis and planning. Monte Carlo simulation results indicated that ignoring a level during analysis did not affect the accuracy of treatment effect estimates; however, it affected the precision (standard errors, power, and Type I error rates). Ignoring the intermediate level did not cause a significant problem. Power rates were slightly underestimated, whereas Type I error rates were stable. In contrast, ignoring a top-level resulted in overestimated power rates; however, severe inflation in Type I error deemed this strategy ineffective. As for the design phase, when the intermediate level was ignored, it is viable to use parameters from a two-level blocked regression discontinuity model (BIRD2) to plan a BIRD3 design. However, level 2 parameters from the BIRD2 model should be substituted for level 3 parameters in the BIRD3 design. When the top level was ignored, using parameters from the BIRD2 model to plan a BIRD3 design should be avoided.more » « less
-
Abstract We evaluate the cost‐effectiveness of two early childhood interventions that use instructional coaching and parent coaching as levers for improvement. The study design allows us to compare the individual effects of each intervention as well as their combined effect on student outcomes. We find that teachers receiving instructional coaching improve their use of evidence‐based instructional practices, while families receiving parent coaching show increases in numerous responsive parenting behaviors associated with positive child outcomes. Both interventions demonstrate positive impacts on students, but only parent coaching shows statistically significant effects across a range of student outcomes. Instructional coaching alone is substantially less costly and may therefore be the most cost‐effective of the three treatment conditions; however, small sample sizes limit our ability to reach definitive conclusions. Policy simulations suggest that implementing these interventions could raise the overall cost‐effectiveness of Head Start by at least 16 percent.more » « less
-
Defining characteristics of a problem domain continues to challenge developers of visualization software, even though it is essential for designing both tools and resulting visualizations. Additionally, effectiveness of a visualization software tool often depends on the context of systems and actors within the domain problem. The nested blocks and guidelines model is a useful template for informing design and evaluation criteria for visualization software development because it aligns design to need. [1] Characterizing the outermost block of the nested model—the domain problem—is challenging, mainly due to the nature of contemporary domain problems, which are dynamic and by definition difficult to problematize. We offer here our emerging conceptual model, based on the central question in our research study—what visualization works for whom and in which situation—to characterize the outermost block, the domain problem, of the nested model. [1] We apply examples from a three-year case study of visualization software design and development to demonstrate how the conceptual model might be used to create evaluation criteria affecting design and development of a visualization tool.more » « less
