Title: Randomized Experiments in Education, with Implications for Multilevel Causal Inference
Education research has experienced a methodological renaissance over the past two decades, with a new focus on large-scale randomized experiments. This wave of experiments has made education research an even more exciting area for statisticians, unearthing many lessons and challenges in experimental design, causal inference, and statistics more broadly. Importantly, educational research and practice almost always occur in a multilevel setting, which makes the statistics relevant to other fields with this structure, including social policy, health services research, and clinical trials in medicine. In this article we first briefly review the history that led to this new era in education research and describe the design features that dominate the modern large-scale educational experiments. We then highlight some of the key statistical challenges in this area, including endogeneity of design, heterogeneity of treatment effects, noncompliance with treatment assignment, mediation, generalizability, and spillover. Though a secondary focus, we also touch on promising trial designs that answer more nuanced questions, such as the SMART design for studying dynamic treatment regimes and factorial designs for optimizing the components of an existing treatment. more »« less
McFarland, Daniel A.; Khanna, Saurabh; Domingue, Benjamin W.; Pardos, Zachary A.(
, AERA Open)
null
(Ed.)
This AERA Open special topic concerns the large emerging research area of education data science (EDS). In a narrow sense, EDS applies statistics and computational techniques to educational phenomena and questions. In a broader sense, it is an umbrella for a fleet of new computational techniques being used to identify new forms of data, measures, descriptives, predictions, and experiments in education. Not only are old research questions being analyzed in new ways but also new questions are emerging based on novel data and discoveries from EDS techniques. This overview defines the emerging field of education data science and discusses 12 articles that illustrate an AERA-angle on EDS. Our overview relates a variety of promises EDS poses for the field of education as well as the areas where EDS scholars could successfully focus going forward.
This Research Full paper focuses on perceptions and experiences of freshman and sophomore engineering students when playing an online serious engineering game that was designed to improve engineering intuition and knowledge of statics. Use of serious educational engineering games has increased in engineering education to help students increase technical competencies in engineering disciplines. However, few have investigated how these engineering games are experienced by the students; how games influence students' perceptions of learning, or how these factors may lead to inequitable perspectives among diverse populations of students. Purpose/Hypothesis: The purpose of this study was to explore the perceptions, appeal, and opinions about the efficacy of educational online games among a diverse population of students in an engineering mechanics statics course. It was hypothesized that compared to majority groups (e.g., men, White), women of color who are engineering students would experience less connections to the online educational game in terms of ease of use and level of frustration while playing. It is believed that these discordant views may negatively influence the game's appeal and efficacy towards learning engineering in this population of students. Design/Method: The Technology Acceptance Model (TAM) is expanded in this study, where the perspectives of women of colour (Latinx, Asian and African American) engineering students are explored. The research approach employed in this study is a mixed-method sequential exploratory design, where students first played the online engineering educational game, then completed a questionnaire, followed by participation in a focus group. Responses were initially analyzed through open and magnitude coding approaches to understand whether students thought these educational games reflected their personal culture. Results: Preliminary results indicate that though the majority of the students were receptive to using the online engineering software for their engineering education, merely a few intimated that they would use this software for engineering exam or technical job interview preparation. A level-one categorical analysis identified a few themes that comprised unintended preservation of inequality in favor of students who enjoyed contest-based education and game technology. Competition-based valuation of presumed mastery of course content fostered anxiety and intimidation among students, which caused some to "game the game" instead of studying the material, to meet grade goals. Some students indicated that they spent more time (than necessary) to learn the goals of the game than engineering content itself, suggesting a need to better integrate course material while minimizing cognitive effort in learning to navigate the game. Conclusions: Preliminary results indicate that engineering software's design and the way is coupled with course grading and assessment of learning outcomes, affect student perceptions of the technology's acceptance, usefulness, and ease of use as a "learning tool." Students were found to have different expectations of serious games juxtaposed software/apps designed for entertainment. Conclusions also indicate that acceptance of inquiry-based educational games in a classroom among diverse populations of students should clearly articulate and connect the game goals/objectives with class curriculum content. Findings also indicate that a multifaceted schema of tools, such as feedback on game challenges, and explanations for predictions of the game should be included in game/app designs.
Machine learning (ML) provides a powerful framework for the analysis of high‐dimensional datasets by modelling complex relationships, often encountered in modern data with many variables, cases and potentially non‐linear effects. The impact of ML methods on research and practical applications in the educational sciences is still limited, but continuously grows, as larger and more complex datasets become available through massive open online courses (MOOCs) and large‐scale investigations. The educational sciences are at a crucial pivot point, because of the anticipated impact ML methods hold for the field. To provide educational researchers with an elaborate introduction to the topic, we provide an instructional summary of the opportunities and challenges of ML for the educational sciences, show how a look at related disciplines can help learning from their experiences, and argue for a philosophical shift in model evaluation. We demonstrate how the overall quality of data analysis in educational research can benefit from these methods and show how ML can play a decisive role in the validation of empirical models. Specifically, we (1) provide an overview of the types of data suitable for ML and (2) give practical advice for the application of ML methods. In each section, we provide analytical examples and reproducible R code. Also, we provide an extensive Appendix on ML‐based applications for education. This instructional summary will help educational scientists and practitioners to prepare for the promises and threats that come with the shift towards digitisation and large‐scale assessment in education.
Context and implicationsRationale for this study
In 2020, the worldwide SARS‐COV‐2 pandemic forced the educational sciences to perform a rapid paradigm shift with classrooms going online around the world—a hardly novel but now strongly catalysed development. In the context of data‐driven education, this paper demonstrates that the widespread adoption of machine learning techniques is central for the educational sciences and shows how these methods will become crucial tools in the collection and analysis of data and in concrete educational applications. Helping to leverage the opportunities and to avoid the common pitfalls of machine learning, this paper provides educators with the theoretical, conceptual and practical essentials.
Why the new findings matter
The process of teaching and learning is complex, multifaceted and dynamic. This paper contributes a seminal resource to highlight the digitisation of the educational sciences by demonstrating how new machine learning methods can be effectively and reliably used in research, education and practical application.
Implications for educational researchers and policy makers
The progressing digitisation of societies around the globe and the impact of the SARS‐COV‐2 pandemic have highlighted the vulnerabilities and shortcomings of educational systems. These developments have shown the necessity to provide effective educational processes that can support sometimes overwhelmed teachers to digitally impart knowledge on the plan of many governments and policy makers. Educational scientists, corporate partners and stakeholders can make use of machine learning techniques to develop advanced, scalable educational processes that account for individual needs of learners and that can complement and support existing learning infrastructure. The proper use of machine learning methods can contribute essential applications to the educational sciences, such as (semi‐)automated assessments, algorithmic‐grading, personalised feedback and adaptive learning approaches. However, these promises are strongly tied to an at least basic understanding of the concepts of machine learning and a degree of data literacy, which has to become the standard in education and the educational sciences.
Demonstrating both the promises and the challenges that are inherent to the collection and the analysis of large educational data with machine learning, this paper covers the essential topics that their application requires and provides easy‐to‐follow resources and code to facilitate the process of adoption.
Somanchi, Sriram; Abbasi, Ahmed; Kelley, Ken; Dobolyi, David; Yuan, Ted Tao(
, ACM Transactions on Information Systems)
Digital experiments are routinely used to test the value of a treatment relative to a status quo control setting — for instance, a new search relevance algorithm for a website or a new results layout for a mobile app. As digital experiments have become increasingly pervasive in organizations and a wide variety of research areas, their growth has prompted a new set of challenges for experimentation platforms. One challenge is that experiments often focus on the average treatment effect (ATE) without explicitly considering differences across major sub-groups — heterogeneous treatment effect (HTE). This is especially problematic because ATEs have decreased in many organizations as the more obvious benefits have already been realized. However, questions abound regarding the pervasiveness of user HTEs and how best to detect them. We propose a framework for detecting and analyzing user HTEs in digital experiments. Our framework combines an array of user characteristics with double machine learning. Analysis of 27 real-world experiments spanning 1.76 billion sessions and simulated data demonstrates the effectiveness of our detection method relative to existing techniques. We also find that transaction, demographic, engagement, satisfaction, and lifecycle characteristics exhibit statistically significant HTEs in 10% to 20% of our real-world experiments, underscoring the importance of considering user heterogeneity when analyzing experiment results, otherwise personalized features and experiences cannot happen, thus reducing effectiveness. In terms of the number of experiments and user sessions, we are not aware of any study that has examined user HTEs at this scale. Our findings have important implications for information retrieval, user modeling, platforms, and digital experience contexts, in which online experiments are often used to evaluate the effectiveness of design artifacts.
Milani, A.S.; Cecil-Xavier; A., Gupta; Cecil, J.; Kennison, S.(
, International journal of human computer interactions)
This article provides a systematic review of research related to Human–Computer
Interaction techniques supporting training and learning in various domains including medicine,
healthcare, and engineering. The focus is on HCI techniques involving Extended Reality (XR) technology which encompasses Virtual Reality, Augmented Reality, and Mixed Reality. HCI-based
research is assuming more importance with the rapid adoption of XR tools and techniques in various
training and learning contexts including education. There are many challenges in the adoption
of HCI approaches, which results in a need to have a comprehensive and systematic review of
such HCI methods in various domains. This article addresses this need by providing a systematic
literature review of a cross-s Q1 ection of HCI approaches involving proposed so far.
The PRISMA-guided search strategy identified 1156 articles for abstract review.
Irrelevant abstracts were discarded. The whole body of each article was reviewed for the remaining
articles, and those that were not linked to the scope of our specific issue were also eliminated.
Following the application of inclusion/exclusion criteria, 69 publications were chosen for review.
This article has been divided into the following sections: Introduction; Research methodology;
Literature review; Threats of validity; Future research and Conclusion. Detailed classifications (pertaining
to HCI criteria and concepts, such as affordance; training, and learning techniques) have
also been included based on different parameters based on the analysis of research techniques
adopted by various investigators. The article concludes with a discussion of the key challenges for
this HCI area along with future research directions.
A review of the research outcomes from these publications underscores the potential for
greater success when such HCI-based approaches are adopted during such 3D-based training
interactions. Such a higher degree of success may be due to the emphasis on the design of userfriendly (and user-centric) training environments, interactions, and processes that positively impact
the cognitive abilities of users and their respective learning/training experiences.
We discovered data validating XR-HCI as an ascending method that brings a new
paradigm by enhancing skills and safety while reducing costs and learning time through replies to
three exploratory study questions. We believe that the findings of this study will aid academics in
developing new research avenues that will assist XR-HCI applications to mature and become more
widely adopted.
Raudenbush, Stephen W., and Schwartz, Daniel. Randomized Experiments in Education, with Implications for Multilevel Causal Inference. Retrieved from https://par.nsf.gov/biblio/10300972. Annual Review of Statistics and Its Application 7.1 Web. doi:10.1146/annurev-statistics-031219-041205.
Raudenbush, Stephen W., & Schwartz, Daniel. Randomized Experiments in Education, with Implications for Multilevel Causal Inference. Annual Review of Statistics and Its Application, 7 (1). Retrieved from https://par.nsf.gov/biblio/10300972. https://doi.org/10.1146/annurev-statistics-031219-041205
@article{osti_10300972,
place = {Country unknown/Code not available},
title = {Randomized Experiments in Education, with Implications for Multilevel Causal Inference},
url = {https://par.nsf.gov/biblio/10300972},
DOI = {10.1146/annurev-statistics-031219-041205},
abstractNote = {Education research has experienced a methodological renaissance over the past two decades, with a new focus on large-scale randomized experiments. This wave of experiments has made education research an even more exciting area for statisticians, unearthing many lessons and challenges in experimental design, causal inference, and statistics more broadly. Importantly, educational research and practice almost always occur in a multilevel setting, which makes the statistics relevant to other fields with this structure, including social policy, health services research, and clinical trials in medicine. In this article we first briefly review the history that led to this new era in education research and describe the design features that dominate the modern large-scale educational experiments. We then highlight some of the key statistical challenges in this area, including endogeneity of design, heterogeneity of treatment effects, noncompliance with treatment assignment, mediation, generalizability, and spillover. Though a secondary focus, we also touch on promising trial designs that answer more nuanced questions, such as the SMART design for studying dynamic treatment regimes and factorial designs for optimizing the components of an existing treatment.},
journal = {Annual Review of Statistics and Its Application},
volume = {7},
number = {1},
author = {Raudenbush, Stephen W. and Schwartz, Daniel},
editor = {null}
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
