An official website of the United States government
Abstract: First-year students who enter college pursuing a STEM degree still face challenges persisting through the STEM pipeline (Chen, 2013; Leu, 2017). In this case study, esearchers examine the impact of a utilitarian scientific literacy based academic intervention on retention of first-year students in STEM using a mixed methods approach. A sample (n = 116) of first-year students identified as at-risk of not persisting in STEM were enrolled in a for credit utilitarian scientific literacy course. Participants of the semester long course were then compared with a control group of first-year students identified as at-risk of persisting in STEM. A two-proportion z test was performed to assess the mean differences between students and participants of the course were given a survey to gauge student experiences. Quantitative results (ϕ 0.34, p < 0.05) indicate that the utilitarian scientific literacy course had a statistically significant impact on retention among first-year students at-risk of persisting in STEM. Moreover, qualitative data obtained from participant responses describe internal and external growth as positive outcomes associated with the intervention.
more »
« less
Random Forests for Evaluating Pedagogy and Informing Personalized Learning
Random forests are presented as an analytics foundation for educational data mining tasks. The focus is on course- and program-level analytics including evaluating pedagogical approaches and interventions and identifying and characterizing at-risk students. As part of this development, the concept of individualized treatment effects (ITE) is introduced as a method to provide personalized feedback to students. The ITE quantifies the effectiveness of intervention and/or instructional regimes for a particular student based on institutional student information and performance data. The proposed random forest framework and methods are illustrated on a study of the efficacy of a supplemental, weekly, one-unit problem-solving session in a large enrollment, bottleneck introductory statistics course. The analytics tools are used to identify factors for student success, characterize students benefitting from the supplemental instruction section, develop an objective criterion to, at the beginning of the semester, identify and advise these students into that section, and suggest intervention initiatives for at-risk groups in the course.
more »
« less
- Award ID(s):
- 1633130
- PAR ID:
- 10096605
- Date Published:
- Journal Name:
- Journal of educational data mining
- Volume:
- 8
- Issue:
- 2
- ISSN:
- 2157-2100
- Page Range / eLocation ID:
- 20-50
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Understanding the thought processes of students as they progress from initial (incorrect) answers toward correct answers is a challenge for instructors, both in this pandemic and beyond. This paper presents a general network visualization learning analytics system that helps instructors to view a sequence of answers input by students in a way that makes student learning progressions apparent. The system allows instructors to study individual and group learning at various levels of granularity. The paper illustrates how the visualization system is employed to analyze student responses collected through an intervention. The intervention is BeginToReason, an online tool that helps students learn and use symbolic reasoning-reasoning about code behavior through abstract values instead of concrete inputs. The specific focus is analysis of tool-collected student responses as they perform reasoning activities on code involving conditional statements. Student learning is analyzed using the visualization system and a post-test. Visual analytics highlights include instances where students producing one set of incorrect answers initially perform better than a different set and instances where student thought processes do not cluster well. Post-test data analysis provides a measure of student ability to apply what they have learned and their holistic understanding.more » « less
-
Engineering requires more bachelor’s degree graduates to meet the growing demand for engineering skills globally. One way to address this demand is increasing student degree completion, which is lower than higher education in general. In particular, Black, Latino/a/x, and Indigenous (BLI) students are less likely to complete an engineering degree than their peers. BLI students experience a host of unwelcoming behaviors in engineering environments that contribute to departure without their intended degree. Improving environments to support belonging may offer one solution. Through an ecological belonging intervention, we seek to improve continued enrollment and increase belonging. Quasi-experimental methods were used in a second-semester engineering programming course. Surveys collected before and after an intervention combined with institutional data were used to test the moderation effects of the intervention on continued enrollment in engineering during the semester following the intervention. BLI students who were enrolled in intervention treatment sections were more likely to be enrolled in engineering the following fall. The intervention treatment increased belonging such that control section participants were less likely to continue to be enrolled in engineering. While research to assess the efficacy and mechanisms of the intervention is ongoing, the intervention offers promising results to address attrition, particularly for BLI students.more » « less
-
Because the average treatment effect (ATE) measures the change in social welfare, even if positive, there is a risk of negative effect on, say, some 10% of the population. Assessing such risk is difficult, however, because any one individual treatment effect (ITE) is never observed, so the 10% worst-affected cannot be identified, whereas distributional treatment effects only compare the first deciles within each treatment group, which does not correspond to any 10% subpopulation. In this paper, we consider how to nonetheless assess this important risk measure, formalized as the conditional value at risk (CVaR) of the ITE distribution. We leverage the availability of pretreatment covariates and characterize the tightest possible upper and lower bounds on ITE-CVaR given by the covariate-conditional average treatment effect (CATE) function. We then proceed to study how to estimate these bounds efficiently from data and construct confidence intervals. This is challenging even in randomized experiments as it requires understanding the distribution of the unknown CATE function, which can be very complex if we use rich covariates to best control for heterogeneity. We develop a debiasing method that overcomes this and prove it enjoys favorable statistical properties even when CATE and other nuisances are estimated by black box machine learning or even inconsistently. Studying a hypothetical change to French job search counseling services, our bounds and inference demonstrate a small social benefit entails a negative impact on a substantial subpopulation. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the Division of Information and Intelligent Systems [Grant 1939704]. Supplemental Material: The data files and online appendices are available at https://doi.org/10.1287/mnsc.2023.4819 .more » « less
-
Mechanics instructors frequently employ hands-on learning with goals such as demonstrating physical phenomena, aiding visualization, addressing misconceptions, exposing students to “real-world” problems, and promoting an engaging classroom environment. This paper presents results from a study exploring the importance of the “hands-on” aspect of a hands-on modeling curriculum we have been developing that spans several topics in statics. The curriculum integrates deep conceptual exploration with analysis procedure tutorials and aims to scaffold students’ development of representational competence, the ability to use multiple representations of a concept as appropriate for learning, problem solving, and communication. We conducted this study over two subsequent terms in an online statics course taught in the context of remote learning amidst the COVID-19 pandemic. The intervention section used a take-home adaptation of the original classroom curriculum. This adaptation consisted of eight activity worksheets with a supplied kit of manipulatives and model-building supplies students could use to construct and explore concrete representations of figures and diagrams used in the worksheets. In contrast, the control section used activity worksheets nearly identical to those used in the hands-on curriculum, but without the associated modeling parts kit. We only made minor revisions to the worksheets to remove reference to the models. The control and intervention sections were otherwise identical in how they were taught by the same instructor. We compare learning outcomes between the two sections as measured via pre-post administration of a test of 3D vector concepts and representations called the Test of Representational Competence with Vectors (TRCV). We also compare end of course scores on the Concept Assessment Test in Statics (CATS) and final exam scores. In addition, we analyze student responses on two “multiple choice plus explain” concept questions paired with each of five activities covering the topics of 3D moments, 3D particle equilibrium, rigid body equilibrium (2D and 3D), and frame analysis (2D). The mean pre/post gain across all ten questions was higher for the intervention section, with the largest differences observed on questions relating to 3D rigid body equilibrium. Students in the intervention section also made larger gains on the TRCV and scored better on the final exam compared to the control section, but these results are not statistically significant perhaps due to the small study population. There were no appreciable differences in end-of-course CATS scores. We also present student feedback on the activity worksheets that was slightly more positive for the versions with the models.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
