An official website of the United States government
Student perceptions of programming can impact their experiences in introductory computer science (CS) courses. For example, some students negatively assess their own ability in response to moments that are natural parts of expert practice, such as using online resources or getting syntax errors. Systems that automatically detect these moments from interaction log data could help us study these moments and intervene when the occur. However, while researchers have analyzed programming log data, few systems detect pre-defined moments, particularly those based on student perceptions. We contribute a new approach and system for detecting programming moments that students perceive as important from interaction log data. We conducted retrospective interviews with 41 CS students in which they identified moments that can prompt negative self-assessments. Then we created a qualitative codebook of the behavioral patterns indicative of each moment, and used this knowledge to build an expert system. We evaluated our system with log data collected from an additional 33 CS students. Our results are promising, with F1 scores ranging from 66% to 98%. We believe that this approach can be applied in many domains to understand and detect student perceptions of learning experiences.
more »
« less
Exploring Novices' Struggle and Progress During Programming Through Data-Driven Detectors and Think-Aloud Protocols
Many students struggle when they are first learning to program. Without help, these students can lose confidence and negatively assess their programming ability, which can ultimately lead to dropouts. However, detecting the exact moment of student struggle is still an open question in computing education. In this work, we conducted a think-aloud study with five high-school students to investigate the automatic detection of progressing and struggling moments using a detector algorithm (SPD). SPD classifies student trace logs into moments of struggle and progress based on their similarity to prior students' correct solutions. We explored the extent to which the SPD-identified moments of struggle aligned with expert-identified moments based on novices' verbalized thoughts and programming actions. Our analysis results suggest that SPD can catch students' struggling and progressing moments with a 72.5% F1-score, but room remains for improvement in detecting struggle. Moreover, we conducted an in-depth examination to discover why discrepancies arose between expert-identified and detector-identified struggle moments. We conclude with recommendations for future data-driven struggle detection systems.
more »
« less
- Award ID(s):
- 1917885
- PAR ID:
- 10528599
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3503-2946-9
- Page Range / eLocation ID:
- 179 -183
- Subject(s) / Keyword(s):
- novice programming computer science education struggle detection progress detection
- Format(s):
- Medium: X
- Location:
- Washington, DC, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Background and Context. The importance of CS to 21st-century life and work has made it important to find ways to integrate learning CS and programming into the regular school day. However, learning CS is difficult, so teachers integrating programming need effective strategies to scaffold the learning. In this study, we analyze students’ log data and apply a novel technique to compare Parsons Problems with from-scratch programming in a middle school science class. Objectives. Our research questions aimed to investigate whether, how, and when Parsons Problems improve learning efficiency for a programming exercise within science, utilizing log data analysis and an automated progress detector (SPD). Method. We conducted a study on 199 students in a 6th-grade science course, divided into two groups: one engaged with Parsons problems, and the other, a control group, worked on the same programming task without scaffolding. Then, we analyzed differences in performance and coding characteristics between the groups. We also adopted an innovative application of SPD to gain a better understanding of how and when Parsons problems helped students make more progress on the coding task, with an objective measure of final student grades. Findings. The experimental group, with scaffolding through Parsons Problems, achieved significantly higher grades, spent significantly less time programming, and toggled less between block category tabs. Interestingly, they ran their code more frequently compared to the control group. The SPD analysis revealed that the experimental group made significantly higher progress in all four quartiles of their coding time. Implications. Our findings suggest that Parsons problems can improve learning efficiency by enhancing novices’ learning experience without negatively impacting their performance or grades, which is especially important when programming is integrated into K12 courses.more » « less
-
Undergraduate programs in computer science (CS) face high dropout rates, and many students struggle while learning to program. Studies show that perceived programming ability is a significant factor in students' decision to major in CS. Fortunately, psychology research shows that promoting the growth mindset, or the belief that intelligence grows with effort, can improve student persistence and performance. However, mindset interventions have been less successful in CS than in other domains. We conducted a small-scale interview study to explore how CS students talk about their intelligence, mindsets, and programming behaviors. We found that students' mindsets rarely aligned with definitions in the literature; some present mindsets that combine fixed and growth attributes, while others behave in ways that do not align with their mindsets. We also found that students frequently evaluate their self-efficacy by appraising their programming intelligence, using surprising criteria like typing speed and ease of debugging to measure ability. We conducted a survey study with 103 students to explore these self-assessment criteria further, and found that students use varying and conflicting criteria to evaluate intelligence in CS. We believe the criteria that students choose may interact with mindsets and impact their motivation and approach to programming, which could help explain the limited success of mindset interventions in CS.more » « less
-
Many undergraduate students encounter struggle as they navigate academic, financial, and social contexts of higher education. The transition to emergency online instruction during the Spring of 2020 due to the COVID-19 pandemic exacerbated these struggles. To assess college students’ struggles during the transition to online learning in undergraduate biology courses, we surveyed a diverse collection of students ( n = 238) at an R2 research institution in the Southeastern United States. Students were asked if they encountered struggles and whether they were able to overcome them. Based on how students responded, they were asked to elaborate on (1) how they persevered without struggle, (2) how they were able to overcome their struggles, or (3) what barriers they encountered that did not allow them to overcome their struggles. Each open-ended response was thematically coded to address salient patterns in students’ ability to either persevere or overcome their struggle. We found that during the transition to remote learning, 67% of students experienced struggle. The most reported struggles included: shifts in class format, effective study habits, time management, and increased external commitments. Approximately, 83% of those struggling students were able to overcome their struggle, most often citing their instructor’s support and resources offered during the transition as reasons for their success. Students also cited changes in study habits, and increased confidence or belief that they could excel within the course as ways in which they overcame their struggles. Overall, we found no link between struggles in the classroom and any demographic variables we measured, which included race/ethnicity, gender expression, first-generation college students, transfer student status, and commuter student status. Our results highlight the critical role that instructors play in supporting student learning during these uncertain times by promoting student self-efficacy and positive-growth mindset, providing students with the resources they need to succeed, and creating a supportive and transparent learning environment.more » « less
-
The ability to predict student performance in introductory programming courses is important to help struggling students and enhance their persistence. However, for this prediction to be impactful, it is crucial that it remains transparent and accessible for both instructors and students, ensuring effective utilization of the predicted results. Machine learning models with explainable features provide an effective means for students and instructors to comprehend students' diverse programming behaviors and problem-solving strategies, elucidating the factors contributing to both successful and suboptimal performance. This study develops an explainable model that predicts student performance based on programming assignment submission information in different stages of the course to enable early explainable predictions. We extract data-driven features from student programming submissions and utilize a stacked ensemble model for predicting final exam grades. The experimental results suggest that our model successfully predicts student performance based on their programming submissions earlier in the semester. Employing SHAP, a game-theory-based framework, we explain the model's predictions, aiding stakeholders in understanding the influence of diverse programming behaviors on students' success. Additionally, we analyze crucial features, employing a mix of descriptive statistics and mixture models to identify distinct student profiles based on their problem-solving patterns, enhancing overall explainability. Furthermore, we dive deeper and analyze the profiles using different programming patterns of the students to elucidate the characteristics of different students where SHAP explanations are not comprehensible. Our explainable early prediction model elucidates common problem-solving patterns in students relative to their expertise, facilitating effective intervention and adaptive support.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/VL-HCC57772.2023.00029
