Search for: All records

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. 

Self-explanations could increase student’s comprehension in complex domains; however, it works most efficiently with a human tutor who could provide corrections and scaffolding. In this paper, we present our attempt to scale up the use of self-explanations in learning program- ming by delegating assessment and scaffolding of explanations to an intel- ligent tutor. To assess our approach, we performed a randomized control trial experiment that measured the impact of automatic assessment and scaffolding of self-explanations on code comprehension and learning. The study results indicate that low-prior knowledge students in the experi- mental condition learn more compared to high-prior knowledge in the same condition but such difference is not observed in a similar grouping of students based on prior knowledge in the control condition. 

The ability to automatically assess learners' activities is the key to user modeling and personalization in adaptive educational systems.The work presented in this paper opens an opportunity to expand the scope of automated assessment from traditional programming problems to code comprehension tasks where students are requested to explain the critical steps of a program. The ability to automatically assess these self-explanations offers a unique opportunity to understand the current state of student knowledge, recognize possible misconceptions, and provide feedback. Annotated datasets are needed to train Artificial Intelligence/Machine Learning approaches for the automated assessment of student explanations. To answer this need, we present a novel corpus called SelfCode which consists of 1,770 sentence pairs of student and expert self-explanations of Java code examples, along with semantic similarity judgments provided by experts. We also present a baseline automated assessment model that relies on textual features. The corpus is available at the GitHub repository (https://github.com/jeevanchaps/SelfCode). 

Self-efficacy, or the belief in one's ability to accomplish a task or achieve a goal, can significantly influence the effectiveness of various instructional methods to induce learning gains. The importance of self-efficacy is particularly pronounced in complex subjects like Computer Science, where students with high self-efficacy are more likely to feel confident in their ability to learn and succeed. Conversely, those with low self-efficacy may become discouraged and consider abandoning the field. The work presented here examines the relationship between self-efficacy and students learning computer programming concepts. For this purpose, we conducted a randomized control trial experiment with university-level students who were randomly assigned into two groups: a control group where participants read Java programs accompanied by explanatory texts (a passive strategy) and an experimental group where participants self-explain while interacting through dialogue with an intelligent 

This work presents two systems, Machine Noun Question Generation (QG) and Machine Verb QG, developed to generate short questions and gap-fill questions, which Intelligent Tutoring Systems then use to guide students’ self-explanations during code comprehension. We evaluate our system by comparing the quality of questions generated by the system against human expert-generated questions. Our result shows that these systems performed similarly to humans in most criteria. Among the machines, we find that Machine Noun QG performed better. 

Domain modeling is a central component in education technologies as it represents the target domain students are supposed to train on and eventually master. Automatically generating domain models can lead to substantial cost and scalability benefits. Automatically extracting key concepts or knowledge components from, for instance, textbooks can enable the development of automatic or semi-automatic processes for creating domain models. We explore in this work the use of transformer based pre-trained models for the task of keyphrase extraction. Specifically, we investigate and evaluate four different variants of BERT, a pre-trained transformer based architecture, that vary in terms of training data, training objective, or training strategy to extract knowledge components from textbooks for the domain of intro-to-programming. We report results obtained using the following BERT-based models: BERT, CodeBERT, SciBERT and RoBERTa. 

This paper presents a novel method to automatically assess self-explanations generated by students during code comprehension activities. The self-explanations are produced in the context of an online learning environment that asks students to freely explain Java code examples line-by-line. We explored a number of models consisting of textual features in conjunction with machine learning algorithms such as Support Vector Regression (SVR), Decision Trees (DT), and Random Forests (RF). Support Vector Regression (SVR) performed best having a correlation score with human judgments of 0.7088. The best model used a combination of features such as semantic measures obtained using a Sentence BERT pre-trained model and from previously developed semantic algorithms used in a state-of-the-art intelligent tutoring system. 

This experience report describes the delivery of round-the-clock help to students using Discord (a popular messaging and voice/video calling platform) in a remote software engineering course. Students in the course learn full-stack web development using Ruby on Rails and PostgreSQL, and work in teams to develop web applications. Our central goal in offering round-the-clock help using Discord was to increase the amount of help that students receive from teachers (i.e., teaching assistants and the instructor). Indeed, we found that our 24/7-Discord approach led to a considerable increase in the amount of student-teacher interaction versus the approach used previously, which emphasized in-person office hours and a question-and-answer forum in Piazza. Moreover, students from underrepresented groups in computer science interacted with teachers at a rate comparable to other students, and we received consistently positive feedback from students regarding the approach. We also made several key observations about when students tended to seek help, including that they sought help the most between 7:00 p.m. and midnight, that help seeking spiked right before deadlines, that students posted the fewest help messages on weekends, and that students posted significantly more messages during the first half of the course, which emphasized skills assignments, versus the second half, which focused on team project work. 

This experience report describes the use of frequent, timed coding tests in a project-intensive software engineering course in which students first learn full-stack web development using Ruby on Rails and then apply their skills in a team project. The goal of the skills tests was twofold: (1) to help motivate students to engage in distributed practice and, thus, gain adequate coding skills to be an effective team member during the team project and (2) to accurately assess whether students had acquired the requisite skills and, thereby, catch deficiencies early, while there was still time to address them. Regarding the first goal, although several students indicated that the tests motivated them to engage in substantial practice coding, it was ultimately inconclusive as to the extent of the tests' impact on students' distributed practice behavior and on their preparation for the project. Regarding the second goal, the skills testing approach was indeed considerably more effective than graded homework assignments for assessing coding skill and detecting struggling students early. Lessons learned from our experiences included that students had significant concerns about the strict time limit on the tests, that the tests caused a spike in mid-semester withdrawals from the course that disproportionately impacted students from underrepresented groups, and that detecting struggling students was one thing, but effectively helping them catch up was a whole other challenge. 

We present in this paper the results of a randomized control trial experiment that compared the effectiveness of two instructional strategies that scaffold learners' code comprehension processes: eliciting Free Self-Explanation and a Socratic Method. Code comprehension, i.e., understanding source code, is a critical skill for both learners and professionals. Improving learners' code comprehension skills should result in improved learning which in turn should help with retention in intro-to-programming courses which are notorious for suffering from very high attrition rates due to the complexity of programming topics. To this end, the reported experiment is meant to explore the effectiveness of various strategies to elicit self-explanation as a way to improve comprehension and learning during complex code comprehension and learning activities in intro-to-programming courses. The experiment showed pre-/post-test learning gains of 30% (M = 0.30, SD = 0.47) for the Free Self-Explanation condition and learning gains of 59% (M = 0.59,SD = 0.39) for the Socratic method. Furthermore, we investigated the behavior of the two strategies as a function of students' prior knowledge which was measured using learners' pretest score. For the Free Self-Explanation condition, there was no significant difference in mean learning gains for low vs. high knowledge students. The magnitude of the difference in performance (mean difference= 0.02,95% CI: -0.34 to 0.39) was very small (eta squared = 0.006). Likewise, the Socratic method showed no significant difference in mean learning gains between low vs. high performing students. The magnitude of the performance difference (mean difference =-0.24,95% CI: -0.534 to 0.03) was large (eta squared = 0.10). These findings suggest that eliciting self-explanations can be used as an effective strategy and that guided self-explanations as in the Socratic method condition is more effective at inducing learning gains.