More Like this

1. The Behavior of Large Language Models When Prompted to Generate Code Explanations

   Oli, Priti; Banjade, Rabin; Chapagain, Jeevan; Rus, Vasile (December 2023, Proceedings of the workshop on Generative AI for Education (GAIED) at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023))

   This paper systematically investigates the generation of code explanations by Large Language Models (LLMs) for code examples commonly encountered in introductory programming courses. Our findings reveal significant variations in the nature of code explanations produced by LLMs, influenced by factors such as the wording of the prompt, the specific code examples under consideration, the programming language involved, the temperature parameter, and the version of the LLM. However, a consistent pattern emerges for Java and Python, where explanations exhibit a Flesch-Kincaid readability level of approximately 7-8 grade and a consistent lexical density, indicating the proportion of meaningful words relative to the total explanation size. Additionally, the generated explanations consistently achieve high scores for correctness, but lower scores on three other metrics: completeness, conciseness, and specificity. 

   more » « less
2. The Behavior of Large Language Models When Prompted to Generate Code Explanations.

   Oli, P. (December 2023, Proceedings of The workshop on Generative AI for Education (GAIED): Advances, Opportunities, and Challenges, The Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023))

   This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. 

   more » « less
3. Explaining Code Examples in Introductory Programming Courses: LLM vs Humans

   Lekshmi-Narayanan, Arun-Balajiee; Oli, Priti; Chapagain, Jeevan; Hassany, Mohammad; Banjade, Rabin; Brusilovsky, Peter; Rus, Vasile (February 2024, Workshop on AI for Education - Bridging Innovation and Responsibility at AAAI 2024)

   Worked examples, which present an explained code for solving typical programming problems are among the most popular types of learning content in programming classes. Most approaches and tools for presenting these examples to students are based on line-by-line explanations of the example code. However, instructors rarely have time to provide explanations for many examples typically used in a programming class. In this paper, we assess the feasibility of using LLMs to generate code explanations for passive and active example exploration systems. To achieve this goal, we compare the code explanations generated by chatGPT with the explanations generated by both experts and students. 

   more » « less
4. XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs

   Chen, Z; Chen, J; Singh, A K; Sra, M (November 2024, Proc. Empirical Methods in Natural Language Processing (EMNLP))

   Large Language Models (LLMs) have achieved remarkable success in natural language tasks, yet understanding their reasoning processes re- mains a significant challenge. We address this by introducing XplainLLM, a dataset accom- panying an explanation framework designed to enhance LLM transparency and reliability. Our dataset comprises 24,204 instances where each instance interprets the LLM’s reasoning behavior using knowledge graphs (KGs) and graph attention networks (GAT), and includes explanations of LLMs such as the decoder- only Llama-3 and the encoder-only RoBERTa. XplainLLM also features a framework for gener- ating grounded explanations and the debugger- scores for multidimensional quality analysis. Our explanations include why-choose and why- not-choose components, reason-elements, and debugger-scores that collectively illuminate the LLM’s reasoning behavior. Our evaluations demonstrate XplainLLM’s potential to reduce hallucinations and improve grounded explana- tion generation in LLMs. XplainLLM is a re- source for researchers and practitioners to build trust and verify the reliability of LLM outputs. Our code and dataset are publicly available. 

   more » « less
5. Human-AI Co-Creation of Worked Examples for Programming Classes

   Hassany, Mohammad; Brusilovsky, Peter; Ke, Jiaze; Akhuseyinoglu, Kamil; Lekshmi_Narayanan, Arun_Balajiee (March 2024, Proceedings of 5th Workshop on Human-AI Co-Creation with Generative Models (HA-GEN 20224) at IUI 2024)

   Worked examples (solutions to typical programming problems presented as a source code in a certain language and are used to explain the topics from a programming class) are among the most popular types of learning content in programming classes. Most approaches and tools for presenting these examples to students are based on line-by-line explanations of the example code. However, instructors rarely have time to provide line-by-line explanations for a large number of examples typically used in a programming class. In this paper, we explore and assess a human-AI collaboration approach to authoring worked examples for Java programming. We introduce an authoring system for creating Java worked examples that generates a starting version of code explanations and presents it to the instructor to edit if necessary. We also present a study that assesses the quality of explanations created with this approach. 

   more » « less