skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • An LLM-Based Framework for Simulating, Classifying, and Correcting Students' Programming Knowledge with the SOLO Taxonomy

This content will become publicly available on February 18, 2026

Title: An LLM-Based Framework for Simulating, Classifying, and Correcting Students' Programming Knowledge with the SOLO Taxonomy
Novice programmers often face challenges in designing computational artifacts and fixing code errors, which can lead to task abandonment and over-reliance on external support. While research has explored effective meta-cognitive strategies to scaffold novice programmers' learning, it is essential to first understand and assess students' conceptual, procedural, and strategic/conditional programming knowledge at scale. To address this issue, we propose a three-model framework that leverages Large Language Models (LLMs) to simulate, classify, and correct student responses to programming questions based on the SOLO Taxonomy. The SOLO Taxonomy provides a structured approach for categorizing student understanding into four levels: Pre-structural, Uni-structural, Multi-structural, and Relational. Our results showed that GPT-4o achieved high accuracy in generating and classifying responses for the Relational category, with moderate accuracy in the Uni-structural and Pre-structural categories, but struggled with the Multi-structural category. The model successfully corrected responses to the Relational level. Although further refinement is needed, these findings suggest that LLMs hold significant potential for supporting computer science education by assessing programming knowledge and guiding students toward deeper cognitive engagement.  more » « less
Award ID(s):
2229612
PAR ID:
10591788
Author(s) / Creator(s):
; ; ; ;
Publisher / Repository:
ACM
Date Published:
ISBN:
9798400705328
Page Range / eLocation ID:
1681 to 1682
Format(s):
Medium: X
Location:
Pittsburgh PA USA
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; (, Proceedings of the twelfth annual International Conference on International Computing Education Research (ICER '16))
    null (Ed.)
    A recent study about the effectiveness of subgoal labeling in an introductory computer science programming course both supported previous research and produced some puzzling results. In this study, we replicate the experiment with a different student population to determine if the results are repeatable. We also gave the experimental task to students in a follow-on course to explore if they had indeed mastered the programming concept. We found that the previous puzzling results were repeated. In addition, for the novice programmers, we found a statistically significant difference in performance based on whether the student had previous programming courses in high school. However, this performance difference disappears in a follow-on course after all students have taken an introductory computer science programming course. The results of this study have implications for how quickly students are evaluated for mastery of knowledge and how we group students in introductory programming courses. 
    more » « less
  2. ; ; (, Proceedings of the 2019 ACM Conference on International Computing Education Research)
    This work extends previous research on subgoal labeled instructions by examining their effect across a semester-long, Java-based CS1 course. Across four quizzes, students were asked to explain in plain English the process that they would use to solve a programming problem. In this mixed methods study, we used the SOLO taxonomy to categorize student responses about problem-solving processes and compare students who learned with subgoal labels to those who did not. The use of the SOLO taxonomy classification allows us to look deeper than the mere correctness of answers to focus on the quality of the answers produced in terms of completeness of relevant concepts and explanation of relationships among concepts. Students who learned with subgoals produced higher-rated answers in terms of complexity and quality on three of four quizzes. Also, they were three times more likely to discuss issues of data type on a question about assignments and expressions than students who did not learn with subgoal labeling. This suggests that the use of subgoal labeling enabled students to gain a deeper and more complex understanding of the material presented in the course. 
    more » « less
  3. ; ; (, IDC '18 Proceedings of the 17th ACM Conference on Interaction Design and Children)
    Introductory computer programming presents a number of challenges for blind and visually impaired screen reader users. In addition to the challenges of navigating complex code documents using a screen reader, novice programmers who are blind are often unable to experience fun coding projects such as programming games or animations. To address these accessibility barriers, we developed Bonk, an accessible programming environment that enables the creation of interactive audio games using a subset of the JavaScript programming language. Bonk enables novice programmers to create, share, play, and remix accessible audio games. In this paper, we introduce the Bonk programming toolkit and describe its use in a week-long programming workshop with blind and visually impaired high school students. Students in the workshop were able to create and share original audio games using Bonk, and expressed enthusiasm about furthering their programming knowledge. 
    more » « less
  4. (, Proceedings of the 55th ACM Technical Symposium on Computer Science Education)
    Parsons problems are drag-and-drop computer programming puzzles that require learners to place code blocks in the correct order and sometimes indentation. Introductory computer programming instructors use them to teach novice programmers how to code while optimizing problem-solving efficiency and cognitive load. While there is research on the design of Parsons problems for programmers without disabilities and programmers with visual or motor impairments, research regarding their accessibility for programmers with cognitive disabilities is scant. To identify the accessibility barriers and benefits of Parsons problems for neurodiverse programmers, an exploratory multiple-case study was conducted. Participants were asked to read eight chapters of an interactive eBook on Python and to solve Parsons problems. Within-case analyses of 15 retrospective think-aloud interviews with five novice programmers with disabilities led to four recommendations for improving the cognitive accessibility of Parsons problems. For example, programmers with seizure disorders may experience seizures when solving programming problems that require numeric calculations. Hence, creating a range of Parsons problems that do not require mental arithmetic could improve the learning experience for programmers with seizure disorders and those who struggle with mental calculations by lowering their cognitive load. Given this study's qualitative and exploratory approach, it does not offer conclusive, broadly generalizable results. Yet, it reveals detailed and promising avenues for exploration in computing education research that might elude many quantitative techniques. 
    more » « less
  5. (, ACM)
    More students are encountering computer science at multiple grade levels and so are learning more than one programming language. There is an ever-growing body of research describing how students transfer knowledge from one language to another. Current research shows that transfer helps students learn a second programming language in the interim and improves attitudes and retention of students in computer science. While novice programmers generally struggle with the same concepts [1, 12], the exact difficulties and benefits of the transition to a second programming language differ depending on the programming languages the student is learning. In order to best serve students of different backgrounds and maintain their interest in the subject, the details of transfer for different programming language combinations must be understood. This poster surveys and analyzes the current research on transfer and provides a summary of the variety of challenges and advantages students face in learning a second programming language. Additionally, interdisciplinary pedagogical approaches are discussed, integrating perspectives from applied linguistics as novel solutions to the specific issues faced in programming language transfer. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email