An official website of the United States government
Large language models (LLMs) can simulate student thinking to help researchers understand learning processes and design interventions. We explore how realistically LLMs can represent students with different systems thinking (ST) and self-regulated learning (SRL) profiles in a high school environmental science curriculum, as a first step toward using LLM simulations to design ST and SRL scaffolds. We first generate chat responses (single- and multi-turn) and reflections with LLMs. We then evaluate the quality of the LLM-generated responses and compare them with real students’ responses to identify overlaps and gaps. Results show that LLMs consistently produced task-relevant, theoretically aligned outputs for balanced profiles (similar ST/SRL levels) for single-turn responses, but were less consistent for multi-turn interactions. Also, LLM-generated responses were more elaborate than students’ responses in ST and SRL strategies. We discuss the potential of LLMs to simulate student thinking and serve as a research and design tool.
more »
« less
This content will become publicly available on June 28, 2027
From Analysis to Feedback: Using Large Language Models to Support Self-Regulated Learning
Supporting students' self-regulated learning (SRL) at scale requires tools that can assess SRL depth and provide tailored feedback. We demonstrate the feasibility of using large language models (LLMs) to code students' written responses for SRL and generate adaptive feedback. Using a dataset of 606 reflection texts from 45 students, we evaluated the inter-rater agreement between several LLMs and human researchers in scoring SRL depth. Based on the LLM assessment, we generated feedback for students to apply SRL in a modeling task and explored the feedback's quality in a focus group with six SRL and science education researchers. LLMs showed substantial agreement with human raters in assessing students' SRL levels (low, medium, high) in specific strategies, including content evaluation and strategy monitoring. LLMs also produced theory-aligned and pedagogically relevant feedback, particularly for learners with higher SRL. We discuss design implications for implementing scalable SRL assessment and adaptive feedback in science learning.
more »
« less
- Award ID(s):
- 2241596
- PAR ID:
- 10693833
- Publisher / Repository:
- Association for Computing Machinery
- Date Published:
- ISBN:
- 9798400722936
- Page Range / eLocation ID:
- 496 to 500
- Format(s):
- Medium: X
- Location:
- Seoul, Republic of Korea
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Large language models (LLMs) present new opportunities for creating pedagogical agents that engage in meaningful dialogue to support student learning. However, current LLM systems used in classrooms often lack the solid theoretical foundations found in earlier intelligent tutoring systems. To bridge this gap, we propose a framework that combines Evidence-Centered Design with Social Cognitive Theory and Zone of Proximal Development for adaptive scaffolding in LLM-based agents focused on STEM+C learning. We instantiate this framework with Inquizzitor, an LLM-based formative assessment agent that integrates human-AI hybrid intelligence and provides feedback grounded in cognitive science principles. Our findings show that Inquizzitor delivers high-quality assessment and interaction aligned with core learning theories, offering effective guidance that students value. This research demonstrates the potential for theory-driven LLM integration in education, highlighting the ability of these systems to provide adaptive and principled instruction.more » « less
-
This research explores a novel human-in-the-loop approach that goes beyond traditional prompt engineering approaches to harness Large Language Models (LLMs) with chain-of-thought prompting for grading middle school students’ short answer formative assessments in science and generating useful feedback. While recent efforts have successfully applied LLMs and generative AI to automatically grade assignments in secondary classrooms, the focus has primarily been on providing scores for mathematical and programming problems with little work targeting the generation of actionable insight from the student responses. This paper addresses these limitations by exploring a human-in-the-loop approach to make the process more intuitive and more effective. By incorporating the expertise of educators, this approach seeks to bridge the gap between automated assessment and meaningful educational support in the context of science education for middle school students. We have conducted a preliminary user study, which suggests that (1) co-created models improve the performance of formative feedback generation, and (2) educator insight can be integrated at multiple steps in the process to inform what goes into the model and what comes out. Our findings suggest that in-context learning and human-in-the-loop approaches may provide a scalable approach to automated grading, where the performance of the automated LLM-based grader continually improves over time, while also providing actionable feedback that can support students’ open-ended science learning.more » « less
-
Self-regulated learning (SRL) is an instrumental skill for success in learning computer science (CS) and software engineering. SRL is an active process where students engage in cycles of planning, strategy-use, monitoring and control, and adaptation to accomplish goals. There have been calls to investigate whether SRL theory and measurement need to be made more specific to the CS education context and to integrate SRL theory when interpreting observations of student behavior in computing education research. In this work we examine the self-regulatory behavior of CS students in a 200-level CS course with an emphasis on software engineering. We performed a think-aloud study with thirty-three students working on their programming projects. Researchers then coded the think aloud data using a theory-driven codebook aligning with Winne and Hadwin’s COPES model of SRL. Using the coded data we performed a thematic analysis of two codes to better understand how students monitor their understanding of the task and set goals. Our results revealed a more complex picture than prior work of how students monitor their understanding of the task and set subgoals. We discuss the implications of our results for CS instruction and intervention designs to promote more elective SRL and learning.more » « less
-
Recent years have seen growing awareness of the potential digital storytelling brings to creating engaging K-12 learning experiences. By fostering students’ interdisciplinary knowledge and skills, digital storytelling holds great promise for realizing positive impacts on student learning in language arts as well as STEM subjects. In parallel, researchers and practitioners increasingly acknowledge the importance of computational thinking in supporting K-12 students’ problem solving across subjects and grade levels, including science and elementary school. Integrating the unique affordances of digital storytelling and computational thinking offers significant potential; however, careful attention must be given to ensure students and teachers are properly supported and not overwhelmed. In this paper, we present our work on a narrative-centered learning environment that engages upper elementary students (ages 9 to 11) in computational thinking and physical science through the creation of interactive science narratives. Leveraging log data from a pilot study with 28 students using the learning environment, we analyze the narrative programs students created across multiple dimensions to better understand the nature of the resulting narratives. Furthermore, we examine automating this analysis using artificial intelligence techniques to support real-time adaptive feedback. Results indicate that the learning environment enabled students to create interactive digital stories demonstrating their understanding of physical science, computational thinking, and narrative concepts, while the automated assessment techniques showed promise for enabling real-time feedback and support.more » « less
