skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Designing and Evaluating Evidence-Centered Design based Conversations for Assessment with LLMs
The current paper discusses conversation-based assessments (CBAs) created with prompt engineering for LLMs based on Evidence-Centered Design (ECD). Conversation-based assessments provide students the opportunity to discuss a given topic with artificial agent(s). These conversations elicit evidence of students’ knowledge, skills and abilities that may not be uncovered by traditional tests. We discuss our previous method of creating such conversations with regular expressions and latent semantic analysis in an expensive methodology requiring time and various expertise. Thus, in this novel work, we created a prompt-engineered version of CBAs based on evidence-centered design that remains on the domain topic throughout the conversation as well as provides evidence of the student knowledge in a less expensive way. We present the methodology for creating these prompts, compare responses to various student speech acts between the previous version and the prompt engineered version, and discuss the evidence gleaned from the conversation and based on the prompt. Finally, limitations, conclusions and implications of this work are discussed.  more » « less
Award ID(s):
2229612
PAR ID:
10591735
Author(s) / Creator(s):
; ; ;
Publisher / Repository:
Proceedings of EDM 2024 Workshop: Leveraging Large Language Models for Next Generation Educational Technologies
Date Published:
ISSN:
2157-2100
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Journal of educational data mining)
    Benjamin, Paaßen; Carrie, Demmans Epp (Ed.)
    This paper explores the differences between two types of natural language conversations between a student and pedagogical agent(s). Both types of conversations were created for formative assessment purposes. The first type is conversation-based assessment created via knowledge engineering which requires a large amount of human effort. The second type, which is less costly to produce, uses prompt engineering for LLMs based on Evidence-Centered design to create these conversations and glean evidence about students¿½f knowledge, skills and abilities. The current work compares linguistic features of the artificial agent(s) discourse moves in natural language conversations created by the two methodologies. Results indicate that more complex conversations are created by the prompt engineering method which may be more adaptive than the knowledge engineering approach. However, the affordances of prompt engi-neered, LLM generated conversation-based assessment may create more challenges for scoring than the original knowledge engineered conversations. Limitations and implications are dis-cussed. 
    more » « less
  2. ; (, Journal of Engineering Education)
    Abstract BackgroundMetacognitive processes have been linked to the development of conceptual knowledge in STEM courses, but previous work has centered on the regulatory aspects of metacognition. PurposeWe interrogated the relationship between epistemic metacognition and conceptual knowledge in engineering statics courses across six universities by asking students a difficult concept question with concurrent reflection prompts that elicited their metacognitive thinking. MethodWe used a mixed‐methods design containing an embedded phase followed by an explanatory phase. This design allowed us to both prompt and measure student epistemic metacognition within the learning context. The embedded phase consisted of quantitative and qualitative analyses of student responses. The explanatory phase consisted of an analysis of six instructor interviews. ResultsAnalysis of 267 student responses showed greater variation in students' epistemic metacognition than in their ability to answer correctly. Students used different kinds of epistemic metacognitive resources about the nature and origin of knowledge, epistemological forms, epistemological activities, and stances toward knowledge. These resources generally assembled into one of two frames: aconstructed knowledge framingvaluing conceptual knowledge and sense‐making, and anauthoritative knowledge framingforegrounding numerical, algorithmic problem‐solving. All six instructors interviewed described resources that align with both frames, and none explicitly considered student epistemic metacognition. ConclusionsInstructors' explicit attention to epistemic metacognition can potentially shift students to more productive frames for engineering learning. Findings here also inform two broader issues in STEM instruction: student resistance to active learning, and the direct instruction versus inquiry‐based learning debate. 
    more » « less
  3. ; ; ; ; ; ; (, Proceedings of the 18th WiPSCE Conference on Primary and Secondary Computing Education Research)
    Background. Software Engineering (SE) is a new and emerging topic in secondary computer science classrooms. However, a review of the recent literature has identified an overall lack of reporting on the development of SE secondary curriculum. Previous studies also report low student engagement when teaching these concepts. Objectives. In this experience report, we discuss the development of a 9-week, project-based learning (PBL) SE curriculum for secondary students. During this curriculum, students create a socially relevant project in groups of two to three. We discuss displays of participant engagement with CS concepts through the PBL pedagogy and the SE curriculum. Method. We examine participant engagement through group artifact interviews about student experiences during a week-long, virtual summer camp that piloted activities from our curriculum. During this camp, students followed a modified SE life cycle created by the authors of the paper. Findings. Participants showed engagement with the curriculum through various aspects of PBL, such as autonomy, creativity, and personal interest in their project topic. Implications. The lessons learned from this experience report suggest that PBL pedagogy can increase student engagement when teaching CS concepts, and this pedagogy provides detail and structure for future secondary SE curriculum implementations to support educators in the classroom 
    more » « less
  4. ; ; ; ; (, Disciplinary and Interdisciplinary Science Education Research)
    Abstract Student-centered instruction allows students to take ownership over their learning in the classroom. However, these settings do not always promote productive engagement. Using discourse analysis, student engagement can be analyzed based on how they are interacting with each other while completing in-class group activities. Previous analyses of student engagement in science settings have used methods that do not capture the intricacies of student group interactions such as the flow of conversation and nature of student utterances outside of argumentation or reasoning. However, these features are important to accurately assess student engagement. This study proposes a tiered analytical framework and visualization scheme for analyzing group discussion patterns that allow for a detailed analysis of student discourse moves while discussing scientific topics. This framework allows a researcher to see the flow of an entire conversation within a single schematic. The Student Interaction Discourse Moves framework can be used to extend studies using discourse analysis to determine how student groups work through problems. 
    more » « less
  5. ; ; ; ; ; ; (, Computersupported collaborative learning)
    de Vries, E. (Ed.)
    We articulate a framework for characterizing student learning trajectories as they progress through a scientific modeling curriculum. By maintaining coherence between modeling representations and leveraging key design principles including evidence-centered design, we develop mechanisms to evaluate student science and computational thinking (CT) proficiency as they transition from conceptual to computational modeling representations. We have analyzed pre-post assessments and learning artifacts from 99 6th grade students and present three contrasting vignettes to illustrate students’ learning trajectories as they work on their modeling tasks. Our analysis indicates pathways that support the transition and identify domain-specific support needs. Our findings will inform refinements to our curriculum and scaffolding of students to further support the integrated learning of science and CT. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email