An official website of the United States government
The study introduces argument chains, a theory-driven analytic tool we developed to improve the automated assessment of argumentative writing. By attending to both the structure and content of written arguments, the argument-chains approach overcomes key limitations of structural analyses commonly used in writing assessment. Drawing on a dataset of 471 argumentative essays written by Grade 5 students, we used argument chains to reconstruct students’ multi-step reasoning, identify implicit assumptions, and assign acceptability and relevance scores to individual propositions. This analysis enabled us to pinpoint specific weaknesses in argumentation at the level of individual propositions, generating precise diagnostic information to support actionable instructional feedback. It also helped us evaluate multiple essay-level dimensions of student performance, including overall argument quality, acceptability, relevance, and consideration of opposing perspectives. Reliability studies indicated high agreement between human raters (Quadratic Weighted Kappa =.84) and high agreement between human raters and large language models (Quadratic Weighted Kappa =.81). Most discrepancies involved assumed propositions, highlighting an area that requires further investigation. While this work is in its early stages, we suggest that argument chains have the potential, with further automation, to become a broadly applicable tool for assessing argumentative writing.
more »
« less
Expanding possibilities for generative AI in qualitative analysis: Fostering student feedback literacy through the application of a feedback quality rubric
Abstract BackgroundCourses in engineering often use peer evaluation to monitor teamwork behaviors and team dynamics. The qualitative peer comments written for peer evaluations hold potential as a valuable source of formative feedback for students, yet little is known about their content and quality. PurposeThis study uses a large language model (LLM) to apply a previously tested feedback quality rubric to peer feedback comments. Our research questions interrogate the reliability of LLMs for qualitative analysis with a rubric and use Bandura's self‐regulated learning theory to assess peer feedback quality of first‐year engineering students' comments. MethodAn open‐source, local LLM was used to score each comment according to four rubric criteria. Inter‐rater reliability (IRR) with human raters using Cohen's quadratic weighted kappa was the primary metric of reliability. Our assessment of peer feedback quality utilized descriptive statistics. ResultsThe LLM achieved lower IRR than human raters, but the model's challenges mimic those of human raters. The model did achieve an excellent quadratic weighted kappa of 0.80 for one rubric criterion, which shows promise for LLM capability. For feedback quality, students generally wrote low‐ to medium‐quality comments that were infrequently grounded in specific teamwork behaviors. We identified five types of peer feedback that inform how students perceive the feedback process. ConclusionsOur implementation of GAI suggests that LLMs can be helpful for rapid iteration of research designs, but consistent and reliable analysis with generative artificial intelligence (GAI) requires significant effort and testing. To develop feedback literacy, students must understand how to provide high‐quality feedback.
more »
« less
- Award ID(s):
- 2107008
- PAR ID:
- 10697361
- Publisher / Repository:
- Wiley
- Date Published:
- Journal Name:
- Journal of Engineering Education
- Volume:
- 114
- Issue:
- 3
- ISSN:
- 1069-4730
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Purpose:Despite an abundance of manual, labor-intensive discourse analysis methods, there remains a dearth of clinically convenient, psychometrically robust instruments to measure change in real-world communication in aphasia. The Brief Assessment of Transactional Success (BATS) addresses this gap while developing automated methods for analyzing story retelling discourse. This study investigated automation of main concept (MC) analysis of stories by comparing scores from three large language models (LLMs) to those of human raters. Method:After watching/listening to each of the eight short video/audio BATS stimuli and retelling each story, 96 persons with aphasia (PWA;n= 48 female) engaged in topic-constrained conversations over Zoom with 94 familiar and 107 unfamiliar conversation partners (CPs). CPs then retold each story as co-constructed during their conversations with PWA. Audio files from the resulting 1,760 story retells were transcribed using Python and AssemblyAI's speech-to-text application programming interface.Each MC was first scored by human raters for presence, accuracy, and completeness. Raters used a semiautomated application, MainConcept. For each transcript, an MC composite ratio score was obtained. We evaluated three state-of-the-art LLMs: two proprietary models, GPT-4 and GPT-4o, and one open-source model, Llama-3-70B. The interrater reliability between each LLM versus human MC scoring was assessed via the Pearson correlation coefficient and reliability coefficients based on the generalizability theory (G-theory). Results:The Pearson correlation coefficients indicate strong positive linear relationships between LLM and human MC scores. G-theory reliability coefficients also indicate reliable scoring between LLM and human scoring across the spectrum of participants and conditions. Conclusions:This promising proof-of-concept study affirms the reliability of three LLMs in evaluating BATS story retell MCs and justifies ongoing investigation into their use. Providing clinicians and clinical researchers with automated tools for analyzing discourse without the need for prohibitively labor-intensive manual scoring could be a paradigm shift, potentially revolutionizing the aphasia intervention landscape.more » « less
-
Supporting students' self-regulated learning (SRL) at scale requires tools that can assess SRL depth and provide tailored feedback. We demonstrate the feasibility of using large language models (LLMs) to code students' written responses for SRL and generate adaptive feedback. Using a dataset of 606 reflection texts from 45 students, we evaluated the inter-rater agreement between several LLMs and human researchers in scoring SRL depth. Based on the LLM assessment, we generated feedback for students to apply SRL in a modeling task and explored the feedback's quality in a focus group with six SRL and science education researchers. LLMs showed substantial agreement with human raters in assessing students' SRL levels (low, medium, high) in specific strategies, including content evaluation and strategy monitoring. LLMs also produced theory-aligned and pedagogically relevant feedback, particularly for learners with higher SRL. We discuss design implications for implementing scalable SRL assessment and adaptive feedback in science learning.more » « less
-
Human-conducted rating tasks are resource-intensive and demand significant time and financial commitments. As Large Language Models (LLMs) like GPT emerge and exhibit prowess across various domains, their potential in automating such evaluation tasks becomes evident. In this research, we leveraged four prominent LLMs: GPT-4, GPT-3.5, Vicuna, and PaLM 2, to scrutinize their aptitude in evaluating teacher-authored mathematical explanations. We utilized a detailed rubric that encompassed accuracy, explanation clarity, the correctness of mathematical notation, and the efficacy of problem-solving strategies. During our investigation, we unexpectedly discerned the influence of HTML formatting on these evaluations. Notably, GPT-4 consistently favored explanations formatted with HTML, whereas the other models displayed mixed inclinations. When gauging Inter-Rater Reliability (IRR) among these models, only Vicuna and PaLM 2 demonstrated high IRR using the conventional Cohen’s Kappa metric for explanations formatted with HTML. Intriguingly, when a more relaxed version of the metric was applied, all model pairings showcased robust agreement. These revelations not only underscore the potential of LLMs in providing feedback on student-generated content but also illuminate new avenues, such as reinforcement learning, which can harness the consistent feedback from these models.more » « less
-
IntroductionThis paper examines the thin-slicing approach—the ability to make accurate judgments based on minimal information—in the context of scientific presentations. Drawing on research from nonverbal communication and personality psychology, we show that brief excerpts (thin slices) of transcribed texts from real presentations reliably predict overall quality evaluations. MethodsUsing a novel corpus of over 100 real-life science talks, we employ Large Language Models (LLMs) to evaluate transcripts of full presentations and their thin slices. By correlating LLM-based evaluations of short excerpts with full-talk assessments, we determine how much information is needed for accurate predictions. ResultsOur results demonstrate that LLM-based evaluations align closely with human evaluations, proving their validity, reliability, and efficiency. Critically, even very short excerpts (<10% of a talk’s transcript) strongly predict overall evaluations. This suggests that the first moments of a presentation convey relevant information that is used in quality evaluations and can shape lasting impressions. The findings are robust across different LLMs and prompting strategies. DiscussionThis work extends thin-slicing research to public speaking and connects theories of impression formation to LLMs and current research on AI communication. We discuss implications for communication and social cognition research on message reception. Lastly, we suggest an LLM-based thin-slicing framework as a scalable feedback tool to enhance human communication.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1002/jee.70024
