More Like this

1. Comparing Different Approaches to Generating Mathematics Explanations Using Large Language Models

   https://doi.org/10.1007/978-3-031-36336-8_45  

   Prihar, Ethan; Lee, Morgan; Hopman, Mia; Kalai, Adam T.; Vempala, Sofia; Wang, Allison; Wickline, Gabriel; Murray, Aly; Heffernan, Neil (July 2023, AIED 2023)

   Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possibility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually using this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to integrate new curricula into online learning platforms. To generate explanations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations using few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/. 

   more » « less
2. Automating individualized, process-focused writing instruction: A design-based research study

   https://doi.org/10.3389/fcomm.2022.933878  

   Dux Speltz, Emily; Roeser, Jens; Chukharev-Hudilainen, Evgeny (October 2022, Frontiers in Communication)

   Writing quality is dependent upon the organization and sequencing of cognitive processes during writing. College students need writing-strategy advice that is tailored to their individual needs and is cognizant of their already-established writing processes. However, there is an obstacle to providing such advice: Both writing instructors and the writers lack awareness of the moment-by-moment actions by which text was produced. This is because switching between the processes of defining the task, coming up with ideas, outputting text, evaluating, and revising is largely regulated implicitly. To address this shortcoming, the present study uses a design-based research approach to develop and evaluate a minimally viable prototype of a system called “ProWrite” that uses novel biometric technology (concurrent keystroke logging and eye tracking) for providing real-time, individualized, automated, process-focused feedback to writers. This feedback is grounded in the analysis of each writer's individual needs and is presented in the context of a learning cycle consisting of an initial diagnostic, an intervention assignment, and a final follow-up. In two iterations, eight students used the system. Effects on student behavior were determined through direct analysis of biometric writing-process data before and after remediation and through changes in writing-process and written-product measures. Semi-structured interviews revealed that students generally considered the system useful, and they would try to use the newly learned strategies in their future writing experiences. The study demonstrated that individualized, real-time feedback informed by biometric technology can effectively modify writers' processes when writing takes place. 

   more » « less
3. Comparing Different Approaches to Generating Mathematics Explanations Using Large Language Models

   Prihar, E. (July 2023, Journal of artificial intelligence in education)

   Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possi- bility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually us- ing this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to inte- grate new curricula into online learning platforms. To generate expla- nations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations us- ing few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers’ process for writing explanations, rather than as a tool to replace them. The prompts, explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at BLINDED FOR REVIEW. 

   more » « less
4. Comparing Different Approaches to Generating Mathematics Explanations Using Large Language Models

   Prihar, E. (July 2023, AIED 2023)

   Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possi- bility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually us- ing this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to inte- grate new curricula into online learning platforms. To generate expla- nations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations us- ing few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teach- ers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/. 

   more » « less
5. Board 121: Using Tutor-led Support to Enhance Engineering Student Writing for All

   Bodenhamer, J.; Weissbach, R.; Pflueger, R.; Renguette, C.; Sorge, B.; Dasgupta, A. (June 2023, American Society for Engineering Education)

   Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication. 

   more » « less