More Like this

1. Analyzing automated content scoring for knowledge integration in science explanations using saliency maps

   Riordan, Brian ( April 2021 , 2021 Annual Meeting of the American Educational Research Association)

   null (Ed.)

   Models for automated scoring of content in educational applications continue to demonstrate improvements in human-machine agreement, but it remains to be demonstrated that the models achieve gains for the “right” reasons. For providing reliable scoring and feedback, both high accuracy and construct coverage are crucial. In this work, we provide an in-depth quantitative and qualitative analysis of automated scoring models for science explanations of middle school students in an online learning environment that leverages saliency maps to explore the reasons for individual model score predictions. Our analysis reveals that top-performing models can arrive at the same predictions for very different reasons, and that current model architectures have difficulty detecting ideas in student responses beyond keywords. 

   more » « less
2. Automated Scoring of Scientific Creativity in German

   https://doi.org/10.1002/jocb.658  

   Goecke, Benjamin ; DiStefano, Paul V ; Aschauer, Wolfgang ; Haim, Kurt ; Beaty, Roger ; Forthmann, Boris ( May 2024 , The Journal of Creative Behavior)

   Automated scoring is a current hot topic in creativity research. However, most research has focused on the English language and popular verbal creative thinking tasks, such as the alternate uses task. Therefore, in this study, we present a large language model approach for automated scoring of a scientific creative thinking task that assesses divergent ideation in experimental tasks in the German language. Participants are required to generate alternative explanations for an empirical observation. This work analyzed a total of 13,423 unique responses. To predict human ratings of originality, we used XLM‐RoBERTa (Cross‐lingual Language Model‐RoBERTa), a large, multilingual model. The prediction model was trained on 9,400 responses. Results showed a strong correlation between model predictions and human ratings in a held‐out test set (n = 2,682;r = 0.80; CI‐95% [0.79, 0.81]). These promising findings underscore the potential of large language models for automated scoring of scientific creative thinking in the German language. We encourage researchers to further investigate automated scoring of other domain‐specific creative thinking tasks.

    

   more » « less
3. Probing Saliency in Short Answer Scoring Models for Science Explanations

   Riordan, Brian ; Bichler, Sarah ; Bradford, Allison ; Linn, Marcia C. ( November 2020 , New York Academy of Sciences Natural Language, Dialog and Speech Symposium)

   null (Ed.)

   Recent work on automated scoring of student responses in educational applications has shown gains in human-machine agreement from neural models, particularly recurrent neural networks (RNNs) and pre-trained transformer (PT) models. However, prior research has neglected investigating the reasons for improvement – in particular, whether models achieve gains for the “right” reasons. Through expert analysis of saliency maps, we analyze the extent to which models attribute importance to words and phrases in student responses that align with question rubrics. We focus on responses to questions that are embedded in science units for middle school students accessed via an online classroom system. RNN and PT models were trained to predict an ordinal score from each response’s text, and experts analyzed generated saliency maps for each response. Our analysis shows that RNN and PT-based models can produce substantially different saliency profiles while often predicting the same scores for the same student responses. While there is some indication that PT models are better able to avoid spurious correlations of high frequency words with scores, results indicate that both models focus on learning statistical correlations between scores and words and do not demonstrate an ability to learn key phrases or longer linguistic units corresponding to ideas, which are targeted by question rubrics. These results point to a need for models to better capture student ideas in educational applications. 

   more » « less
4. Automated Assessment in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses

   https://doi.org/10.5281/zenodo.12729932  

   Baral, Sami ; Worden, Eamon ; Lim, Wen-Chiang ; Luo, Zhuang ; Santorelli, Christopher ; Gurung, Ashish ( January 2024 , International Educational Data Mining Society)

   Benjamin, Paaßen ; Carrie, Demmans Epp (Ed.)

   The effectiveness of feedback in enhancing learning outcomes is well documented within Educational Data Mining (EDM). Various prior research have explored methodologies to enhance the effectiveness of feedback to students in various ways. Recent developments in Large Language Models (LLMs) have extended their utility in enhancing automated feedback systems. This study aims to explore the potential of LLMs in facilitating automated feedback in math education in the form of numeric assessment scores. We examine the effectiveness of LLMs in evaluating student responses and scoring the responses by comparing 3 different models: Llama, SBERT-Canberra, and GPT4 model. The evaluation requires the model to provide a quantitative score on the student's responses to open-ended math problems. We employ Mistral, a version of Llama catered to math, and fine-tune this model for evaluating student responses by leveraging a dataset of student responses and teacher-provided scores for middle-school math problems. A similar approach was taken for training the SBERT-Canberra model, while the GPT4 model used a zero-shot learning approach. We evaluate and compare the models' performance in scoring accuracy. This study aims to further the ongoing development of automated assessment and feedback systems and outline potential future directions for leveraging generative LLMs in building automated feedback systems. 

   more » « less
5. Human and server docking prediction for CAPRI round 30‐35 using LZerD with combined scoring functions

   https://doi.org/10.1002/prot.25165  

   Peterson, Lenna X. ; Kim, Hyungrae ; Esquivel‐Rodriguez, Juan ; Roy, Amitava ; Han, Xusi ; Shin, Woong‐Hee ; Zhang, Jian ; Terashi, Genki ; Lee, Matt ; Kihara, Daisuke ( October 2016 , Proteins: Structure, Function, and Bioinformatics)

   We report the performance of protein–protein docking predictions by our group for recent rounds of the Critical Assessment of Prediction of Interactions (CAPRI), a community‐wide assessment of state‐of‐the‐art docking methods. Our prediction procedure uses a protein–protein docking program named LZerD developed in our group. LZerD represents a protein surface with 3D Zernike descriptors (3DZD), which are based on a mathematical series expansion of a 3D function. The appropriate soft representation of protein surface with 3DZD makes the method more tolerant to conformational change of proteins upon docking, which adds an advantage for unbound docking. Docking was guided by interface residue prediction performed with BindML and cons‐PPISP as well as literature information when available. The generated docking models were ranked by a combination of scoring functions, including PRESCO, which evaluates the native‐likeness of residues' spatial environments in structure models. First, we discuss the overall performance of our group in the CAPRI prediction rounds and investigate the reasons for unsuccessful cases. Then, we examine the performance of several knowledge‐based scoring functions and their combinations for ranking docking models. It was found that the quality of a pool of docking models generated by LZerD, that is whether or not the pool includes near‐native models, can be predicted by the correlation of multiple scores. Although the current analysis used docking models generated by LZerD, findings on scoring functions are expected to be universally applicable to other docking methods. Proteins 2017; 85:513–527. © 2016 Wiley Periodicals, Inc.

    

   more » « less