More Like this

1. Towards a Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis

   https://doi.org/10.1007/978-3-031-64312-5_2  

   Cohn, Clayton; Snyder, Caitlin; Montenegro, Justin; Biswas, Gautam (January 2024, Springer Nature Switzerland)

   LLMs have demonstrated proficiency in contextualizing their outputs using human input, often matching or beating human-level performance on a variety of tasks. However, LLMs have not yet been used to characterize synergistic learning in students’ collaborative discourse. In this exploratory work, we take a first step towards adopting a human-in-the-loop prompt engineering approach with GPT-4-Turbo to summarize and categorize students’ synergistic learning during collaborative discourse. Our preliminary findings suggest GPT-4-Turbo may be able to characterize students’ synergistic learning in a manner comparable to humans and that our approach warrants further investigation. 

   more » « less
2. Personalizing Student-Agent Interactions Using Log-Contextualized Retrieval-Augmented Generation (RAG)

   Cohn, C; Rayala, S; Snyder, C; Fonteles, J-H; Jain, S; Mohammed, N; Timalsina, U; Burriss, S; Ashwin, TS; Srivastava, N; et al (June 2025, AIED 2025 Workshop on Epistemics and Decision-Making in AI-Supported Education)

   Zhai, X; Latif, E; Liu, N; Biswas, G; Yin, Y (Ed.)

   Collaborative dialogue offers rich insights into students’ learning and critical thinking, which is essential for personalizing pedagogical agent interactions in STEM+C settings. While large language models (LLMs) facilitate dynamic pedagogical interactions, hallucinations undermine confidence, trust, and instructional value. Retrieval-augmented generation (RAG) grounds LLM outputs in curated knowledge, but requires a clear semantic link between user input and a knowledge base, which is often weak in student dialogue. We propose log-contextualized RAG (LC-RAG), which enhances RAG retrieval by using the environment logs to contextualize collaborative discourse. Our findings show that LCRAG improves retrieval over a discourse-only baseline and allows our collaborative peer agent, Copa, to deliver relevant, personalized guidance that supports students’ critical thinking and epistemic decision-making in a collaborative computational modeling environment, C2STEM. 

   more » « less
3. Evaluating LLMs Capabilities Towards Understanding Social Dynamics

   Tahir, A; Cheng, L; Sandoval, M; Silva, Y N; Hall, D; Liu, H (September 2024, The 2024 International Conference on Advances in Social Networks Analysis and Mining (ASONAM))

   Social media discourse involves people from different backgrounds, beliefs, and motives. Thus, often such discourse can devolve into toxic interactions. Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. Because these models are increasingly being used to ask questions of social significance, a crucial research question is whether they can understand social media dynamics. This work provides a critical analysis regarding generative LLM’s ability to understand language and dynamics in social contexts, particularly considering cyberbullying and anti-cyberbullying (posts aimed at reducing cyberbullying) interactions. Specifically, we compare and contrast the capabilities of different large language models (LLMs) to understand three key aspects of social dynamics: language, directionality, and the occurrence of bullying/anti-bullying messages. We found that while fine-tuned LLMs exhibit promising results in some social media understanding tasks (understanding directionality), they presented mixed results in others (proper paraphrasing and bullying/anti-bullying detection). We also found that fine-tuning and prompt engineering mechanisms can have positive effects in some tasks. We believe that a understanding of LLM’s capabilities is crucial to design future models that can be effectively used in social applications. 

   more » « less
4. Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy

   Islam, Tunazzina; Goldwasser, Dan (May 2025, Association for Computational Linguistics)

   The widespread use of social media has led to a surge in popularity for automated methods of analyzing public opinion. Supervised methods are adept at text categorization, yet the dynamic nature of social media discussions poses a continual challenge for these techniques due to the constant shifting of the focus. On the other hand, traditional unsupervised methods for extracting themes from public discourse, such as topic modeling, often reveal overarching patterns that might not capture specific nuances. Consequently, a significant portion of research into social media discourse still depends on labor-intensive manual coding techniques and a human-in-the-loop approach, which are both time-consuming and costly. In this work, we study the problem of discovering arguments associated with a specific theme. We propose a generic **LLMs-in-the-Loop** strategy that leverages the advanced capabilities of Large Language Models (LLMs) to extract latent arguments from social media messaging. To demonstrate our approach, we apply our framework to contentious topics. We use two publicly available datasets: (1) the climate campaigns dataset of 14k Facebook ads with 25 themes and (2) the COVID-19 vaccine campaigns dataset of 9k Facebook ads with 14 themes. Additionally, we design a downstream task as stance prediction by leveraging talking points in climate debates. Furthermore, we analyze demographic targeting and the adaptation of messaging based on real-world events. 

   more » « less
5. QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing

   https://doi.org/10.18653/v1/2023.emnlp-main.325  

   Wu, Yating; Mangla, Ritika; Durrett, Greg; Li, Junyi Jessy (December 2023, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing)

   Questions Under Discussion (QUD) is a versatile linguistic framework in which discourse progresses as continuously asking questions and answering them. Automatic parsing of a discourse to produce a QUD structure thus entails a complex question generation task: given a document and an answer sentence, generate a question that satisfies linguistic constraints of QUD and can be grounded in an anchor sentence in prior context. These questions are known to be curiosity-driven and open-ended. This work introduces the first framework for the automatic evaluation of QUD parsing, instantiating the theoretical constraints of QUD in a concrete protocol. We present QUDeval, a dataset of fine-grained evaluation of 2,190 QUD questions generated from both fine-tuned systems and LLMs. Using QUDeval, we show that satisfying all constraints of QUD is still challenging for modern LLMs, and that existing evaluation metrics poorly approximate parser quality. Encouragingly, human-authored QUDs are scored highly by our human evaluators, suggesting that there is headroom for further progress on language modeling to improve both QUD parsing and QUD evaluation. 

   more » « less