An official website of the United States government
Large Language Models (LLMs) have recently transformed both the academic and industrial landscapes due to their remarkable capacity to understand, analyze, and generate texts based on their vast knowledge and reasoning ability. Nevertheless, one major drawback of LLMs is their substantial computational cost for pre-training due to their unprecedented amounts of parameters. The disadvantage is exacerbated when new knowledge frequently needs to be introduced into the pre-trained model. Therefore, it is imperative to develop effective and efficient techniques to update pre-trained LLMs. Traditional methods encode new knowledge in pre-trained LLMs through direct fine-tuning. However, naively re-training LLMs can be computationally intensive and risks degenerating valuable pre-trained knowledge irrelevant to the update in the model. Recently,Knowledge-based Model Editing(KME), also known asKnowledge EditingorModel Editing, has attracted increasing attention, which aims at precisely modifying the LLMs to incorporate specific knowledge, without negatively influencing other irrelevant knowledge. In this survey, we aim at providing a comprehensive and in-depth overview of recent advances in the field of KME. We first introduce a general formulation of KME to encompass different KME strategies. Afterward, we provide an innovative taxonomy of KME techniques based on how the new knowledge is introduced into pre-trained LLMs, and investigate existing KME strategies while analyzing key insights, advantages, and limitations of methods from each category. Moreover, representative metrics, datasets, and applications of KME are introduced accordingly. Finally, we provide an in-depth analysis regarding the practicality and remaining challenges of KME and suggest promising research directions for further advancement in this field.
more »
« less
The Power of Personalization and Contextualization: Early Student Performance Forecasting with Language Models
Early forecasting of student performance in a course is a critical component of building effective intervention systems. However, when the available student data is limited, accurate early forecasting is challenging. We present a language generation transfer learning approach that leverages the general knowledge of pre-trained language models to address this challenge. We hypothesize that early forecasting can be significantly improved by fine-tuning large language models (LLMs) via personalization and contextualization using data on students' distal factors (academic and socioeconomic) and proximal non-cognitive factors (e.g., motivation and engagement), respectively. Results obtained from extensive experimentation validate this hypothesis and thereby demonstrate the prowess of personalization and contextualization for tapping into the general knowledge of pre-trained LLMs for solving the downstream task of early forecasting.
more »
« less
- Award ID(s):
- 2142558
- PAR ID:
- 10499296
- Publisher / Repository:
- The 2023 NeurIPS (Neural Information Processing Systems) Workshop on Generative AI for Education (GAIED)
- Date Published:
- Journal Name:
- The 2023 NeurIPS (Neural Information Processing Systems) Workshop on Generative AI for Education (GAIED)
- Format(s):
- Medium: X
- Location:
- Louisiana, New Orleans
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Student retention in undergraduate STEM programs is frequently undermined not by cognitive deficits, but by non-cognitive factors such as declining confidence, performance-related worry, and negative self-evaluation. Traditional early-warning systems, which rely on lagging indicators such as grades, often fail to intervene before disengagement solidifies. This work-in-progress paper presents a human-guided generative AI intervention framework that leverages large language models (LLMs) in two components combined with human expertise: (1) an LLM-based forecasting model that predicts weekly student engagement states from longitudinal self-report data, and (2) a personalization model (GPT-4o) that contextualizes human-expert-crafted intervention messages. A counseling psychology expert designs messages based on the forecasted engagement states, and GPT-4o personalizes these messages while preserving the expert's core content. We deploy this system with 41 engineering students and compare outcomes against a historical baseline cohort (N=96). The intervention group maintains significantly higher Performance Self-Evaluation throughout the semester (75.4% vs. 56.5% positive responses), with a large effect size (Cohen's d = 0.90, 95% CI [11.72, 26.14], t(91) = 5.22, p < 0.001). These preliminary findings suggest that human-guided generative AI systems hold promise for scaling personalized support in large-enrollment STEM courses.more » « less
-
Student retention in undergraduate STEM programs is frequently undermined not by cognitive deficits, but by non-cognitive factors such as declining confidence, performance-related worry, and negative self-evaluation. Traditional early-warning systems, which rely on lagging indicators such as grades, often fail to intervene before disengagement solidifies. This work-in-progress paper presents a human-guided generative AI intervention framework that leverages large language models (LLMs) in two components combined with human expertise: (1) an LLM-based forecasting model that predicts weekly student engagement states from longitudinal self-report data, and (2) a personalization model (GPT-4o) that contextualizes human-expert-crafted intervention messages. A counseling psychology expert designs messages based on the forecasted engagement states, and GPT-4o personalizes these messages while preserving the expert's core content. We deploy this system with 41 engineering students and compare outcomes against a historical baseline cohort (N=96). The intervention group maintains significantly higher Performance Self-Evaluation throughout the semester (75.4% vs. 56.5% positive responses), with a large effect size (Cohen's d = 0.90, 95% CI [11.72, 26.14], t(91) = 5.22, p < 0.001). These preliminary findings suggest that human-guided generative AI systems hold promise for scaling personalized support in large-enrollment STEM courses.more » « less
-
Recently, remarkable progress has been made over large language models (LLMs), demonstrating their unprecedented capability in varieties of natural language tasks. However, completely training a large general-purpose model from the scratch is challenging for time series analysis, due to the large volumes and varieties of time series data, as well as the non-stationarity that leads to concept drift impeding continuous model adaptation and re-training. Recent advances have shown that pre-trained LLMs can be exploited to capture complex dependencies in time series data and facilitate various applications. In this survey, we provide a systematic overview of existing methods that leverage LLMs for time series analysis. Specifically, we first state the challenges and motivations of applying language models in the context of time series as well as brief preliminaries of LLMs. Next, we summarize the general pipeline for LLM-based time series analysis, categorize existing methods into different groups (i.e., direct query, tokenization, prompt design, fine-tune, and model integration), and highlight the key ideas within each group. We also discuss the applications of LLMs for both general and spatial-temporal time series data, tailored to specific domains. Finally, we thoroughly discuss future research opportunities to empower time series analysis with LLMs.more » « less
-
Building a knowledge graph is a time-consuming and costly process which often applies complex natural language processing (NLP) methods for extracting knowledge graph triples from text corpora. Pre-trained large Language Models (PLM) have emerged as a crucial type of approach that provides readily available knowledge for a range of AI applications. However, it is unclear whether it is feasible to construct domain-specific knowledge graphs from PLMs. Motivated by the capacity of knowledge graphs to accelerate data-driven materials discovery, we explored a set of state-of-the-art pre-trained general-purpose and domain-specific language models to extract knowledge triples for metal-organic frameworks (MOFs). We created a knowledge graph benchmark with 7 relations for 1248 published MOF synonyms. Our experimental results showed that domain-specific PLMs consistently outperformed the general-purpose PLMs for predicting MOF related triples. The overall benchmarking results, however, show that using the present PLMs to create domain-specific knowledge graphs is still far from being practical, motivating the need to develop more capable and knowledgeable pre-trained language models for particular applications in materials science.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- The DOI is not currently available.
