Search for: All records

In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of 100 open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets. 

Predicting the temporal and spatial patterns of South Asian monsoon rainfall within a season is of critical importance due to its impact on agriculture, water availability, and flooding. The monsoon intraseasonal oscillation (MISO) is a robust northward-propagating mode that determines the active and break phases of the monsoon and much of the regional distribution of rainfall. However, dynamical atmospheric forecast models predict this mode poorly. Data-driven methods for MISO prediction have shown more skill, but only predict the portion of the rainfall corresponding to MISO rather than the full rainfall signal. Here, we combine state-of-the-art ensemble precipitation forecasts from a high-resolution atmospheric model with data-driven forecasts of MISO. The ensemble members of the detailed atmospheric model are projected onto a lower-dimensional subspace corresponding to the MISO dynamics and are then weighted according to their distance from the data-driven MISO forecast in this subspace. We thereby achieve improvements in rainfall forecasts over India, as well as the broader monsoon region, at 10- to 30-d lead times, an interval that is generally considered to be a predictability gap. The temporal correlation of rainfall forecasts is improved by up to 0.28 in this time range. Our results demonstrate the potential of leveraging the predictability of intraseasonal oscillations to improve extended-range forecasts; more generally, they point toward a future of combining dynamical and data-driven forecasts for Earth system prediction. 

Abstract—This Research Work In Progress Paper examines empirical evidence on the impacts of feedback from an intelligent tutoring software on sketching skill development. Sketching is a vital skill for engineering design, but sketching is only taught limitedly in engineering education. Teaching sketching usually involves one-on-one feedback which limits its application in large classrooms. To meet the demands of feedback for sketching instruction, SketchTivity was developed as an intelligent tutoring software. SketchTivity provides immediate personalized feedback on sketching freehand practice. The current study examines the effectiveness of the feedback of SketchTivity by comparing students practicing with the feedback and without. Students were evaluated on their motivation for practicing sketching, the development of their skills, and their perceptions of the software. This work in progress paper examines preliminary analysis in all three of these areas. 

Abstract—This Research Work In Progress Paper examines empirical evidence on the impacts of feedback from an intelligent tutoring software on sketching skill development. Sketching is a vital skill for engineering design, but sketching is only taught limitedly in engineering education. Teaching sketching usually involves one-on-one feedback which limits its application in large classrooms. To meet the demands of feedback for sketching instruction, SketchTivity was developed as an intelligent tutoring software. SketchTivity provides immediate personalized feedback on sketching freehand practice. The current study examines the effectiveness of the feedback of SketchTivity by comparing students practicing with the feedback and without. Students were evaluated on their motivation for practicing sketching, the development of their skills, and their perceptions of the software. This work in progress paper examines preliminary analysis in all three of these areas.