More Like this

1. Assessing the Benefits of Model Ensembles in Neural Re-ranking for Passage Retrieval

   https://doi.org/10.1007/978-3-030-72240-1_19  

   Borges, Luís; Martins, Bruno; Callan, Jamie (April 2021, Lecture notes in computer science)

   Hiemstra, D.; Moens, MF.; Mothe, J.; Perego, R.; Potthast, M.; Sebastiani, F. (Ed.)

   Our work aimed at experimentally assessing the benefits of model ensembling within the context of neural methods for passage re-ranking. Starting from relatively standard neural models, we use a previous technique named Fast Geometric Ensembling to generate multiple model instances from particular training schedules, then focusing or attention on different types of approaches for combining the results from the multiple model instances (e.g., averaging the ranking scores, using fusion methods from the IR literature, or using supervised learning-to-rank). Tests with the MS-MARCO dataset show that model ensembling can indeed benefit the ranking quality, particularly with supervised learning-to-rank although also with unsupervised rank aggregation. 

   more » « less
2. How instructors frame students' interactions with educational technologies can enhance or reduce learning with multiple representations

   https://doi.org/https://doi.org/10.1016/j.compedu.2018.09.012  

   Wu, S. P.; Corr, J.; Rau, M. A. (April 2019, Computers & education)

   Instructors in STEM classrooms often frame students' interactions with technologies to help them learn content. For instance, in many STEM domains, instructors commonly help students translate physical 3D models into 2D drawings by prompting them to focus on (a) orienting physical 3D models and (b) generating 2D drawings. We investigate whether framing prompts that target either of these practices enhance the effectiveness of an educational technology that supports collaborative translation among multiple representations. To this end, we conducted a quasi-experiment with 565 undergraduate chemistry students. All students collaboratively built physical 3D models of molecules and translated them into 2D drawings. In a business-as-usual control condition, students drew on paper, without support from an educational technology. In two experimental conditions, students drew in an educational technology that provided feedback and prompted collaboration. One condition received framing prompts to focus on physical models (model condition); another received prompts to generate intermediary drawings on paper (draw condition). Compared to the control condition, the model condition showed higher learning gains, but the draw condition showed lower learning gains. Analyses of log data showed that students made more model-based errors, and the prompts in the model condition reduced these model-based errors. However, interviews with instructors showed that they prefer drawing-focused prompts, in contrast to our results. These findings offer theoretical insights into how students learn to translate among representations. Furthermore, they yield practical recommendations for the use of educational technologies that support learning with multiple representations. 

   more » « less
3. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling

   Hou, Bairu; Liu, Yujian; Qian, Kaizhi; Andreas, Jacob; Chang, Shiyu; Zhang, Yang (January 2024, ICML)

   Uncertainty decomposition refers to the task of decomposing the total uncertainty of a predictive model into aleatoric (data) uncertainty, resulting from inherent randomness in the data-generating process, and epistemic (model) uncertainty, resulting from missing information in the model’s training data. In large language models (LLMs) specifically, identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability, but remains an important open research question. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling, which can be applied to any pre-trained LLM. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions. We show that, when aleatoric uncertainty arises from ambiguity or under-specification in LLM inputs, this approach makes it possible to factor an (un-clarified) LLM’s predictions into separate aleatoric and epistemic terms, using a decomposition similar to the one employed by Bayesian neural networks. Empirical evaluations demonstrate that input clarification ensembling provides accurate and reliable uncertainty quantification on several language processing tasks. 

   more » « less
4. Joint Gaussian graphical model estimation: A survey

   https://doi.org/10.1002/wics.1582  

   Tsai, Katherine; Koyejo, Oluwasanmi; Kolar, Mladen (April 2022, WIREs Computational Statistics)

   Abstract Graphs representing complex systems often share a partial underlying structure across domains while retaining individual features. Thus, identifying common structures can shed light on the underlying signal, for instance, when applied to scientific discovery or clinical diagnoses. Furthermore, growing evidence shows that the shared structure across domains boosts the estimation power of graphs, particularly for high‐dimensional data. However, building a joint estimator to extract the common structure may be more complicated than it seems, most often due to data heterogeneity across sources. This manuscript surveys recent work on statistical inference of joint Gaussian graphical models, identifying model structures that fit various data generation processes. This article is categorized under:Data: Types and Structure > Graph and Network DataStatistical Models > Graphical Models 

   more » « less
5. Decomposing uncertainty for large language models through input clarification ensembling

   Hou, Bairu; Liu, Yujian; Qian, Kaizhi; Andreas, Jacob; Chang, Shiyu; Zhang, Yang (January 2024, ICML)

   Uncertainty decomposition refers to the task of decomposing the total uncertainty of a model into data (aleatoric) uncertainty, resulting from the inherent complexity or ambiguity of the data, and model (epistemic) uncertainty, resulting from the lack of knowledge in the model. Performing uncertainty decomposition for large language models (LLMs) is an important step toward improving the reliability, trustworthiness, and interpretability of LLMs, but this research task is very challenging and remains unresolved. The existing canonical method, Bayesian Neural Network (BNN), cannot be applied to LLMs, because BNN requires training and ensembling multiple variants of models, which is infeasible or prohibitively expensive for LLMs. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarifications ensemble, which bypasses the need to train new models. Rather than ensembling models with different parameters, our approach generates a set of clarifications for the input, feeds them into the fixed LLMs, and ensembles the corresponding predictions. We show that our framework shares a symmetric decomposition structure with BNN. Empirical evaluations demonstrate that the proposed framework provides accurate and reliable uncertainty quantification on various tasks. Code will be made publicly available at https://github.com/UCSB-NLP-Chang/llm_uncertainty . 

   more » « less