skip to main content

Attention:

The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 11:00 PM ET on Thursday, October 10 until 2:00 AM ET on Friday, October 11 due to maintenance. We apologize for the inconvenience.

Explore Research Products in the NSF-PAR

Title: Synthetic Model Combination: A new machine‐learning method for pharmacometric model ensembling
Abstract

When aiming to make predictions over targets in the pharmacological setting, a data‐focused approach aims to learn models based on a collection of labeled examples. Unfortunately, data sharing is not always possible, and this can result in many different models trained on disparate populations, leading to the natural question of how best to use and combine them when making a new prediction. Previous work has focused on global model selection or ensembling, with the result of a single final model across the feature space. Machine‐learning models perform notoriously poorly on data outside their training domain, however, due to a problem known as covariate shift, and so we argue that when ensembling models the weightings for individual instances must reflect their respective domains—in other words, models that are more likely to have seen information on that instance should have more attention paid to them. We introduce a method for such an instance‐wise ensembling of models called Synthetic Model Combination (SMC), including a novel representation learning step for handling sparse high‐dimensional domains. We demonstrate the use of SMC on an example with dosing predictions for vancomycin, although emphasize the applicability of the method to any scenario involving the use of multiple models.

  more » « less
Award ID(s):
1722516
NSF-PAR ID:
10481514
Author(s) / Creator(s):
; ; ;
Publisher / Repository:
Wiley
Date Published:
Journal Name:
CPT: Pharmacometrics & Systems Pharmacology
Volume:
12
Issue:
7
ISSN:
2163-8306
Page Range / eLocation ID:
953 to 962
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ( , Lecture notes in computer science)
    Hiemstra, D. ; Moens, MF. ; Mothe, J. ; Perego, R. ; Potthast, M. ; Sebastiani, F. (Ed.)
    Our work aimed at experimentally assessing the benefits of model ensembling within the context of neural methods for passage re-ranking. Starting from relatively standard neural models, we use a previous technique named Fast Geometric Ensembling to generate multiple model instances from particular training schedules, then focusing or attention on different types of approaches for combining the results from the multiple model instances (e.g., averaging the ranking scores, using fusion methods from the IR literature, or using supervised learning-to-rank). Tests with the MS-MARCO dataset show that model ensembling can indeed benefit the ranking quality, particularly with supervised learning-to-rank although also with unsupervised rank aggregation. 
    more » « less
  2. ; ; ( , Computers & education)
    Instructors in STEM classrooms often frame students' interactions with technologies to help them learn content. For instance, in many STEM domains, instructors commonly help students translate physical 3D models into 2D drawings by prompting them to focus on (a) orienting physical 3D models and (b) generating 2D drawings. We investigate whether framing prompts that target either of these practices enhance the effectiveness of an educational technology that supports collaborative translation among multiple representations. To this end, we conducted a quasi-experiment with 565 undergraduate chemistry students. All students collaboratively built physical 3D models of molecules and translated them into 2D drawings. In a business-as-usual control condition, students drew on paper, without support from an educational technology. In two experimental conditions, students drew in an educational technology that provided feedback and prompted collaboration. One condition received framing prompts to focus on physical models (model condition); another received prompts to generate intermediary drawings on paper (draw condition). Compared to the control condition, the model condition showed higher learning gains, but the draw condition showed lower learning gains. Analyses of log data showed that students made more model-based errors, and the prompts in the model condition reduced these model-based errors. However, interviews with instructors showed that they prefer drawing-focused prompts, in contrast to our results. These findings offer theoretical insights into how students learn to translate among representations. Furthermore, they yield practical recommendations for the use of educational technologies that support learning with multiple representations. 
    more » « less
  3. ( , Proceedings of The 14th International Conference on Educational Data Mining)
    null (Ed.)
    We describe a data mining pipeline to convert data from educational systems into knowledge component (KC) models. In contrast to other approaches, our approach employs and compares multiple model search methodologies (e.g., sparse factor analysis, covariance clustering) within a single pipeline. In this preliminary work, we describe our approach's results on two datasets when using 2 model search methodologies for inferring item or KCs relations (i.e., implied transfer). The first method uses item covariances which are clustered to determine related KCs, and the second method uses sparse factor analysis to derive the relationship matrix for clustering. We evaluate these methods on data from experimentally controlled practice of statistics items as well as data from the Andes physics system. We explain our plans to upgrade our pipeline to include additional methods of finding item relationships and creating domain models. We discuss advantages of improving the domain model that go beyond model fit, including the fact that models with clustered item KCs result in performance predictions transferring between KCs, enabling the learning system to be more adaptive and better able to track student knowledge. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al ( , Computer graphics forum)
    Many visual analytics systems allow users to interact with machine learning models towards the goals of data exploration and insight generation on a given dataset. However, in some situations, insights may be less important than the production of an accurate predictive model for future use. In that case, users are more interested in generating of diverse and robust predictive models, verifying their performance on holdout data, and selecting the most suitable model for their usage scenario. In this paper, we consider the concept of Exploratory Model Analysis (EMA), which is defined as the process of discovering and selecting relevant models that can be used to make predictions on a data source. We delineate the differences between EMA and the well‐known term exploratory data analysis in terms of the desired outcome of the analytic process: insights into the data or a set of deployable models. The contributions of this work are a visual analytics system workflow for EMA, a user study, and two use cases validating the effectiveness of the workflow. We found that our system workflow enabled users to generate complex models, to assess them for various qualities, and to select the most relevant model for their task. 
    more » « less
  5. ; ; ; ; ; ( , ICML)
    Uncertainty decomposition refers to the task of decomposing the total uncertainty of a predictive model into aleatoric (data) uncertainty, resulting from inherent randomness in the data-generating process, and epistemic (model) uncertainty, resulting from missing information in the model’s training data. In large language models (LLMs) specifically, identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability, but remains an important open research question. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling, which can be applied to any pre-trained LLM. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions. We show that, when aleatoric uncertainty arises from ambiguity or under-specification in LLM inputs, this approach makes it possible to factor an (un-clarified) LLM’s predictions into separate aleatoric and epistemic terms, using a decomposition similar to the one employed by Bayesian neural networks. Empirical evaluations demonstrate that input clarification ensembling provides accurate and reliable uncertainty quantification on several language processing tasks. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email