More Like this

1. Foundation Models using Self-Improving Data Foundation Models using Self-Improving Data Augmentation

   Ma, Mingqian; Ma, Taigao; Guo, L Jay (November 2024, Foundation Models for Science Workshop, 38th Conference on Neural Information Processing Systems (NeurIPS 2024).)

   Optical multilayer thin film structures are widely used in many photonic applica- tions, including filters, absorbers, photovoltaics, display devices. The important part to enable these applications is the inverse design, which seeks to identify a suitable structure that satisfy desired optical responses. Recently, a Foundation model-based OptoGPT is proposed and has shown great potential to solve a wide range of inverse design problems. However, OptoGPT fails to design certain types of optical responses that are important to practical applications. The major rea- son is that the training data is randomly sampled and it is highly probable that these design targets are not selected in training, leading to the out-of-distribution issue. In this work, we propose a self-improving data augmentation technique by leveraging neural networks’ extrapolation ability. Using this method, we show sig- nificant improvement in various application design tasks with minimum fine-tuning. The approach can be potentially generalized to other inverse scientific foundation models. 

   more » « less
2. Geo-Foundation Models: Reality, Gaps and Opportunities

   https://doi.org/10.1145/3589132.3625616  

   Xie, Yiqun; Wang, Zhaonan; Mai, Gengchen; Li, Yanhua; Jia, Xiaowei; Gao, Song; Wang, Shaowen (November 2023, ACM)
3. Foundation models for generalist medical artificial intelligence

   https://doi.org/10.1038/s41586-023-05881-4  

   Moor, Michael; Banerjee, Oishi; Abad, Zahra Shakeri; Krumholz, Harlan M.; Leskovec, Jure; Topol, Eric J.; Rajpurkar, Pranav (April 2023, Nature)

   The exceptionally rapid development of highly flexible, reusable artificial intelligence (AI) models is likely to usher in newfound capabilities in medicine. We proposed a new paradigm for medical AI, which we refer to as generalist medical AI (GMAI). GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities. We identified a set of high-impact potential applications for GMAI and laid out specific technical capabilities and training datasets necessary to enable them. We expect that GMAI-enabled applications will challenge current strategies for regulating and validating AI devices for medicine and will shift practices associated with the collection of large medical datasets. 

   more » « less
4. Mudjacking: Patching Backdoor Vulnerabilities in Foundation Models

   Liu, Hongbin; Reiter, Michael K; Gong, Neil Zhenqiang (July 2024, USENIX Security Symposium)
5. Mudjacking: Patching Backdoor Vulnerabilities in Foundation Models

   Liu, Hongbin; Reiter, Michael K; Gong, Neil Zhenqiang (August 2024, USENIX Security Symposium)