More Like this

1. Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models

   Calle, Paul; Bates, Averi; Reynolds, Justin; Liu, Yunlong; Cui, Haoyang; Ly, Sinaro; Wang, Chen; Zhang, Qinghao; Armendi, Alberto; Shettar, Shashank; et al (March 2025, arXivorg)

   The variability and biases in the real-world performance benchmarking of deep learning models for medical imaging compromise their trustworthiness for real-world deployment. The common approach of holding out a single fixed test set fails to quantify the variance in the estimation of test performance metrics. This study introduces NACHOS (Nested and Automated Cross-validation and Hyperparameter Optimization using Supercomputing) to reduce and quantify the variance of test performance metrics of deep learning models. NACHOS integrates Nested Cross-Validation (NCV) and Automated Hyperparameter Optimization (AHPO) within a parallelized high-performance computing (HPC) framework. NACHOS was demonstrated on a chest X-ray repository and an Optical Coherence Tomography (OCT) dataset under multiple data partitioning schemes. Beyond performance estimation, DACHOS (Deployment with Automated Cross-validation and Hyperparameter Optimization using Supercomputing) is introduced to leverage AHPO and cross-validation to build the final model on the full dataset, improving expected deployment performance. The findings underscore the importance of NCV in quantifying and reducing estimation variance, AHPO in optimizing hyperparameters consistently across test folds, and HPC in ensuring computational feasibility. By integrating these methodologies, NACHOS and DACHOS provide a scalable, reproducible, and trustworthy framework for DL model evaluation and deployment in medical imaging. 

   more » « less
2. Uncertainty-aware deep learning in healthcare: A scoping review

   https://doi.org/10.1371/journal.pdig.0000085  

   Loftus, Tyler J.; Shickel, Benjamin; Ruppert, Matthew M.; Balch, Jeremy A.; Ozrazgat-Baslanti, Tezcan; Tighe, Patrick J.; Efron, Philip A.; Hogan, William R.; Rashidi, Parisa; Upchurch, Gilbert R.; et al (August 2022, PLOS Digital Health)

   Lai, Yuan (Ed.)

   Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics. 

   more » « less
3. FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization

   Tarzanagh, Davoud Ataee; Li, Mingchen; Thrampoulidis, Christos; Oymak, Samet (January 2022, Proceedings of the 39th International Conference on Machine Learning)

   Standard federated optimization methods successfully apply to stochastic problems with singlelevel structure. However, many contemporary ML problems – including adversarial robustness, hyperparameter tuning, actor-critic – fall under nested bilevel programming that subsumes minimax and compositional optimization. In this work, we propose FEDNEST: A federated alternating stochastic gradient method to address general nested problems. We establish provable convergence rates for FEDNEST in the presence of heterogeneous data and introduce variations for bilevel, minimax, and compositional optimization. FEDNEST introduces multiple innovations including federated hypergradient computation and variance reduction to address inner-level heterogeneity. We complement our theory with experiments on hyperparameter & hyper-representation learning and minimax optimization that demonstrate the benefits of our method in practice. 

   more » « less
4. Deep reinforcement learning in medical imaging: A literature review

   Zhou, S. Kevin; Le, Hoang Ngan; Luu, Khoa; Nguyen, Hien V.; Ayache, Nicholas (January 2021, Medical image analysis)

   null (Ed.)

   Deep reinforcement learning (DRL) augments the reinforcement learning framework, which learns a sequence of actions that maximizes the expected reward, with the representative power of deep neural networks. Recent works have demonstrated the great potential of DRL in medicine and healthcare. This paper presents a literature review of DRL in medical imaging. We start with a comprehensive tutorial of DRL, including the latest model-free and model-based algorithms. We then cover existing DRL applications for medical imaging, which are roughly divided into three main categories: (I) parametric medical image analysis tasks including landmark detection, object/lesion detection, registration, and view plane localization; (ii) solving optimization tasks including hyperparameter tuning, selecting augmentation strategies, and neural architecture search; and (iii) miscellaneous applications including surgical gesture segmentation, personalized mobile health intervention, and computational model personalization. The paper concludes with discussions of future perspectives. 

   more » « less
5. Multivariate Deep Causal Network for Time Series Forecasting in Interdependent Networks

   https://doi.org/10.1109/CDC.2018.8619668  

   Madhavi, K. S.; Gilanifar, Mostafa; Zhou, Yuxun; Ozguven, Eren E.; Arghandeh, Reza (December 2018, 2018 IEEE Conference on Decision and Control (CDC))

   A novel multivariate deep causal network model (MDCN) is proposed in this paper, which combines the theory of conditional variance and deep neural networks to identify the cause-effect relationship between different interdependent time-series. The MCDN validation is conducted by a double step approach. The self validation is performed by information theory - based metrics, and the cross validation is achieved by a foresting application that combines the actual interdependent electricity, transportation, and weather datasets in the City of Tallahassee, Florida, USA. 

   more » « less