More Like this

1. ATTAIN: Attention-based Time-Aware LSTM Networks for Disease Progression Modeling.

   Zhang, Y. (January 2019, In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-2019), pp. 4369-4375, Macao, China.)

   Modeling patient disease progression using Electronic Health Records (EHRs) is critical to assist clinical decision making. Long-Short Term Memory (LSTM) is an effective model to handle sequential data, such as EHRs, but it encounters two major limitations when applied to EHRs: it is unable to interpret the prediction results and it ignores the irregular time intervals between consecutive events. To tackle these limitations, we propose an attention-based time-aware LSTM Networks (ATTAIN), to improve the interpretability of LSTM and to identify the critical previous events for current diagnosis by modeling the inherent time irregularity. We validate ATTAIN on modeling the progression of an extremely challenging disease, septic shock, by using real-world EHRs. Our results demonstrate that the proposed framework outperforms the state-of-the-art models such as RETAIN and T-LSTM. Also, the generated interpretative time-aware attention weights shed some lights on the progression behaviors of septic shock. 

   more » « less
2. Invertibility aware Integration of Static and Time-series data: An application to Lake Temperature Modeling. (2022 SDM Best Paper Award)

   https://doi.org/10.1137/1.9781611977172.79  

   Tayal, K.; Jia X.; Ghosh R.; Willard J.; Read J.; Kumar V. (April 2022, 2022 SIAM International Conference on Data Mining (SDM))

   Accurate predictions of water temperature are the foundation for many decisions and regulations, with direct impacts on water quality, fishery yields, and power production. Building accurate broad-scale models for lake temperature prediction remains challenging in practice due to the variability in the data distribution across different lake systems monitored by static and time-series data. In this paper, to tackle the above challenges, we propose a novel machine learning based approach for integrating static and time-series data in deep recurrent models, which we call Invertibility-Aware-Long Short-Term Memory(IA-LSTM), and demonstrate its effectiveness in predicting lake temperature. Our proposed method integrates components of the Invertible Network and LSTM to better predict temperature profiles (forward modeling) and infer the static features (i.e., inverse modeling) that can eventually enhance the prediction when static variables are missing. We evaluate our method on predicting the temperature profile of 450 lakes in the Midwestern U.S. and report a relative improvement of 4\% to capture data heterogeneity and simultaneously outperform baseline predictions by 12\% when static features are unavailable. 

   more » « less
3. Multi-series Time-aware Sequence Partitioning for Disease Progression Modeling.

   Yang, X. (January 2021, Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI))

   Electronic healthcare records (EHRs) are comprehensive longitudinal collections of patient data that play a critical role in modeling the disease progression to facilitate clinical decision-making. Based on EHRs, in this work, we focus on sepsis – a broad syndrome that can develop from nearly all types of infections (e.g., influenza, pneumonia). The symptoms of sepsis, such as elevated heart rate, fever, and shortness of breath, are vague and common to other illnesses, making the modeling of its progression extremely challenging. Motivated by the recent success of a novel subsequence clustering approach: Toeplitz Inverse Covariance-based Clustering (TICC), we model the sepsis progression as a subsequence partitioning problem and propose a Multi-series Time-aware TICC (MT-TICC), which incorporates multi-series nature and irregular time intervals of EHRs. The effectiveness of MT-TICC is first validated via a case study using a real-world hand gesture dataset with ground-truth labels. Then we further apply it for sepsis progression modeling using EHRs. The results suggest that MT-TICC can significantly outperform competitive baseline models, including the TICC. More importantly, it unveils interpretable patterns, which sheds some light on better understanding the sepsis progression. 

   more » « less
4. A Mass Conservation Relaxed (MCR) LSTM Model for Streamflow Simulation Across CONUS

   https://doi.org/10.1029/2024WR039131  

   Wang, Yihan; Zhang, Lujun; Erichson, N Benjamin; Yang, Tiantian (August 2025, Water Resources Research)

   Abstract The recent development of the physics‐aware Mass‐Conserving Long Short‐Term Memory network (MC‐LSTM) provides an alternative to other data‐driven Deep Learning (DL) models in hydrology. Mass‐Conserving Long Short‐Term Memory incorporates mass conservation directly into the LSTM architecture. Despite the theoretical advancements, studies have reported a surprisingly limited performance of the MC‐LSTM in streamflow simulation. We hypothesize that such a limitation is due to the unrealistic mass conservation scheme in MC‐LSTM, which overlooks unobserved incoming water fluxes beyond precipitation. As an attempt to verify this hypothesis, we propose a Mass Conservation Relaxed LSTM (MCR‐LSTM), which incorporates a bi‐directional mass relaxation (MR) component to account for potential incoming water fluxes beyond precipitation. We train and test the proposed MCR‐LSTM model across 531 watersheds in the contiguous United States (CONUS) against three baseline models: the Sacramento Soil Moisture Accounting, LSTM, and MC‐LSTM. Our results show that MCR‐LSTM outperforms MC‐LSTM despite its underperformance compared to LSTM. Specifically, MCR‐LSTM's advantage over MC‐LSTM is mainly seen in the Plains and Western U.S., where the newly incorporated MR component better simulates water loss and suggests the likely existence of additional incoming water fluxes beyond precipitation, respectively. The novelty and contribution of this study are twofold: firstly, it introduces an alternative physics‐aware DL tool (i.e., MCR‐LSTM) in hydrology with higher accuracy in specific regions compared to MC‐LSTM. Secondly, it provides a diagnosis of regions where strict, precipitation‐based mass conservation constraints may be unrealistic in streamflow simulation. 

   more » « less
5. From RNNs to Transformers: Benchmarking deep learning architectures for hydrologic prediction

   https://doi.org/10.5194/hess-29-6811-2025  

   Liu, Jiangtao; Shen, Chaopeng; O'Donncha, Fearghal; Song, Yalan; Zhi, Wei; Beck, Hylke E; Bindas, Tadd; Kraabel, Nicholas; Lawson, Kathryn (January 2025, Hydrology and Earth System Sciences)

   Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM) have achieved significant success in hydrological modeling. However, the recent breakthroughs of foundation models like ChatGPT and the Segment Anything Model (SAM) in natural language processing and computer vision have raised interest in the potential of attention mechanism-based models for hydrologic predictions. In this study, we propose a deep learning framework that seamlessly integrates multi-source, multi-scale data and multi-model modules, creating an automated platform for multi-dataset benchmarking and attention-based model comparisons beyond LSTM-centered tasks. The proposed framework enables evaluation of deep learning models across diverse hydrologic prediction tasks, including regression (daily runoff, soil moisture, snow water equivalent, and dissolved oxygen prediction), forecasting (using lagged hydrologic observations combined with meteorological inputs), autoregression (forecasting based solely on historical observations), spatial cross-validation (assessing model generalization to ungauged regions), and zero-shot forecasting (prediction without task-specific training data). Specifically, we benchmarked 11 Transformer-based architectures against a baseline Long Short-Term Memory (LSTM) model and further evaluated pretrained Large Language Models (LLMs) and Time Series Attention Models (TSAMs) regarding their capabilities for zero-shot hydrologic forecasting. Results show that LSTM models perform best in regression tasks, especially on the global streamflow dataset (median KGE = 0.75), surpassing the best-performing Transformer-based model's KGE value by 0.11. However, as tasks become more complex (from regression and forecasting to autoregression and zero-shot prediction), attention-based models gradually surpass LSTM models. This study provides a robust framework for comparing and developing different model structures in the era of large-scale models, providing a valuable benchmark for water resource modeling, forecasting, and management. 

   more » « less