ICUs provide life-saving capabilities to patients hospitalized for severe diseases, comorbidities, and other life-threatening conditions. However, ICUs also consume significant resources in their utilization of clinical staff and equipment. Prior studies have shown that as much as a third of hospital budgets are spent on ICUs, and a third of inpatient costs can be attributed to ICU stays (Multz et al. 1998, Shweta et al. 2013). Hence, it is in the best interest of hospitals, taxpayers, and insurers to reduce ICU costs while ensuring delivery of high-quality patient care. Since hospitals use LoS to measure the effectiveness of treatments, schedule resources and make staffing decisions, accurate LoS prediction for ICU patients should lead to better ways of managing scarce ICU resources (Romano et al. 2014). Furthermore, since LoS serves as an early indicator of future hospital readmissions, effective LoS prediction should allow healthcare practitioners to manage ICU resources better by reducing readmission rates (Singh andTerwiesch 2012, Oh et al. 2018).
With widespread adoption and use of EHR systems in recent years, researchers can use AI to analyze clinical and administrative claims data to develop more accurate predictions of patient health outcomes. However, extant research has often prioritized predictive performance over actionable and interpretable insights, a gap that undermines the practical utility of predictive models for clinical decisionmaking (Chen et al. 2023). Computer scientists and healthcare professionals have advocated for integrating intrinsic explanations within predictive models in healthcare settings, especially to promote greater utilization of AI-based, clinical decision support tools (Rudin 2019, Petch et al. 2022). AI systems with intrinsic explanation capabilities are designed to inherently explain the prediction process. Unlike post-hoc explanation methods, which generate explanations by approximating the inner working of black-box AI models, intrinsic explanations accurately reflect the prediction process with no approximation (Molnar 2022). In healthcare settings, this enhanced transparency is critical to increase physician trust in the prediction and underlying logic of AI-based clinical decision support tools.
Furthermore, recent research suggests that simple feature-based explanations are inadequate to explain the complex relationships that AI-based models utilize to generate accurate predictions (Fernández-Loría et al. 2022, Carmichael andScheirer 2023;Jiang et al. 2023). Evidence-based medicine also emphasizes the importance of recognizing the complex interactions among clinical factors in understanding patient health outcomes (Singbartl andKellum 2012, Jankovic et al. 2018). Hence, it is important to provide explanations that emphasize key interactions among features to better represent the underlying prediction process and align with clinical domain knowledge (Ahmad et al. 2018).
Yet, there remains a significant gap in the development and application of intrinsically interpretable models which effectively identify key feature interactions, particularly in healthcare settings.
To address such a gap, we develop a novel graph learning-based prediction model to intrinsically identify complex interactions between patient attributes and their impact on LoS prediction. Our model constructs patient-level relational graphs that serve as instruments to predict ICU LoS with high accuracy and interpret the contribution of salient features and feature interactions toward LoS prediction. We compare our graph learning model against alternative state-of-the-art interaction-based XAI methods. Such methods either provide interaction-based explanations of complex prediction methods in a post-hoc manner or construct intrinsically explainable models that offer interaction-based explanations. Our results indicate that prior XAI methods fail to generate meaningful explanation based on feature interactions and are computationally less efficient. In comparison, our model not only identifies the importance of feature interactions but does so more efficiently than existing XAI methods, demonstrating its superiority in providing more transparent explanations, while offering comparable predictive accuracy.
We further validate our interaction-based explanations through multiple tests to evaluate our model properties based on the Co-12 framework, which defines a set of conceptual properties for evaluation of XAI methods (Nauta et al. 2023). Utilizing perturbation analysis, we demonstrate that modifications to input features result in appropriate changes in model explanation based on the importance of the perturbed features. We deploy a distance-based separation index to test the significance of feature interactions identified by our model and confirm their relevance for ICU LoS prediction. Finally, we validate the coherence of explanations generated by our model by ensuring that salient feature interactions identified by our model are medically relevant and corroborated by prior medical research. We conduct a small-scale user study to gather feedback from ICU physicians on the insights generated by our XAI approach, and their feedback further supports the practical usability and explanations generated by our model.
In summary, we introduce a novel graph-learning, intrinsically explainable prediction model to predict ICU LoS. Our model offers enhanced explainability by generating medically-relevant explanations of interactions between patient attributes. Our intrinsic approach provides rich patient-level insights compared to existing XAI methods, thereby fostering greater trust in the prediction and enabling clinicians to make better-informed decisions using AI-based clinical decision support tools (Petch et al. 2022). Hence, our approach represents a significant contribution from a methodological and application perspective, with respect to its ability to identify salient interactions and generate patient-specific (instead of populationlevel) explanations of salient features and interactions that contribute to model prediction. Although initially designed for ICU LoS prediction, our framework is generalizable to other types of health risk prediction and can be extended to non-healthcare prediction tasks that require intrinsically explainable models capable of highlighting important feature interactions. Such methods can advance the deployment of AI applications in critical sectors, where it is paramount to deliver accurate and comprehensive explanations based on nuanced interactions between input features.
In this section, we review existing research on application of machine learning (ML) and deep learning (DL) techniques for prediction of patient health outcomes, with a focus on LoS. Subsequently, we discuss advancements in graph learning algorithms-a subset of deep learning that processes data with graph structures-and their emerging use in healthcare. We identify and discuss significant gaps and limitations in the current XAI literature and describe how our proposed graph learning model addresses these challenges. In doing so, we illustrate the potential of our research to enhance the reliability and interpretability of AI-based predictions in healthcare settings.
Length of stay in the ICU is one of the most important measures of patient health, a proxy for resource allocation decisions, and an indicator of future readmissions. Hence, accurate prediction of LoS is critical for effective ICU management and care delivery, especially for high-risk patients with severe complications (Singh and Terwiesch 2012, Romano et al. 2014, Oh et al. 2018). Severity score-based measures, such as Acute Physiology and Chronic Health Evaluation IV (APACHE IV), have been deployed in ICUs to predict patient outcomes such as mortality and LoS (Zimmerman et al. 2006). These scores were derived from regression models using patient characteristics and vital signs as independent variables. However, the efficacy of risk-score-based systems, such as APACHE IV, has come under greater scrutiny due to their limited selection of independent variables and over-reliance on the underlying statistical assumptions of logistic regression models (Zangmo and Khwannimit 2023).
In recent years, medical researchers have deployed ensemble-based ML techniques, such as random forests and gradient boosting, to predict LoS in ICU settings. These methodologies have been applied to diverse patient populations, ranging from general ICU patients to those with specific conditions such as lung cancer and COVID-19 (Alsinglawi et al. 2022, Saadatmand et al. 2023). Information systems researchers have also utilized these techniques to study preventable readmissions among patients with chronic conditions (Ben-Assuli and Padman 2020). While ensemble methods outperform statistical approaches, they are unable to exploit latent relationships, such as temporal dependencies, in clinical data.
In contrast, recent computational advancements have enabled the development of increasingly sophisticated DL models that utilize latent relationships within healthcare datasets (Morid et al. 2023). For instance, researchers have studied the application of Temporal Pointwise Convolutional Neural Networks to predict ICU LoS (Rocheteau et al. 2021, Al-Dailami et al. 2022b). These models represent distinct variations of temporal convolutional neural networks (T-CNNs) that were developed to analyze timevarying data. Alternatively, attention mechanisms have also been utilized to improve LoS prediction. These mechanisms allow neural networks to focus on relevant input data segments and have established healthcare applications, such as the Reverse Time Attention (RETAIN) model (Choi et al. 2016). Recent innovations in this stream of literature have applied variants of attention mechanisms specifically designed to handle complex healthcare data. These innovations include additional designs to process multi-modal and time series data. Examples include the Attention-Based Memory Fusion Network and Temporal-Spatial Correlation Attention Network (Al-Dailami et al. 2022a, Nie et al. 2023).
Graph learning, or deep graph learning, has emerged as a powerful approach to analyze and model data with complex interactions between entities. Traditional deep learning techniques, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), excel at handling structured data, such as images and numeric or text sequences, but struggle with complex, unstructured interactions. Graphs naturally depict these interactions through nodes and edges, making them suitable to represent a variety of real-world phenomena, including social networks, molecular structures, and transportation systems (Wu et al. 2020). In turn, graph learning methods, also known as graph neural networks (GNNs), offer a general framework for learning representations of graph data. These models aggregate and process information from the neighbors of nodes in graphs and capture complex interaction patterns within the data.
In healthcare settings, prior studies have explored the applications of graph learning methods, particularly for ICU risk prediction and chronic disease management. For example, Ma et al. (2023) constructed a patient graph to predict mortality risk in ICU patients, where the graph edges are weighted by patient similarity. The patient graph was used to identify missing patient features, and a dynamic attention mechanism was used to learn additional structural features for each patient. Carvalho et al. (2023) predicted 30-day ICU readmission risk by enriching electronic health record (EHR) data with a knowledge graph (KG) and used KG embeddings to integrate ontology information. Similarly, Sun et al. (2024) addressed EHR data heterogeneity by using multi-view graphs to encode diagnosis and medication cooccurrence and analyzed their impact on ICU outcomes. Tong et al. (2021) proposed an ICU LoS prediction model that combines Long Short-Term Memory (LSTMs) networks to extract temporal features and GNNs for exploiting similarity in patient diagnoses.
Despite a rapid increase in the deployment of AI applications in healthcare, the "black box" nature of ensemble and DL models poses a barrier to clinical use and integration, as they lack transparency in decision-making. Without being able to interpret the recommendations proposed by AI models, the adoption of AI in clinical practice has sparked criticism and raised questions about numerous legal, ethical, equity, and medical concerns (Rai 2020, Bauer andGill 2024). Due to these challenges, there has been greater emphasis in recent years on the role of XAI methods in enhancing the transparency and acceptance of AI models in healthcare. The field of XAI seeks to develop methods that explain AI-based models to enhance model interpretability, fairness, and transparency. Such explainability allows for better human understanding of AI decision-making and fosters greater trust in model outputs (Chaddad et al. 2023).
Previous studies have classified XAI methods based on various attributes (Chaddad et al. 2023).
As described in Table 1, we review various XAI methodologies and their specific applications in healthcare, emphasizing two defining characteristics: the type of explanation-intrinsic or post-hoc, and granularity of explanation-feature-based or interaction-based. In Appendix A, we provide a comprehensive comparison of the XAI methods discussed. Extant research on XAI methods has mainly focused on developing and using post-hoc, feature-based explanations to explain deep learning models. Prominent examples include Gradient-weighted Class Activation Mapping (Grad-CAM), Layer-wise Relevance Propagation (LRP), and Integrated Gradient (IG) (Bach et al. 2015, Sundararajan et al. 2017, Selvaraju et al. 2020). These methods determine the importance of input features by examining the gradient associated with each input.
Applications in healthcare include identification of critical regions for medical imaging, such as chest CT scans for COVID-19 detection (Zhang et al. 2021), creating explainable early warning scores for conditions such as sepsis (Lauritsen et al. 2020), and evaluating the significance of various input features in predicting ICU LoS (Rocheteau et al. 2021).
Outside the scope of deep learning models, post-hoc, feature-based XAI techniques are more model agnostic, offering interpretability irrespective of their underlying model architecture. Perturbation-based methods, which modify specific features to evaluate their impact on model output, have been particularly useful to identify vulnerabilities in prediction models (Finlayson et al. 2019). Model distillation techniques, such as Local Interpretable Model-agnostic Explanations (LIME), create localized linear models to interpret more complex models and have been used to explain the predictions of heart failure incidents during hospitalization (Khedkar et al. 2020). Building on LIME, the ROLEX approach was developed to provide locally faithful explanations and used to explain predictions of fragility-related fractures in patients (Kim et al. 2023). Shapley Additive Explanations (SHAP) utilize Shapley values, derived from cooperative game theory, to assign importance to individual features. SHAP has been particularly useful in explaining and predicting hospital LoS for lung cancer patients (Alsinglawi et al. 2022).
Beyond feature-based explanations, there is an emerging body of research aimed at developing techniques that provide explanations based on interactions between features. A prominent stream of work involves extending SHAP values to account for feature interactions, such as the Shapley Interaction Index (SII), Shapley Taylor Index (STI), and Faith Shapley Index (FSI) (Grabisch and Roubens 1999, Sundararajan et al. 2020, Tsai et al. 2023). These methods extend the SHAP framework to include subsets of features, thereby calculating the importance of feature interactions. Another significant stream of research in building post-hoc, interaction-based XAI methods for DL models involves extending gradientbased explanation methods to calculate the gradient of feature interactions. A notable example is Integrated Hessian (IH), an extension of IG, which utilizes the gradient of IG values to identify the importance of interactions between pairs of features. The effectiveness of IH has been studied in the context of identifying drug-drug interactions in the treatment of leukemia (Janizek et al. 2021).
Lastly, a related stream of literature focuses on developing intrinsic, interaction-based XAI methods by expanding Generalized Additive Models (GAMs) to include pairwise interaction terms. Initial work in this area started with GA 2 M, later evolving to explainable boosting machines (EBM), which constructs GAMs using ensemble decision trees, and more recently NODE-GAM, which employs deep neural networks to build GAMs (Lou et al. 2013, Nori et al. 2019, Chang et al. 2021). These methods excel at generating intrinsically explainable models but are limited by the additive nature of the GAM framework.
Computer scientists and medical professionals have increasingly advocated for using inherently interpretable models in healthcare, highlighting significant concerns with the limitations of post-hoc explanation methods. Yet, most applications of XAI in healthcare focus on utilizing feature-based, posthoc XAI methods, such as SHAP and CAM (Chaddad et al. 2023). Such post-hoc methods primarily rely on approximations and often fail to accurately represent the nuances of black box models. Information lost in this approximation process can potentially erode users' trust in the prediction model or lead to misinterpretation of predictions (Rudin 2019, Petch et al. 2022). Petch et al. (2022, p. 211) articulated the challenges associated with post-hoc explanations and argued for inherently interpretable prediction models: "…. The most notable limitation of explainability techniques is that most of them are approximations of black-box models and therefore do not precisely account for the inner workings of those models […]. A key advantage of many ML methodologies is that they can model nonlinear relationships, but the strategy of explaining black-box models through approximations may be particularly limiting […]. Even with nonlinear explainability techniques such as decision trees, the relative simplicity of explanations compared with the black-box models means that any nonlinear relationships surfaced through the explanation are likely to be oversimplifications and thus should be interpreted with caution […]. If there is no meaningful difference in accuracy between an interpretable model and a black box, an interpretable method should be used …."
This perspective highlights the critical importance of deploying intrinsically interpretable models in healthcare. Such models ensure that physicians can rely on the accuracy of the explanations provided, avoiding error-prone decisions based on prior beliefs and superficial information (Jussupow et al. 2021).
Similarly, we argue that feature-based explanations alone are insufficient to understand the complex relationships that AI models exploit to generate predictions. Instead, it is important to offer explanations that highlight key interactions between features, especially in real-world healthcare settings.
Extant research has shown that feature-based XAI methods fail to accurately explain complex AIbased models, often providing misleading or incorrect explanations (Fernández-Loría et al. 2022, Carmichael andScheirer 2023;Jiang et al. 2023). Providing misleading or incorrect interpretations can significantly impair the effectiveness of AI-based tools. Similarly, modern evidence-based medical research suggests that understanding patient health outcomes requires recognizing the complex interactions of multiple factors. In ICUs, for example, acute kidney injuries-which affect up to 25% of ICU patients-occur due to complex interactions of clinical conditions instead of individual factors (Singbartl and Kellum 2012). Likewise, drug-drug interactions in ICUs may have unexpected synergistic or antagonistic effects, further complicating patient outcomes (Jankovic et al. 2018). Therefore, it is critical to offer explanations that account for interactions among input features. We posit that such explanations may not only represent the prediction process more accurately but also align with domain knowledge, making them accessible to practitioners who often lack a background in machine learning (Ahmad et al. 2018).
Despite the importance of intrinsic, interaction-based explanations, a significant gap persists in the development and application of AI models with these capabilities. In this research, we address this gap by developing a graph learning-based XAI approach that provides intrinsic, interaction-based explanations to predict patient health outcomes, using ICU LoS as our primary research context.
Our graph learning-based model constructs patient-specific, relational graphs that not only serve as predictive instruments of ICU LoS but also explain the relationships between patient attributes that contribute to the predicted outcome. In comparison, prior studies on ICU LoS prediction primarily seek to improve prediction capabilities without explaining these models. Furthermore, prior studies that attempt to offer explanatory insights into their prediction models utilize post-hoc, feature-based XAI methods that exhibit major limitations discussed in the previous section (Rocheteau et al. 2021, Al-Dailami et al. 2022a).
Our model is different from existing graph learning models as it aims to address the task of constructing patient-level graphs to provide intrinsic, interaction-based explanations. Unlike previous graph learning applications in healthcare that analyze cohort-level graphs for patient outcome predictions, our approach utilizes patient-level graphs, enhancing both the accuracy and explainability of predictions (Tong et al. 2021, Carvalho et al. 2023, Ma et al. 2023). Furthermore, our approach autonomously constructs graph structures from data lacking any predefined graph format, identifying key interactions or edges, that are unobserved in the initial data. This approach is superior to prior graph learning models that rely on predefined graph structures or are limited to exploring only a subset of potential unobserved interactions (Kreuzer et al. 2021, Zhu et al. 2021).
We also introduce an innovative, attention-based method to assess the importance of both nodes and edges for graph-level prediction tasks. Existing methods, such as Graph Attention Networks (GAT), primarily focus on studying edge importance at the node or edge level (Veličković et al. 2018). In contrast, our model evaluates the hierarchical significance of both nodes and edges, providing a deeper and more nuanced understanding of the final prediction at the graph level. This comprehensive approach to assessing node and edge importance, combined with an ability to generate and analyze unique graphs for individual patients, allows us to develop intrinsically interpretable and accurate predictive models.
Compared to existing XAI methods, our proposed graph learning model offers a distinct advantage by providing intrinsic, interaction-based explanations. By representing each patient as a relational graph, where nodes correspond to clinical features and edges denote their interactions, our model can accurately identify nuanced interactions that contribute to the predicted outcome. In contrast, existing post-hoc interaction-based explanation methods rely on approximation of the internal mechanism of complex blackbox models. While these techniques can offer some insights, they are limited in their capacity to faithfully represent the intricate feature interactions within the prediction model.
Although recent advances in intrinsically interpretable models provide interaction-level explanations, the family of GA 2 M models, which include EBM and NODE-GAM, prioritize an optimal GAM based on features alone, before identifying and ranking potential feature interactions within the residuals. This design treats interactions as less important than individual features and limits the magnitude of their contribution to the final prediction. We empirically demonstrate that our graph learning model produces explanations that are computationally more efficient compared to post-hoc, interaction-based XAI methods and offers more insightful interaction-based explanations. This enhancement in computational efficiency and explanatory power establishes our model as a superior approach from an application and methodological perspective in understanding the key features and interactions that influence ICU LoS.
Our graph learning-based XAI approach is also distinct from extant studies that apply XAI methods to graph learning models. While XAI techniques have been developed and applied to graph learning models, these applications focus on post-hoc interpretations that illuminate the internal mechanism of black-box graph learning models (Ying et al. 2019, Zhang et al. 2022). These methods identify critical nodes and edges within predefined graph structures based on pre-trained graph learning models. In contrast, we build an intrinsically explainable model which constructs graphs from data that do not have a graph-structure format. Our approach then uses this constructed graph to predict and explain predictions of patient LoS, integrating graph construction directly into the prediction and explanation process.
In summary, we develop a novel graph-learning-based model to generate explainable predictions that highlight important interactions between input features that are not easily observable in the underlying data. This model is distinct in its ability to provide intrinsic and interaction-based explanations. We addresses the challenge of generating intrinsic, interaction-based explanations by transforming it into a graph-based task. Specifically, the objective is to construct graphs from data that initially lack graph structure and reveal the significance of features and their interactions utilizing the structure of the constructed graph. To accomplish this, we extend graph learning techniques not originally designed for this purpose to improve the explainability of our XAI approach. Table 2 summarizes the contributions of our graph learning approach to the relevant streams of literature.
In this section, we describe the specific task of predicting patient ICU LoS, data utilized in this study, as well as the design and implementation of our proposed model.
Previous studies have primarily focused on predicting the numeric value of ICU LoS by calculating the exact duration between patient admission and discharge from the ICU. However, current state-of-the-art models have shown severe limitations with this approach, as prediction errors are measured in days, rendering them less useful in real-life clinical settings (Al-Dailami et al. 2022b, Sun et al. 2024). This drawback has prompted a shift toward more accurate and interpretable LoS prediction methodologies. For example, Harutyunyan et al. (2019) transformed the task of predicting the numeric value of LoS into a multilabel classification problem and predicted the specific day of discharge for a patient after admission, with each label corresponding to a different discharge date. Alsinglawi et al. (2022) and Saadatmand et al. (2023) utilized binary predictions based on whether a patient is likely to be discharged from the ICU within a specific time window, such as within seven days of admission.
In this study, we embrace prior research and adopt a binary prediction strategy based on the likelihood of patient discharge within seven days following ICU admission. Identifying patients with predicted ICU stay exceeding one week enables early intervention of specialized care management teams, enhancing the quality of care, especially for at-risk patients (Dahl et al. 2012). We also conduct robustness tests using alternate prediction tasks, specifically the binary prediction of ICU discharge within 3 days and numeric prediction of ICU LoS, as discussed in Appendix C.
We utilize data collected from MIMIC III, a publicly accessible database provided by the MIT Lab for Computational Physiology, to assess the prediction and explanation capability of our proposed model. MIMIC III encompasses de-identified health records from 61,532 ICU admissions, compiled between 2001 and 2012 at a large academic medical center in Boston (Johnson et al. 2016). Since the MIMIC III data spans twelve years, some patients have multiple records from recurrent ICU admissions in one or more hospital visits. We only consider the first ICU stay for each patient as a qualifying stay and eliminate successive ICU visits (if applicable) to limit our research scope to prediction of LoS based on clinical data from their first ICU visit. This preempts the potential for serial correlation across multiple visits, since LoS on a later visit may depend on treatments performed during the prior ICU visit.
We further refine our dataset by excluding ICU admissions with LoS less than two days-the data collection period-to ensure that comprehensive and relevant data is used for model training. Selecting the data collection period is crucial; an excessive duration can hamper model operability, while a shorter period might not offer sufficient data for training, culminating in suboptimal prediction. Our choice of a 48-hour window is consistent with prior research and addresses the relative scarcity of clinical data for selected input variables within the first 24 hours of ICU admission (Rotar et al. 2022). Our final data set contains 22,243 ICU stays which provide the relevant data for our predictive models. Since our data has a one-toone correspondence between patients and ICU stays, we refer to them interchangeably in the following discussion. For reference, we do not remove patients who passed away during their ICU stay.
We propose a novel graph learning model to generate intrinsically explainable predictions of ICU LoS, capable of highlighting key interactions between features. This model predicts ICU LoS by constructing patient-level graphs that illustrate the importance of individual features and interactions between features at the patient level. A visual representation of the model structure is provided in Figure 1.
First, during node attribute generation (step 1), each type of input feature is transformed into a fixed-length vector within a unified feature space utilizing different projection layers, each corresponding to a specific type of input feature. These projection layers are customized based on defining characteristics of the associated type of input features-LSTM units for processing temporal data and feed-forward neural layers for the remaining types of features. Let x be the input for a given patient. In step 1, x is transformed into h, a combination of transformed feature representations htemporal and hstatic, as defined in equation (1).
Specifically, projections for temporal data are generated through LSTMs as shown in equation ( 2),
and a feed-forward layer for other feature types as shown in equation (3).
where Wstatic and bstatic are the weights and biases of the feed-forward layers.
Step 2 involves graph construction, where a fully connected directed graph, G=(V, E), is constructed based on the projected input features h. In this graph, each node i in V corresponds to a specific type of input feature with the associated projection hi encapsulated as the node attribute. Each edge eji = (j, i) in E represents the potential flow of information, or interaction, from node j to node i.
Next, we calculate edge importance in step 3, where we leverage a GAT to refine the node attributes in the constructed graph G. GAT is a specialized type of message-passing GNN that utilizes attention mechanisms to selectively focus on and aggregate relevant node-level information (Veličković et al. 2018).
Specifically, the attributes of each node are updated with the weighted sum of attributes of its neighboring nodes. These weights are dynamically determined by an edge-level attention mechanism, which assesses the relevance of each neighbor in relation to the attribute vector of the focal node. For each node i in G, its updated attribute h'i is computed as shown in equation ( 4),
where W is a learnable weight matrix, and αij are attention coefficients computed as shown in equation ( 5),
with a being a learnable weight vector of the attention mechanism. The calculated edge-level attention coefficients, αji's, describe the relevance of edge eji for updating the attributes of node i. Given the fully connected nature of the patient-level graph constructed in step 2, α's are calculated for all possible combinations of 𝑖, 𝑗 ∈ 𝑉, enabling the model to comprehensively assess all potential interactions.
Next, in step 4, the node importance calculation, an attention-based read-out mechanism is utilized to generate a vector representation, hg, of the entire graph. This process involves creating a weighted sum of the updated node attributes h' shown in equation ( 6), where βi's are attention weights computed similarly to the α's by evaluating the relevance of each node's transformed attributes h'i for the prediction task.
The β's assigned to each node not only determine hg but also serve as indicators for the importance of the updated node attributes.
Finally, in the graph-based prediction step (step 5a), the graph-level representation hg obtained from the attention-based read-out in step 4 is processed using a multi-layer perceptron, consisting of multiple feed-forward layers, to generate the final prediction for ICU LoS, as shown in equation ( 7).
Simultaneously, in the graph-based explanation step (step 5b), we construct a patient-level directed graph utilizing the attention values from steps 3 and 4. This graph encapsulates the importance of individual types of features and their interactions in contributing to the ICU LoS prediction for each patient.
Specifically, we define the importance of the node i, 𝐹𝑒𝑎𝑡𝐼𝑚𝑝 𝑖 , as the corresponding node-level attention value, βi, which represents the importance of the feature type represented by node i, shown in equation ( 8)
We then define the importance of the edge eji, 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝐼𝑚𝑝 𝑗,𝑖 , as the product of the attention value attributed to the edge, αji, with the attention value assigned to the destination node, βi. Its value equals the proportion of importance assigned to node i in step 4 attributed to the flow of information from the feature represented by node j to the feature represented by node i. 1 We interpret this value as the importance of the interaction between the features represented by node i and node j, as shown in equation ( 9).
The product term in equation ( 9) is particularly important in representing the true importance of a given edge/feature interaction to the overall prediction process. While the attention values generated by the GAT represent the relative importance of an edge for information flow to a particular node, such values are assigned at the node level and do not measure the global relevance of that edge for the graph-level prediction task. By multiplying the edge-specific attention with node-specific attention, we derive a measure of the overall importance of the edge (or interaction). Through this effective integration of GAT and an attention-
Based on the model structure described in section 3.3, we now discuss the implementation of the model for LoS prediction using the MIMIC III data. It is important to note that while we apply the model in the context of ICU LoS, the model structure described in Section 3.3 is adaptable to other prediction tasks by simply modifying the process of transforming input features into a unified vector space. The rest of the model is designed to be general purpose and applicable across various domains and datasets.
For each patient, we utilize 47 types of features across four categories: patient administrative data, diagnosis, medication data, and vital signs.
Table 3 provides descriptive statistics of selected input features. Specifically, we utilize 7 types of patient administrative data: patient age, gender, ethnicity, marital status, type of hospital admission, insurance status, and ICU admission type, which together form a 1x71 vector. Patient diagnosis is represented as a 1x18 vector, indicating the presence or absence of disease diagnoses based on 18 top-level ICD-9 categories. We include 8 types of vital sign measures: heart rate, glucose level, body temperature level, oxygen level, respiration rate, systolic blood pressure, diastolic blood pressure, and
Due to the intrinsically explanatory nature of our graph learning-based model, it is imperative to assess its predictive capabilities and the quality of explanations generated. This two-pronged evaluation ensures a comprehensive understanding of the ability of our graph learning model to not only predict accurately but also explain its prediction. We first compare the predictive performance of our graph learning-based model
(henceforth referred to as our graph model) with EBM, a custom-built DL model (henceforth referred to as the DL model), and other widely used ML algorithms.foot_2 This comparative evaluation is designed to validate the reliability and efficacy of our graph-centric approach for accurate prediction. Next, we shift our focus to the explanation dimension of our model and compare the explanations generated by various XAI techniques. Due to our interest in generating interaction-based explanations, we compare the following XAI approaches: our graph model, EBM, Integrated Hessian, and FSI, with the two latter approaches explaining the DL model (henceforth referred to as DL-IH and DL-FS, respectively). These alternative XAI methods are included as benchmarking targets based on the classification shown in Table 1. Implementations details of these methods are provided in Appendix B.
Table 4 offers a detailed comparison of the predictive performance of various models, including our graph learning model, the DL model, EBM, and conventional ML models such as XGBoost, random forests, and logistic regressions, in predicting the likelihood of ICU discharge within 7 days (of admission) across 10 cross-validation runs with an 80/20 split of train/test data. Notably, our graph model and DL models demonstrate identical performance in terms of the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC), with scores of 0.824 and 0.899, respectively. The EBM model also reports comparable AUROC and AUPRC values of 0.824 and 0.898, respectively, and exhibits the highest F1 score of 0.839 with a prediction accuracy of 0.771, matching that of the DL model. Since approximately 30% of patients in our data remain in the ICU for more than seven days, metrics such as AUROC and AUPRC gain importance for being less prone to the effects of class imbalance, compared to accuracy or F1 scores. The predictive performance of our graph learning model is superior to conventional ML models and comparable to the DL model and EBM.
While the focus of our study is to predict 7-day ICU discharge, Appendix C broadens the scope of our analysis by evaluating both prediction and explanation capabilities of our graph learning model and the EBM, to predict 3-day ICU discharge and numeric values of LoS. Although EBM demonstrates slightly higher precision in predicting 3-day discharge, our model offers superior capability in predicting numeric LoS. However, with a mean average error exceeding 5 days, we observe that numeric LoS predictions lack practical relevance. Our analysis reveals that the graph learning model and EBM identify significantly different features and interactions for the three prediction tasks. This finding highlights the complexity of feature dynamics for predictive modeling in critical care settings.
After assessing the predictive accuracy of our graph learning model, our focus shifts to its ability to explain the relationships identified by the model. Our goal is to assess whether our graph learning model and various interaction-based XAI techniques can generate meaningful explanations based on the interaction of patient attributes. In the ensuing analysis, we visually contrast the explanations generated by our graph learning model with those of other interaction-based XAI methods. This includes comparison of individual patients, patient cohorts, and evaluations of the significance of interaction-based explanations. Further, we explore their computational efficiency based on the time required to produce explanations for an individual patient.
Before we present explanations provided by various XAI methods, we first evaluate the computation time required by each method to generate explanations for an individual patient. This is particularly pertinent for interaction-based explanations that necessitate computing the importance of at least N 2 pair-wise interactions-compared to feature-based explanations that only require computing the significance of N features. Table 5 compares the computational efficiency of four types of XAI methods deployed to explain the predicted outcome for a single patient. These include two intrinsic (EBM and our graph learning model) and two post-hoc (DL-FS and DL-IH) methods.
We observe a notable discrepancy between intrinsic and post-hoc methods with respect to the computation time to generate explanations. Specifically, both EBM and our graph learning model can produce explanations in under 0.1 seconds, whereas post-hoc methods require significantly more time. For instance, the DL-FS method averages 20 seconds while DL-IH requires up to 4 minutes to generate explanations for a single patient. Due to the considerably poorer performance of DL-IH, we exclude it from subsequent analysis. 4 We note that generating explanations is significantly quicker for intrinsic XAI methods, since a simple forward pass through the neural network (for our graph learning model) or the ensemble of decision trees (for EBM) is sufficient to generate the relevant explanation. On the contrary, both post-hoc techniques must, by design, calculate the relevance of each feature and their interactions at the time of generating the explanation, a process that is more computationally demanding.
In this analysis, we compare the explanations generated by DL-FS, EBM, and our graph learning model, to predict the LoS of a 46-year-old male patient, admitted through the emergency department and treated in the surgical ICU. The patient had an ICU stay exceeding seven days which was accurately predicted by all models (i.e., binary prediction LoS > 7 days). Figure D1 in Appendix D displays the explanations from the EBM model for the top 15 terms-either a feature or interaction between two specific features-that impact LoS prediction for this patient. Based on the EBM results, there is a 34.2% chance of this patient being discharged within 7 days. It identifies respiratory system-related diagnosis as a critical factor, suggesting its presence decreases the likelihood of ICU discharge within seven days by 27.3% (=1-e^-0.32). It is important to note that all 15 factors are features and do not include any feature interactions.
Figure D2 in Appendix D provides a graphical illustration of the explanation of the DL-FS model,
showing salient features and interactions that explain the LoS prediction. It estimates a 4.6% likelihood of ICU discharge within a 7-day period and identifies the prevalence of respiratory system-related diagnosis in reducing the likelihood of 7-day discharge by 12%. Vital signs, such as mean blood pressure and glucose levels, are also noteworthy as predictive indicators. DL-FS does not assign significant importance to feature interactions, with the most significant interaction only having a -0.007% impact on LoS prediction.
On the other hand, Figure 2 displays the explanation of our graph learning model, which predicts an 8.5% likelihood of discharge within 7 days. The graph representation visually emphasizes the importance of features and feature interactions through the size of nodes and width of edges in the personalized graph.
While our model identifies respiratory system-related diagnosis as a prominent explanatory feature that receives 71.81% of the total attention, closer examination reveals significant portions of this attentionspecifically, 17.954%, 17.954%, and 15.629%-can be attributed to the interactions between respiratory system diagnosis and patient age, nutritional and anti-inflammatory agents, and analgesics and central nervous system agents, respectively. This suggests that the attention assigned to respiratory system diagnosis is a function of its interactions with patient age and medications.
Compared to EBM and DL-FS, our graph learning model generates a more comprehensive explanation of ICU LoS that identifies the nuanced impact of feature interactions. For instance, only our model identifies the interaction between patient age and respiratory system diagnosis as important for this patient. Such an interaction is medically sound, as prior research has observed gradual deterioration in lung function as patients age, emphasizing the need to consider patient age when diagnosing and treating respiratory system conditions (Sharma and Goodwin 2006).
Next, we aggregate patient-level explanations generated by the three XAI methods-DL-FS, EBM, and our graph learning model-at the cohort level to demonstrate the importance of features and feature interactions across the patient population. By averaging the attention scores of nodes (representing features)
and edges (representing interactions) from our graph learning model across patients, we can identify key features and interactions that are relevant for LoS prediction across the patient population. Similarly, the EBM model calculates global term importance for each feature and pairwise interaction as the mean of absolute importance values across all patients. We apply a similar approach to calculate the mean absolute values of Faith SHAP scores that quantify the average influence of each feature or interaction on prediction of LoS. Tables 6 and 7 provide a comparative evaluation of the salient features and interactions determined by the three XAI methods. Although the specific importance values are not directly comparable across the three methods, the rankings of the features and interactions can be compared.
Based on the DL-FS results shown in the left panel of
Table 6, we observe that patient diagnoses, age, and blood pressure during the final two hours of the ICU stay, are significant predictors of LoS. Similarly, the EBM model identifies different types of diagnosis, ICU type, and medications, as salient factors in predicting LoS, with respiratory system diagnosis emerging as the most influential factor. Our graph learning model also reports different types of patient diagnoses, age, vital signs, and ICU type, as important features. On average, 11.5% of the attention is assigned to respiratory system diagnosis, demonstrating its role as the most important feature. Our results indicate a high level of consistency across the three XAI methods with respect to salient features for LoS prediction. On the other hand, the results reported in Table 7 provide a different perspective of the explanations based on feature interactions. A closer examination of mean absolute Faith SHAP scores shows that DL-FS largely overlooks the importance of feature interactions. For example, admission to a Medical Intensive Care Unit (MICU) is ranked as the tenth most important feature by Faith SHAP, influencing the predicted likelihood of an extended ICU stay by an average of 0.69%. In comparison, the top-rated interaction between Systolic Blood Pressure and Mean Blood Pressure (during hours 46-48) has a trivial importance score of 0.02%, implying an almost negligible impact on the likelihood of LoS prediction. Such findings suggest that DL-FS struggles to assign substantive importance to feature interactions. In contrast, EBM and our graph learning model attribute meaningful significance to feature interactions. The mean importance scores for feature interactions assigned by these methods are substantial compared to individual features.
The quantitative evaluation of explanations generated by various interaction-based XAI methods poses a significant challenge. Existing evaluation techniques deployed in prior XAI research primarily focus on feature-level analysis or utilize synthetic data with predetermined underlying relationships (Janizek et al. 2021, Kim et al. 2023). Consequently, these methods are not directly applicable in assessing the quality of interaction-based explanations in our research context. Drawing on the literature on concept-based explanations, we develop an approach to evaluate the efficacy of interaction-based explanations by measuring the enhanced differentiation provided (Crabbé and van der Schaar 2022). We propose that including interactions in explanations, as opposed to utilizing only features, should improve the ability to distinguish patients with different LoS outcomes. We utilize t-distributed stochastic neighbor embedding (T-SNE), a well-known technique for dimension reduction and visualization, to process and visualize the high-dimensional explanations generated. Specifically, we generate 2-dimensional T-SNE plots for the patient cohort using our explanations from the DL-FS, EBM, and graph learning model. For each XAI method, two plots are generated: one based on the importance scores of the top features, and another based on the scores of top features and interactions. Each patient is then classified based on their outcomes, specifically whether they were discharged within 7 days.
Our proposition suggests that T-SNE plots generated from feature and interaction importance scores should offer better separation between patients with different outcomes compared to T-SNE plots that only include feature importance scores. We assess the separation within the T-SNE plots using the Distance-based Separation Index (DSI) (Guan and Loew 2022). DSI measures the degree of separability between two sets of data, with a value between 0 to 1, with a higher DSI value indicating a greater degree of separation. By examining the DSI values, we can determine the effectiveness of including interactionbased elements in explanations, with the expectation of observing greater separation based on explanations provided by including feature and interaction-based importance scores.
Table 8 shows the average DSI improvement for the DL-FS, EBM, and graph learning models, highlighting the impact of interaction-based explanations in T-SNE plots. These improvements are calculated across ten cross-validation runs, utilizing different training/test splits generated randomly with each pair of T-SNE plots derived from the respective test dataset. Furthermore, we evaluate the statistical significance of these improvements using a paired t-test. The results demonstrate that inclusion of interaction terms in the graph learning model enhances patient separation in the corresponding T-SNE plots, i.e., they accurately explain whether patients are likely to be discharged (or not) within 7 days. This improvement is statistically significant in all T-SNE configurations for the graph learning model in Table 8. However, inclusion of interaction terms in the EBM and DL-FS models does not yield statistically significant enhancements in patient separation. This indicates that the interaction-based portion of explanation from EBM and DL-FS do not augment the insights provided by the feature-based portion.
The failure of DL-FS to enhance patient separation with interaction terms can be attributed to its insufficient emphasis on these terms, as demonstrated in Table 7. Similarly, the issue with EBM arises from the design philosophy of the GA 2 M family of algorithms. GA 2 M prioritizes building an optimal GAMbased on features before identifying and ranking potential feature interactions within the residuals. Only the top-ranked feature pairs determined through cross-validation are included in the final model. This sequential estimation approach, which focuses initially on individual features, and subsequently on their interactions, may explain why EBM yields high predictive accuracy without providing interaction-based explanations that offer additional insights. In contrast, our graph learning model adopts a novel methodology by first assessing the significance of feature interactions using GAT before evaluating the importance of individual features. This ensures that interaction-based explanations can consistently enrich the separation between patients with different LoS outcomes.
We illustrate this separation in Figures 3 and 4 where we present the T-SNE plots derived from explanations generated by our graph learning model. These plots compare the visual clustering of patients based on 30 features versus a combination of 30 features and 30 interactions, with both plots subjected to a perplexity of 100 and 10,000 iterations. Figure 3 The DSI scores are 0.171 and 0.097 for the T-SNE plots with and without interactions, respectively, which suggest that interaction-based explanations contribute to an improvement of 0.074 in DSI.
In Appendix E, we further evaluate the utility of interaction-based explanations of our graph learning model, focusing on the degree of separation enabled by feature interactions. We demonstrate that the attention values attributed to two interactions, exhibit significantly different distributions across patient groups in Figures E1 and E2. Specifically, patients with ICU stay longer than seven days are more likely to exhibit salient attention on the interaction between patient age and respiratory system diagnoses. In contrast, patients with stays shorter than seven days are more likely to exhibit salient attention on the interaction between patient age and mental disorders. Although we highlight only two of the top ten interactions, the importance of all interactions in Table 7 show significant differences between the two patient groups.
In this section, we further validate the explanations generated by our graph learning model, based on the Co-12 framework (Nauta et al. 2023). The Co-12 framework is a collection of 12 key properties that can be used to systematically evaluate explanations generated by machine learning models. Evaluation of XAI methods is a nascent area of research and early studies have primarily focused on the assessment of posthoc, feature-based XAI methods. These studies focus on the consistency between the explanations and underlying prediction model either by observing the impact of removing features deemed as important or by assessing local fidelity scores (Janizek et al. 2021, Kim et al. 2023). However, there is a notable research gap with respect to systematic evaluation of intrinsic, interaction-based XAI methods. Our evaluation approach includes several tests designed to assess whether the explanations generated by the graph learning model adhere to the Co-12 framework. A summary of our evaluation approach is provided in Table 9.
The first two properties, correctness and completeness, are foundational to assess the quality of explanations provided by any XAI method. Correctness ensures that explanations accurately reflect the predictions of the underlying model, while completeness emphasizes the need for explanations to fully represent the model decision-making process. Since our model explanations are intrinsically derived from the prediction model, the correctness of explanations is inherently assured. Since these explanations are generated directly by the model, they offer a complete view of the decision process by design. This relationship between model prediction and explanations distinguish intrinsic methods from post-hoc alternatives. Hence, we argue that our graph learning model satisfies both correctness and completeness properties of the Co-12 framework.
The compactness property states that explanations should be succinct and sparse. Figure 2 suggests that the explanations of our graph learning model are compact, focusing on a limited set of key features and interactions. Furthermore, the results described in Appendix F demonstrate that the distribution of node and edge importance scores adheres to a zero-inflated pattern, indicating that our model assigns substantial attention only to a limited number of nodes and edges across all patients.
Next, we evaluate the explanations provided by our graph learning model through the lens of consistency, continuity, and contrastivity. These properties suggest that the ability of the XAI model to provide explanations should accurately reflect the importance of input features, highlighting its capability to generate reliable and meaningful insights into the prediction. We evaluate these properties by perturbing the most and least significant diagnoses-respiratory system-related diagnosis as the most important and blood-forming organs as the least important. For all patients diagnosed with both categories of conditions, we generate explanations using the original data, a perturbed version excluding blood-forming organrelated diagnosis, and another excluding respiratory system-related diagnoses. The impact of perturbation analyses using aggregated graph-based explanation is represented in Figures 5a, 5b, and 5c.
While Figure 5a represents the explanations based on the original data, Figure 5b demonstrates that perturbation of a less critical diagnosis category across the relevant patient cohort-blood-forming organs-does not significantly alter the explanation or attention values of the nodes (features). For instance, the prominence of respiratory system diagnosis remains unaffected as do other important nodes such as patient age, mental disorder, and injury and poisoning diagnosis. This stability indicates that variations in less critical diagnostic categories have negligible effects on model explanation. Conversely, Figure 5c demonstrates that perturbation of an important feature-respiratory system diagnosis-leads to a drastic change in the importance of other features and interactions. For example, patient age emerges as the most significant node in explaining LoS prediction when respiratory diagnosis is perturbed. Overall, perturbation analysis confirms our model adherence to the principles of consistency, continuity, and contrastivity.
Next, we utilize logit regressions to assess the property of confidence, which is related to probability-based confidence measures of model explanation. Specifically, we focus on providing statistical confidence in the significance of interactions identified by our model. We choose logit regressions due to their ability to support goodness-of-fit tests. We compare two logit models: one with only salient features identified by our graph learning model as independent variables, and another which includes both salient features and interactions. The goal is to determine whether inclusion of the interaction terms improves the goodness of fit for predicting ICU stay. We present a comparison of the two logit regression models in Table 10. The logit regression utilizing interactions increases McFadden's R-square from 0.218 to 0.228, which is statistically significant based on the likelihood ratio test and reduces the Akaike Information Criterion (AIC) from 22,156 to 22,069. These results confirm the importance of the interactions identified by our graph learning model and establish their statistical significance with high confidence.
Next, we evaluate the coherence of our model explanations with the established medical literature.
Coherence ensures that explanations align with domain knowledge while covariate complexity requires explanations to be understandable to the target audience. Such properties are particularly important as prior research suggests that XAI methods can enhance user trust in algorithms and confidence in decision making when designed using task-specific domain knowledge (Lee and Ram 2023). We assess these two properties by cross-referencing salient interactions identified by our model with findings from extant clinical research.
Table 11 presents the top 10 salient interactions identified by our graph learning model (as reported in Table 7), along with supporting clinical evidence which provide evidence-based support for the validity of our model explanations. For example, our model highlights the interaction between "nutritional and antiinflammatory agents" and heart rate as important predictors of ICU LoS. This is supported by the medical literature, which suggest that short-term usage of corticosteroids, a type of anti-inflammatory agent, is associated with significant decrease in heart rate and can lead to bradycardia (Brotman et al. 2005). In other words, attention should be given to patient heart rate when prescribing this type of medication. Similarly, the interaction between respiratory system diagnosis and patient age reflects the impact of age on lung capacity and increased risk of respiratory failure, which may affect recovery time and LoS (Sharma and Goodwin 2006). These results indicate that our model explanations are not only consistent with evidencebased medicine but also provide insights that are unavailable using traditional XAI methods.
In this section, we discuss the results of a small-scale user study followed by the research implications.
To evaluate the usefulness of our graph learning model in clinical settings, we designed a small-scale user study based on a survey of ICU clinicians (Kim et al. 2023). The questionnaire was comprised of five statements based on the explanations between patient attributes and ICU LoS, as identified by our graph learning XAI approach. The respondents include six practicing ICU physicians in central Texas, who rated the statements on a 5-point Likert scale, where 1 = "Strongly Disagree," 2 = "Disagree," 3 = "Neither Agree nor Disagree," 4 = "Agree," and 5 = "Strongly Agree." The survey statements describe the interactions between patient age and different diagnoses on ICU LoS. The survey also included an open-ended question to elicit physician feedback regarding the feasibility of using our XAI approach to improve care delivery.
Tables H1 and H2 in Appendix H present the survey statements, mean Likert scores from the respondents, and their written responses to the final question. The results show that ICU clinicians generally disagreed with the cohort-level statements (i.e., Q1 and Q2), with mean Likert scores below 3. However, they generally agreed with individual patient-level statements (i.e., Q3). The salient interaction between patient age and respiratory system diagnoses, as illustrated in Figure 2, is discussed in statement 3a.
Statements 3b and 3c provide additional explanations on the role of skin issues and mental disorders.
We anticipate that physicians would concur with our model prediction based on statement 3a, with statements 3b and 3c having minimal impact on physician opinion. We observe that physicians generally agreed with statement 3a, with a mean Likert score of 3.33, with similar responses to statements 3b and 3c.
Hence, their feedback with respect to individual patient-level insights indicates a valuable role for our model in clinical settings, especially in tailoring care plans to the unique characteristics of each patient.
Based on their free-form responses, four out of six physicians observed that our model can improve staffing and resource management efficiency, enabling better planning for patient placement and future expansion.
From an application perspective, our research demonstrates the importance of intrinsically generated explanations that identify important interactions between patient attributes for accurate prediction of ICU LoS. Compared to interaction-based XAI methods in the prior literature, our graph learning model accurately identifies complex, non-linear relationships in the underlying data, thereby offering a more nuanced understanding of their impact on LoS prediction. We demonstrate empirically that interactionbased explanations can provide more accurate and comprehensive understanding of the underlying prediction model, which is particularly important in healthcare. Furthermore, the results of our user study indicate general agreement among ICU clinicians with the patient-level explanations offered by our XAI approach, which underscores the practical relevance of our graph learning model. We posit that our model can enhance clinical decision-making and improve ICU operational efficiency by providing physicians with more transparent and comprehensible insights into LoS prediction in the ICU.
From a methodology perspective, our model provides a unique solution to the challenge of designing prediction models that are not only accurate but also capable of providing intrinsic, interactionbased explanations. By operationalizing the problem of interaction-based explanations as a patient-level graph that describes the relationships between patient attributes, our model learns the structure of patientlevel graphs by deploying an end-to-end attention-based learning approach. Our approach provides accurate identification of the underlying feature interactions that explain outcome prediction, and thereby, bridges extant research on graph learning and XAI methods. Hence, our research expands the application space of graph learning techniques and provides new tools for developing XAI methods. Overall, our proposed graph learning approach provides a novel contribution from methodological and application perspectives to address the problem of generating intrinsically interpretable solutions for risk prediction.
Our results also provide a solution to the computation complexity of O(N 2 ) associated with examining all potential interactions between features. By adding intrinsic explanation capabilities to the prediction model, computation complexity is internalized in model training, thereby enabling more computationally efficient explanations compared to extant post-hoc methods. In other words, our model generates interaction-based explanations significantly faster than alternate post-hoc methods.
Although our primary research setting is the ICU, our graph learning model can be generalized to other healthcare contexts to accurately identify key feature interactions for predicting patient health outcomes such as mortality, readmission risk, or hospitalizations. As experts increasingly recognize the limitations of post-hoc, feature-based explanations, our model offers a novel approach to generate intrinsic, interaction-based explanations that hold promise in any domain that requires an understanding of complex feature interactions and their impact on risk prediction (Petch et al. 2022, Carmichael andScheirer 2023).
In this study, we propose and test a novel graph learning-based prediction model to address the challenge of developing explainable predictions of ICU LoS which identify the importance of patient attributes and interactions between attributes. Our model intrinsically constructs a patient-level graph that describes the importance of features and feature interactions during prediction. Our model demonstrates superior explanation capability based on identification of important feature interactions, compared to traditional interaction-based, XAI methods for predicting ICU LoS. We supplement our model-based approach with a small-scale user study which demonstrates that our model provides accurate explanations that can lead to practical improvements in the care delivery process. Our model lays the foundation to develop interpretable, predictive tools which healthcare professionals can utilize to improve ICU resource allocation and enhance the clinical relevance of AI systems in providing effective patient care.
In recognizing the limitations of our research, we also identify several avenues for future research.
Our graph learning model is designed with broader applicability in mind. It can be adapted to various healthcare settings and potentially extended beyond healthcare applications. Future research may validate our model using data collected across a diverse group of hospitals, such as non-teaching institutions or safety-net hospitals, for various health outcome prediction tasks. We acknowledge that clinical features, such as diagnosis and medications, are represented in an abstract form, while clinical notes and lab results are omitted due to data sparsity and quality challenges. Future research can extend our model to include such data at a more granular level or examine the possibility of integrating large language models in our graph learning model to utilize unstructured clinical notes. Lastly, while the user study offers preliminary insights into the practical relevance of our model, future studies should adopt a comprehensive approach, using randomized field experiments, to study how such models can help practitioners improve care delivery. Figure 1. Structure of Graph Learning-Based Model Figure 5a. Original Data Figure 5b. Perturbation of Diagnosis associated with Blood-forming organ Figure 5c. Perturbation of Diagnosis associated with Respiratory system. Table 1. Axis of Comparison for XAI methods Feature-Based Interaction-Based Post-Hoc Integrated Gradient (IG), Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive Explanations (SHAP) Gradient based: Integrated Hessian (IH) SHAP based: Shapley Interaction Index (SII), Shapley Taylor Index (STI), Faithful Shapley Index (FSI) Intrinsic Regressions, simple decision trees Generalized Additive Model with Interaction (GA 2 M) *our graph learning-based model
Table 2. Contributions to Literature Body of Literature LoS Prediction Graph Learning XAI Contribution Propose a prediction model for ICU LoS that provides intrinsically explainable predictions by highlighting key interaction-based explanations. Extends application of graph learning models to the field of XAI by synthesizing existing methods to construct interpretable graphs from nonstructured data. Extends graph learning beyond data with inherent graph structures Proposes an XAI framework which provides intrinsic explanations based on the significance of feature interactions (beyond featurebased metrics alone).
Table 3. Descriptive Statistics of Selected Variables Binary Output Variable Variable Name Description and Unit of Measure Distribution 7-day discharge Binary indicator of ICU patients discharged after the seventh day 28.79% Selected Inputs Variables Vital Signs Heart Rate Heart rate of the patient measured in beats per minute 94.06 (25.22) Mean BP Mean blood arterial pressure of the patient in mmHg 78.76 (14.02) Respiration Rate Patient respiration rate in breaths per minute 19.36 (5.20) Body Temperature Body temperature of the patient in degree Celsius 37.02 (0.83) Glucose Concentration of glucose present in the blood of patients in mg/dl 141.60 (55.82) Bihourly Entry Count Count of vital sign recordings during the 2-hour period 2.97 (4.32) Administrative Variables Age Patient age in years 55.47 (27.59) GenderF Binary (1 = patient is female) 44.10% Ins Medicare Binary (1 = patient insurance is Medicare) 47.65% Adm Elective Binary (1 = patient from elective admission) 12.35% Diagnosis Variables Respiratory Binary (1 = patient has respiratory system related diagnosis under ICD 9) 50.39% Circulatory Binary (1 = patient has circulatory system related diagnosis under ICD 9) 73.74% Injury Binary (1 = patient has diagnosis pertain to injuries and poisoning under ICD 9) 43.51% Standard deviations (if applicable) are shown in parentheses.
Table 4. Predictive Performance Comparison Method Accuracy AUROC AUPRC F1 score 1 Graph Learning-based Model 0.767 (0.003) 0.824 (0.002) 0.899 (0.003) 0.829 (0.003) 2 DL Model 0.771 (0.004) 0.824 (0.004) 0.899 (0.003) 0.831 (0.004) 3 EBM 0.771 (0.005) 0.824 (0.006) 0.898 (0.005) 0.839 (0.003) 4 XGBoost 0.762 (0.004) 0.810 (0.006) 0.890 (0.005) 0.831 (0.003) 5 Random Forest 0.755 (0.05) 0.803 (0.007) 0.880 (0.005) 0.836 (0.003) 6 Logistic Regression 0.732 (0.005) 0.729 (0.007) 0.826 (0.007) 0.818 (0.004) Standard deviations are shown in parentheses.
Table 5. Computation Time for Patient-Level Explanation Intrinsic Methods Post-Hoc Methods XAI Method EBM Ours DL-FS DL-IH Time (in seconds) 0.05 0.1 20 240 Table 6. Salient Features identified by XAI Methods DL-FS Model EBM Model Graph Learning Model Feature Name Average Absolute FSI Score Feature Name Global Term Importance Feature Name Average Node Attention Respiratory system related diagnosis 0.03537 Respiratory system related diagnosis 0.56907 Respiratory system related diagnosis 0.11531 Patient Age 0.02792 Infectious and parasitic diseases 0.18134 Heart Rate 0.08628 Mean BP (Hr 46-48) 0.02719 Injuries and poisoning 0.15509 Patient Age 0.07790 Systolic BP (Hr 46-48) 0.01418 Diseases of the genitourinary system 0.10952 Injuries and poisoning 0.06780 Injuries and poisoning 0.01313 Symptoms, signs, and ill-defined conditions 0.10670 Mental Disorders 0.05178 Diastolic BP (Hr 46-48) 0.01080 Nervous system and sense organs diagnosis 0.10099 Body Temperature 0.04150 Glucose (Hr 46-48) 0.00951 Nutritional and Antiinflammatory Agents (Day 2) 0.09521 Glucose 0.03807 Infectious and parasitic disease 0.00899 ICU type (CSRU) 0.08764 MSK and Connective Tissue Diagnosis 0.03585 Glucose (Hr 44-46) 0.00702 Analgesics and Central Nervous System Agents (Day 2) 0.07710 ICU Type 0.03502 ICU type (MICU) 0.00694 ICU type (MICU) 0.07092 Symptoms, signs, and ill-defined conditions 0.03161 CSRU: Cardiac Surgery Recovery Unit, MICU: Medical Intensive Care Unit.
Table 7. Salient Interactions identified by XAI Methods DL-FS Model EBM Model Graph Learning Model Interaction Name Average Absolute FSI Score Interaction Name Global Term Importance Interaction Name Average Edge Attention Systolic BP × Mean BP (Hr 46-48) 0.0002 Metabolic and Antiinfective × Analgesics and Central Nervous System Agent (Day 2) 0.0399 Nutritional and Antiinflammatory Agents (Day 2) -> Heart Rate 0.0301 Mean BP (Hour 46-48) × Insurance (Government Subsidy) 0.0002 Patient Age × Admission (elective) 0.0377 Patient Age -> Respiratory system related diagnosis 0.0285 Oxygen Level (Hr 46-48) × neoplasms 0.0002 Oxygen Level (Hr 12-14) × Vital Sign Count (Hr 34-36) 0.0337 Nutritional and Antiinflammatory Agents (Day 2) -> Patient Age 0.0228 Oxygen Level (Hr 46-48) × Ethnicity (Eastern European) 0.0002 ICU type (CSRU) × Diseases of the genitourinary system 0.0297 Body Temperature -> Respiratory system related diagnosis 0.0199 Oxygen Level (Hr 46-48) × ethnicity (Filipino) 0.0002 Mean BP (Hr 44-46) × Diastolic BP (Hr 46-48) 0.0294 Body Temperature -> Patient Age 0.0169 Oxygen Level (Hr 46-48) + Dermatological and Respiratory Agents (Day 1) 0.0002 Diastolic BP (Hr 4-6) × Antineoplastic and Immunomodulating Agents (Day 2) 0.0248 Patient Age -> Injuries and poisoning 0.0166 Mean BP (Hr 46-48) × complications of pregnancy 0.0001 ICU type (CSRU) × Injuries and poisoning 0.0242 Analgesics and Central Nervous System Agents (Day 2)-> Respiratory system related diagnosis 0.0164 Glucose (Hr 44-46) × ethnicity (Thai) 6E-05 Heart Rate (Hr 20-22) × Nutritional and Antiinflam Agents (Day 2) 0.0228 Nutritional and Antiinflam Agents (Day 2) -> Mental Disorders 0.0142 Oxygen Level (Hr 38-40) × symptoms, signs, and ill-defined conditions 4E-05 Heart Rate (Hr 12-14) × Antineoplastic & Immuno Agents (Day 2) 0.0205 Patient Age -> Heart Rate 0.0131 Respiration Rate (Hr 38-40) × Diastolic BP (Hr 40-42) 3E-05 ICU type (MICU) × Respiratory system related diagnosis 0.0203 Patient Age -> Mental Disorders 0.0128 CSRU: Cardiac Surgery Recovery Unit, MICU: Medical Intensive Care Unit. Table 10. Goodness of Fit of Logistic Regressions Top 10 Features with highest node attention Top 10 interactions with highest edge attention AIC McFadden's R-square P-Value of Likelihood Ratio Test 1 ✓ 22156 0.218 < 2.2e-16 2 ✓ ✓ 22069 0.228 Note: Dependent variable: ICU LoS <= 7 days. Normal body temperature differs based on age (Geneva et al. 2019).
Patient Age -> Injuries and poisoning Injury severity increased as age increased (Lee et al. 2019).
Analgesics and Central Nervous System Agents (Day 2) -> Respiratory system related diagnosis Opioids utilization can lead to opioid-induced respiratory depression (Boom et al. 2012) Nutritional and Anti-inflammatory Agents (Day 2) -> Mental Disorders Corticosteroids utilization can lead to a variety of mental health problems, such as such as anxiety, depression, and psychosis (Alturaymi et al. 2023) Patient Age -> Heart Rate
Heart rate variability, a reliable indicator of heart condition, becomes less random and more predictable with aging (Acharya et al. 2004).
Older adults are more prone to cognitive and mood disorders, with latelife depression linked to increased disability, poorer physical health, and higher mortality rate (McKinnon et al. 2016).
It should be noted that the constructed graph is directional in nature, 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝐼𝑚𝑝 𝑖,𝑗 and 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝐼𝑚𝑝 𝑗,𝑖 represent different values. 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝐼𝑚𝑝 𝑖,𝑗 represent the importance of the information flow from node i to node j, while 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝐼𝑚𝑝 𝑗,𝑖 represent the importance of the information flow from node j to node i.
Alternative filling approaches, such as backward/forward filling, mean filling, or 0 filling, were examined and do not significantly influence the results.
Microsoft actively supports the EBM package, which is more up-to-date and accessible compared to NODE-GAM.
In Appendix B, we explain why the DL-IH method is much slower compared to other methods.