skip to main content

Title: Chromatic fusion: Generative multimodal neuroimaging data fusion provides multi‐informed insights into schizophrenia

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. We apply our proposed framework, which disentangles multimodal data into private and shared sets of features from pairs of structural (sMRI), functional (sFNC and ICA), and diffusion MRI data (FA maps). With our approach, we find that heterogeneity in schizophrenia is potentially a function of modality pairs. Results show (1) schizophrenia is highly multimodal and includes changes in specific networks, (2) non‐linear relationships with schizophrenia are observed when interpolating among shared latent dimensions, and (3) we observe a decrease in the modularity of functional connectivity and decreased visual‐sensorimotor connectivity for schizophrenia patients for the FA‐sFNC and sMRI‐sFNC modality pairs, respectively. Additionally, our results generally indicate decreased fractional corpus callosum anisotropy, and decreased spatial ICA map and voxel‐based morphometry strength in the superior frontal lobe as found in the FA‐sFNC, sMRI‐FA, and sMRI‐ICA modality pair clusters. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data which we hope challenges the reader to think differently about how modalities interact.

more » « less
Author(s) / Creator(s):
 ;  ;  ;  ;  ;  ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Human Brain Mapping
Medium: X Size: p. 5828-5845
p. 5828-5845
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    There is growing evidence that rather than using a single brain imaging modality to study its association with physiological or symptomatic features, the field is paying more attention to fusion of multimodal information. However, most current multimodal fusion approaches that incorporate functional magnetic resonance imaging (fMRI) are restricted to second‐level 3D features, rather than the original 4D fMRI data. This trade‐off is that the valuable temporal information is not utilized during the fusion step. Here we are motivated to propose a novel approach called “parallel group ICA+ICA” that incorporates temporal fMRI information from group independent component analysis (GICA) into a parallel independent component analysis (ICA) framework, aiming to enable direct fusion of first‐level fMRI features with other modalities (e.g., structural MRI), which thus can detect linked functional network variability and structural covariations. Simulation results show that the proposed method yields accurate intermodality linkage detection regardless of whether it is strong or weak. When applied to real data, we identified one pair of significantly associated fMRI‐sMRI components that show group difference between schizophrenia and controls in both modalities, and this linkage can be replicated in an independent cohort. Finally, multiple cognitive domain scores can be predicted by the features identified in the linked component pair by our proposed method. We also show these multimodal brain features can predict multiple cognitive scores in an independent cohort. Overall, results demonstrate the ability of parallel GICA+ICA to estimate joint information from 4D and 3D data without discarding much of the available information up front, and the potential for using this approach to identify imaging biomarkers to study brain disorders.

    more » « less
  2. Abstract

    Multimodal neuroimaging research plays a pivotal role in understanding the complexities of the human brain and its disorders. Independent component analysis (ICA) has emerged as a widely used and powerful tool for disentangling mixed independent sources, particularly in the analysis of functional magnetic resonance imaging (fMRI) data. This paper extends the use of ICA as a unifying framework for multimodal fusion, introducing a novel approach termed parallel multilink group joint ICA (pmg-jICA). The method allows for the fusion of gray matter maps from structural MRI (sMRI) data to multiple fMRI intrinsic networks, addressing the limitations of previous models. The effectiveness of pmg-jICA is demonstrated through its application to an Alzheimer’s dataset, yielding linked structure-function outputs for 53 brain networks. Our approach leverages the complementary information from various imaging modalities, providing a unique perspective on brain alterations in Alzheimer’s disease. The pmg-jICA identifies several components with significant differences between HC and AD groups including thalamus, caudate, putamen with in the subcortical (SC) domain, insula, parahippocampal gyrus within the cognitive control (CC) domain, and the lingual gyrus within the visual (VS) domain, providing localized insights into the links between AD and specific brain regions. In addition, because we link across multiple brain networks, we can also compute functional network connectivity (FNC) from spatial maps and subject loadings, providing a detailed exploration of the relationships between different brain regions and allowing us to visualize spatial patterns and loading parameters in sMRI along with intrinsic networks and FNC from the fMRI data. In essence, developed approach combines concepts from joint ICA and group ICA to provide a rich set of output characterizing data-driven links between covarying gray matter networks, and a (potentially large number of) resting fMRI networks allowing further study in the context of structure/function links. We demonstrate the utility of the approach by highlighting key structure/function disruptions in Alzheimer’s individuals.

    more » « less
  3. Abstract

    In this article, we focus on estimating the joint relationship between structural magnetic resonance imaging (sMRI) gray matter (GM), and multiple functional MRI (fMRI) intrinsic connectivity networks (ICNs). To achieve this, we propose a multilink joint independent component analysis (ml‐jICA) method using the same core algorithm as jICA. To relax the jICA assumption, we propose another extension called parallel multilink jICA (pml‐jICA) that allows for a more balanced weight distribution over ml‐jICA/jICA. We assume a shared mixing matrix for both the sMRI and fMRI modalities, while allowing for different mixing matrices linking the sMRI data to the different ICNs. We introduce the model and then apply this approach to study the differences in resting fMRI and sMRI data from patients with Alzheimer's disease (AD) versus controls. The results of the pml‐jICA yield significant differences with large effect sizes that include regions in overlapping portions of default mode network, and also hippocampus and thalamus. Importantly, we identify two joint components with partially overlapping regions which show opposite effects for AD versus controls, but were able to be separated due to being linked to distinct functional and structural patterns. This highlights the unique strength of our approach and multimodal fusion approaches generally in revealing potentially biomarkers of brain disorders that would likely be missed by a unimodal approach. These results represent the first work linking multiple fMRI ICNs to GM components within a multimodal data fusion model and challenges the typical view that brain structure is more sensitive to AD than fMRI.

    more » « less
  4. Abstract

    There are a growing number of neuroimaging studies motivating joint structural and functional brain connectivity. Brain connectivity of different modalities provides insight into brain functional organization by leveraging complementary information, especially for brain disorders such as schizophrenia. In this paper, we propose a multi-modal independent component analysis (ICA) model that utilizes information from both structural and functional brain connectivity guided by spatial maps to estimate intrinsic connectivity networks (ICNs). Structural connectivity is estimated through whole-brain tractography on diffusion-weighted MRI (dMRI), while functional connectivity is derived from resting-state functional MRI (rs-fMRI). The proposed structural-functional connectivity and spatially constrained ICA (sfCICA) model estimates ICNs at the subject level using a multi-objective optimization framework. We evaluated our model using synthetic and real datasets (including dMRI and rs-fMRI from 149 schizophrenia patients and 162 controls). Multi-modal ICNs revealed enhanced functional coupling between ICNs with higher structural connectivity, improved modularity, and network distinction, particularly in schizophrenia. Statistical analysis of group differences showed more significant differences in the proposed model compared to the unimodal model. In summary, the sfCICA model showed benefits from being jointly informed by structural and functional connectivity. These findings suggest advantages in simultaneously learning effectively and enhancing connectivity estimates using structural connectivity.

    more » « less
  5. Abstract

    Acquisition of multimodal brain imaging data for the same subject has become more common leading to a growing interest in determining the intermodal relationships between imaging modalities to further elucidate the pathophysiology of schizophrenia. Multimodal data have previously been individually analyzed and subsequently integrated; however, these analysis techniques lack the ability to examine true modality inter‐relationships. The utilization of a multiset canonical correlation and joint independent component analysis (mCCA + jICA) model for data fusion allows shared or distinct abnormalities between modalities to be examined. In this study, first‐episode schizophrenia patients (nSZ=19) and matched controls (nHC=21) completed a resting‐state functional magnetic resonance imaging (fMRI) scan at 7 T. Grey matter (GM), white matter (WM), cerebrospinal fluid (CSF), and amplitude of low frequency fluctuation (ALFF) maps were used as features in a mCCA + jICA model. Results of the mCCA + jICA model indicated three joint group‐discriminating components (GM‐CSF, WM‐ALFF, GM‐ALFF) and two modality‐unique group‐discriminating components (GM, WM). The joint component findings are highlighted by GM basal ganglia, somatosensory, parietal lobe, and thalamus abnormalities associated with ventricular CSF volume; WM occipital and frontal lobe abnormalities associated with temporal lobe function; and GM frontal, temporal, parietal, and occipital lobe abnormalities associated with caudate function. These results support and extend major findings throughout the literature using independent single modality analyses. The multimodal fusion of 7 T data in this study provides a more comprehensive illustration of the relationships between underlying neuronal abnormalities associated with schizophrenia than examination of imaging data independently.

    more » « less