An official website of the United States government
Brain large-scale dynamics is constrained by the heterogeneity of intrinsic anatomical substrate. Little is known how the spatiotemporal dynamics adapt for the heterogeneous structural connectivity (SC). Modern neuroimaging modalities make it possible to study the intrinsic brain activity at the scale of seconds to minutes. Diffusion magnetic resonance imaging (dMRI) and functional MRI reveals the large-scale SC across different brain regions. Electrophysiological methods (i.e. MEG/EEG) provide direct measures of neural activity and exhibits complex neurobiological temporal dynamics which could not be solved by fMRI. However, most of existing multimodal analytical methods collapse the brain measurements either in space or time domain and fail to capture the spatio-temporal circuit dynamics. In this paper, we propose a novel spatio-temporal graph Transformer model to integrate the structural and functional connectivity in both spatial and temporal domain. The proposed method learns the heterogeneous node and graph representation via contrastive learning and multi-head attention based graph Transformer using multimodal brain data (i.e. fMRI, MRI, MEG and behavior performance). The proposed contrastive graph Transformer representation model incorporates the heterogeneity map constrained by T1-to-T2-weighted (T1w/T2w) to improve the model fit to structurefunction interactions. The experimental results with multimodal resting state brain measurements demonstrate the proposed method could highlight the local properties of large-scale brain spatio-temporal dynamics and capture the dependence strength between functional connectivity and behaviors. In summary, the proposed method enables the complex brain dynamics explanation for different modal variants.
more »
« less
This content will become publicly available on April 1, 2026
DSAM: A deep learning framework for analyzing temporal and spatial dynamics in brain networks
Resting-state functional magnetic resonance imaging (rs-fMRI) is a noninvasive technique pivotal for understanding human neural mechanisms of intricate cognitive processes. Most rs-fMRI studies compute a single static functional connectivity matrix across brain regions of interest, or dynamic functional connectivity matrices with a sliding window approach. These approaches are at risk of oversimplifying brain dynamics and lack proper consideration of the goal at hand. While deep learning has gained substantial popularity for modeling complex relational data, its application to uncovering the spatiotemporal dynamics of the brain is still limited. In this study we propose a novel interpretable deep learning framework that learns goal-specific functional connectivity matrix directly from time series and employs a specialized graph neural network for the final classification. Our model, DSAM, leverages temporal causal convolutional networks to capture the temporal dynamics in both low- and high-level feature representations, a temporal attention unit to identify important time points, a self-attention unit to construct the goal-specific connectivity matrix, and a novel variant of graph neural network to capture the spatial dynamics for downstream classification. To validate our approach, we conducted experiments on the Human Connectome Project dataset with 1075 samples to build and interpret the model for the classification of sex group, and the Adolescent Brain Cognitive Development Dataset with 8520 samples for independent testing. Compared our proposed framework with other state-of-art models, results suggested this novel approach goes beyond the assumption of a fixed connectivity matrix, and provides evidence of goal-specific brain connectivity patterns, which opens up potential to gain deeper insights into how the human brain adapts its functional connectivity specific to the task at hand.
more »
« less
- Award ID(s):
- 2112455
- PAR ID:
- 10569590
- Publisher / Repository:
- Elsevier
- Date Published:
- Journal Name:
- Medical Image Analysis
- Volume:
- 101
- Issue:
- C
- ISSN:
- 1361-8415
- Page Range / eLocation ID:
- 103462
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Identifying Structural Brain Networks from Functional Connectivity: A Network Deconvolution ApproachWe address the problem of identifying structural brain networks from signals measured by resting-state functional magnetic resonance imaging (fMRI). To this end, we model functional brain activity as graph signals generated through a linear diffusion process on the unknown structural network. While this is admittedly an oversimplification of the complex mechanisms at work in the brain, recent studies have shown it is an accurate generative model for the second-order statistics of functional signals. We show the diffusion model implies that the signal covariance matrix (a.k.a. functional connectivity) is an unknown polynomial function of the structural network’s adjacency matrix. Accordingly, we advocate a network deconvolution approach whereby we: (i) use the fMRI signals to estimate the eigenvectors of the structural network from those of the empirical covariance; and (ii) solve a convex, sparsity-regularized inverse problem to recover the eigenvalues that were obscured by diffusion. The inferred structural networks capture some key patterns that match known pathology in attention deficit/hyper activity disorder. We also offer preliminary evidence supporting their role as potential biomarkers for subject diagnosis and classification.more » « less
-
Abstract In neuroimaging research, understanding the intricate dynamics of brain networks over time is paramount for unraveling the complexities of brain function. One approach commonly used to explore the dynamic nature of brain networks is functional connectivity analysis. However, while functional connectivity offers valuable insights, it fails to consider the diverse timescales of coupling between different brain regions. This gap in understanding leaves a significant aspect of brain dynamics unexplored in neuroimaging research. We propose an innovative approach that delves into the dynamic coupling/connectivity timescales of brain regions relative to one another, focusing on how brain region couplings stretch or shrink over time, rather than relying solely on functional connectivity measures. Our method introduces a novel metric called “warping elasticity,” which utilizes dynamic time warping (DTW) to capture the temporal nuances of connectivity. Unlike traditional methods, our approach allows for (potentially nonlinear) dynamic compression and expansion of the time series, offering a more intricate understanding of how coupling between brain regions evolves. Through the adaptive windows employed by the DTW method, we can effectively capture transient couplings within varying connectivity timescales of brain network pairs. In extensive evaluations, our method exhibits high replicability across subjects and diverse datasets, showcasing robustness against noise. More importantly, it uncovers statistically significant distinctions between healthy control (HC) and schizophrenia (SZ) groups through the identification of warp elasticity states. These states are cluster centroids, representing the warp elasticity across subjects and time, offering a novel perspective on the dynamic nature of brain connectivity, distinct from conventional metrics focused solely on functional connectivity. For instance, controls spend more time in a warp elasticity state characterized by timescale stretching of the visual domain relative to other domains, suggesting disruptions in the visual cortex. Conversely, patients show increased time spent in a warp elasticity state with stretching timescales in higher cognitive areas relative to sensory regions, indicative of prolonged cognitive processing of sensory input. Overall, our approach presents a promising avenue for investigating the temporal dynamics of brain network interactions in functional magnetic resonance imaging (fMRI) data. By focusing on the elasticity of connectivity timescales, rather than adhering to functional connectivity metrics, we pave the way for a deeper understanding of neuropsychiatric disorders in neuroscience research.more » « less
-
Patterns of estimated neural activity derived from resting state functional magnetic resonance imaging (rs-fMRI) have been shown to predict a wide range of cognitive and behavioral outcomes in both normative and clinical populations. Yet, without links to established cognitive processes, the functional brain states associated with the resting brain will remain unexplained, and potentially confounded, markers of individual differences. In this work we demonstrate the application of multivoxel pattern classifiers (MVPCs) to predict the valence and arousal properties of spontaneous affect processing in the task-non-engaged resting state. rs-fMRI data were acquired from subjects that were held out from a subject set that underwent image-based affect induction concurrent with fMRI to train the MVPCs. We also validated these affective predictions against a well-established, independent measure of autonomic arousal, skin conductance response. These findings suggest a new neuroimaging methodology for resting state analysis in which models, trained on cognition-specific task-based fMRI acquired from well-matched cohorts, capably predict hidden cognitive processes operating within the resting brain.more » « less
-
One of the most significant health issues the world is now ex- periencing is mental health. Since it is challenging to mea- sure, it fails to motivate behavioral change or other inter- ventions targeted at improving it. The lack of such mea- surements discourages learning and behavior modification, which are critical for enhancing healthy habits. Neuroimag- ing promises to provide a window into mental health. The use of resting fMRI (rs-fMRI) models capable of categorizing mental health at the individual level has the potential to pro- vide insights into how it impacts the brain. Here, we applied a deep learning approach to classify a mental health score using static functional network connectivity (FNC) derived from rs-fMRI data. Comparisons were made against traditional machine learning approaches to evaluate the model’s perfor- mance. The discriminative features present in each mental health category were analyzed for interpreting the deep learn- ing model. The experiments resulted in a classification ac- curacy of 91%, 91%, 100%, and 100% for excellent, good, fair, and poor mental health classes. Results also highlighted the most salient brain network in the sFNC matrix for each mental health score classification.more » « less
