In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set
of under-sampled and noisy measurements. Deep learning approaches have been
proven to be successful in solving this ill-posed inverse problem and are capable of
producing very high quality reconstructions. However, current architectures heavily
rely on convolutions, that are content-independent and have difficulties modeling
long-range dependencies in images. Recently, Transformers, the workhorse of
contemporary natural language processing, have emerged as powerful building
blocks for a multitude of vision tasks. These models split input images into nonoverlapping
patches, embed the patches into lower-dimensional tokens and utilize
a self-attention mechanism that does not suffer from the aforementioned weaknesses
of convolutional architectures. However, Transformers incur extremely
high compute and memory cost when 1) the input image resolution is high and 2)
when the image needs to be split into a large number of patches to preserve fine
detail information, both of which are typical in low-level vision problems such
as MRI reconstruction, having a compounding effect. To tackle these challenges,
we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit
bias and efficiency of convolutions with the power of Transformer blocks
in an unrolled and multi-scale network. HUMUS-Net extracts high-resolution
features via convolutional blocks and refines low-resolution features via a novel
Transformer-based multi-scale feature extractor. Features from both levels are then
synthesized into a high-resolution output reconstruction. Our network establishes
new state of the art on the largest publicly available MRI dataset, the fastMRI
dataset. We further demonstrate the performance of HUMUS-Net on two other
popular MRI datasets and perform fine-grained ablation studies to validate our
design.
more »
« less
Efficient Diffeomorphic Image Registration using Multi-Scale Dual-Phased Learning
Diffeomorphic registration faces challenges for high dimensional images, especially in terms of memory limits. Existing approaches either downsample/crop original images or approximate underlying transformations to reduce the model size. To mitigate this, we propose a Dividing and Down-sampling mixed Registration network (DDR-Net), a general architecture that preserves most of the image information at multiple scales while reducing memory cost. DDR-Net leverages the global context via downsampling the input and utilizes local details by dividing the input images to subvolumes. Such design fuses global and local information and obtains both coarse- and fine-level alignments in the final deformation fields. We apply DDR-Net to the OASIS dataset. The proposed simple yet effective architecture is a general method and could be extended to other registration architectures for better performance with limited computing resources.
more »
« less
- Award ID(s):
- 1755970
- NSF-PAR ID:
- 10350886
- Date Published:
- Journal Name:
- IEEE 19th International Symposium on Biomedical Imaging (ISBI)
- Page Range / eLocation ID:
- 1 to 5
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
As scaling of conventional memory devices has stalled, many high-end computing systems have begun to incorporate alternative memory technologies to meet performance goals. Since these technologies present distinct advantages and tradeoffs compared to conventional DDR* SDRAM, such as higher bandwidth with lower capacity or vice versa, they are typically packaged alongside conventional SDRAM in a heterogeneous memory architecture. To utilize the different types of memory efficiently, new data management strategies are needed to match application usage to the best available memory technology. However, current proposals for managing heterogeneous memories are limited, because they either (1) do not consider high-level application behavior when assigning data to different types of memory or (2) require separate program execution (with a representative input) to collect information about how the application uses memory resources. This work presents a new data management toolset to address the limitations of existing approaches for managing complex memories. It extends the application runtime layer with automated monitoring and management routines that assign application data to the best tier of memory based on previous usage, without any need for source code modification or a separate profiling run. It evaluates this approach on a state-of-the-art server platform with both conventional DDR4 SDRAM and non-volatile Intel Optane DC memory, using both memory-intensive high-performance computing (HPC) applications as well as standard benchmarks. Overall, the results show that this approach improves program performance significantly compared to a standard unguided approach across a variety of workloads and system configurations. The HPC applications exhibit the largest benefits, with speedups ranging from 1.4× to 7× in the best cases. Additionally, we show that this approach achieves similar performance as a comparable offline profiling-based approach after a short startup period, without requiring separate program execution or offline analysis steps.more » « less
-
Cardiac Cine Magnetic Resonance (CMR) Imaging has made a significant paradigm shift in medical imaging technology, thanks to its capability of acquiring high spatial and temporal resolution images of different structures within the heart that can be used for reconstructing patient-specific ventricular computational models. In this work, we describe the development of dynamic patient-specific right ventricle (RV) models associated with normal subjects and abnormal RV patients to be subsequently used to assess RV function based on motion and kinematic analysis. We first constructed static RV models using segmentation masks of cardiac chambers generated from our accurate, memory-efficient deep neural architecture - CondenseUNet - featuring both a learned group structure and a regularized weight-pruner to estimate the motion of the right ventricle. In our study, we use a deep learning-based deformable network that takes 3D input volumes and outputs a motion field which is then used to generate isosurface meshes of the cardiac geometry at all cardiac frames by propagating the end-diastole (ED) isosurface mesh using the reconstructed motion field. The proposed model was trained and tested on the Automated Cardiac Diagnosis Challenge (ACDC) dataset featuring 150 cine cardiac MRI patient datasets. The isosurface meshes generated using the proposed pipeline were compared to those obtained using motion propagation via traditional non-rigid registration based on several performance metrics, including Dice score and mean absolute distance (MAD).more » « less
-
The architectures of many neural networks rely heavily on the underlying grid associated with the variables, for instance, the lattice of pixels in an image. For general biomedical data without a grid structure, the multi‐layer perceptron (MLP) and deep belief network (DBN) are often used. However, in these networks, variables are treated homogeneously in the sense of network structure; and it is difficult to assess their individual importance. In this paper, we propose a novel neural network called Variable‐block tree Net (VtNet) whose architecture is determined by an underlying tree with each node corresponding to a subset of variables. The tree is learned from the data to best capture the causal relationships among the variables. VtNet contains a long short‐term memory (LSTM)‐like cell for every tree node. The input and forget gates of each cell control the information flow through the node, and they are used to define a significance score for the variables. To validate the defined significance score, VtNet is trained using smaller trees with variables of low scores removed. Hypothesis tests are conducted to show that variables of higher scores influence classification more strongly. Comparison is made with the variable importance score defined in Random Forest from the aspect of variable selection. Our experiments demonstrate that VtNet is highly competitive in classification accuracy and can often improve accuracy by removing variables with low significance scores.more » « less
-
Multi-instance learning (MIL) has demonstrated its usefulness in many real-world image applications in recent years. However, two critical challenges prevent one from effectively using MIL in practice. First, existing MIL methods routinely model the predictive targets using the instances of input images, but rarely utilize an input image as a whole. As a result, the useful information conveyed by the holistic representation of an input image could be potentially lost. Second, the varied numbers of the instances of the input images in a data set make it infeasible to use traditional learning models that can only deal with single-vector inputs. To tackle these two challenges, in this paper we propose a novel image representation learning method that can integrate the local patches (the instances) of an input image (the bag) and its holistic representation into one single-vector representation. Our new method first learns a projection to preserve both global and local consistencies of the instances of an input image. It then projects the holistic representation of the same image into the learned subspace for information enrichment. Taking into account the content and characterization variations in natural scenes and photos, we develop an objective that maximizes the ratio of the summations of a number of L1 -norm distances, which is difficult to solve in general. To solve our objective, we derive a new efficient non-greedy iterative algorithm and rigorously prove its convergence. Promising results in extensive experiments have demonstrated improved performances of our new method that validate its effectiveness.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ISBI52829.2022.9761693
