An official website of the United States government
Deformable image registration (DIR) is an active research topic in biomedical imaging. There is a growing interest in developing DIR methods based on deep learning (DL). A traditional DL approach to DIR is based on training a convolutional neural network (CNN) to estimate the registration field between two input images. While conceptually simple, this approach comes with a limitation that it exclusively relies on a pre-trained CNN without explicitly enforcing fidelity between the registered image and the reference. We present plug-and-play image registration network (PIRATE) as a new DIR method that addresses this issue by integrating an explicit data-fidelity penalty and a CNN prior. PIRATE pre-trains a CNN denoiser on the registration field and "plugs" it into an iterative method as a regularizer. We additionally present PIRATE+ that fine-tunes the CNN prior in PIRATE using deep equilibrium models (DEQ). PIRATE+ interprets the fixed-point iteration of PIRATE as a network with effectively infinite layers and then trains the resulting network end-to-end, enabling it to learn more task-specific information and boosting its performance. Our numerical results on OASIS and CANDI datasets show that our methods achieve state-of-the-art performance on DIR.
more »
« less
Quasiconformal model with CNN features for large deformation image registration
Image registration has been widely studied over the past several decades, with numerous applications in science, engineering and medicine. Most of the conventional mathematical models for large deformation image registration rely on prescribed landmarks, which usually require tedious manual labeling. In recent years, there has been a surge of interest in the use of machine learning for image registration. In this paper, we develop a novel method for large deformation image registration by a fusion of quasiconformal theory and convolutional neural network (CNN). More specifically, we propose a quasiconformal energy model with a novel fidelity term that incorporates the features extracted using a pre-trained CNN, thereby allowing us to obtain meaningful registration results without any guidance of prescribed landmarks. Moreover, unlike many prior image registration methods, the bijectivity of our method is guaranteed by quasiconformal theory. Experimental results are presented to demonstrate the effectiveness of the proposed method. More broadly, our work sheds light on how rigorous mathematical theories and practical machine learning approaches can be integrated for developing computational methods with improved performance.
more »
« less
- Award ID(s):
- 2002103
- PAR ID:
- 10470094
- Publisher / Repository:
- AIMS
- Date Published:
- Journal Name:
- Inverse Problems and Imaging
- Volume:
- 16
- Issue:
- 4
- ISSN:
- 1930-8337
- Page Range / eLocation ID:
- 1019
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract An unsupervised machine learning method is introduced to align medical images in the context of the large deformation elasticity coupled with growth and remodeling biophysics. The technique, which stems from the principle of minimum potential energy in solid mechanics, consists of two steps: Firstly, in the predictor step, the geometric registration is achieved by minimizing a loss function composed of a dissimilarity measure and a regularizing term. Secondly, the physics of the problem, including the equilibrium equations along with growth mechanics, are enforced in a corrector step by minimizing the potential energy corresponding to a Dirichlet problem, where the predictor solution defines the boundary condition and is maintained by distance functions. The features of the new solution procedure, as well as the nature of the registration problem, are highlighted by considering several examples. In particular, registration problems containing large non-uniform deformations caused by extension, shearing, and bending of multiply-connected regions are used as benchmarks. In addition, we analyzed a benchmark biological example (registration for brain data) to showcase that the new deep learning method competes with available methods in the literature. We then applied the method to various datasets. First, we analyze the regrowth of the zebrafish embryonic fin from confocal imaging data. Next, we evaluate the quality of the solution procedure for two examples related to the brain. For one, we apply the new method for 3D image registration of longitudinal magnetic resonance images of the brain to assess cerebral atrophy, where a first-order ODE describes the volume loss mechanism. For the other, we explore cortical expansion during early fetal brain development by coupling the elastic deformation with morphogenetic growth dynamics. The method and examples show the ability of our framework to attain high-quality registration and, concurrently, solve large deformation elasticity balance equations and growth and remodeling dynamics.more » « less
-
G., Renata Raidou; Sommer, Björn; W., Torsten Kuhlen; Krone, Michael; Schultz, Thomas; Wu, Hsiang-Yun (Ed.)We introduce a novel multi-modal 3D image registration framework based on 3D user-guided deformation of both volume's shape and intensity values. Being able to apply deformations in 3D gives access to a wide new range of interactions allowing for the registration of images from any acquisition method and of any organ, complete or partial. Our framework uses a state of the art 3D volume rendering method for real-time feedback on the registration accuracy as well as the image deformation. We propose a novel methodological variation to accurately display 3D segmented voxel grids, which is a requirement in a registration context for visualizing a segmented atlas. Our pipeline is implemented in an open-source software (available via GitHub) and was directly used by biologists for registration of mouse brain model autofluorescence acquisition on the Allen Brain Atlas. The latter mapping allows them to retrieve regions of interest properly identified on the segmented atlas in acquired brain datasets and therefore extract only high-resolution images of those areas, avoiding the creation of images too large to be processed.more » « less
-
Abstract This paper presents a novel application of convolutional neural network (CNN) models for filtering the intraseasonal variability of the tropical atmosphere. In this deep learning filter, two convolutional layers are applied sequentially in a supervised machine learning framework to extract the intraseasonal signal from the total daily anomalies. The CNN-based filter can be tailored for each field similarly to fast Fourier transform filtering methods. When applied to two different fields (zonal wind stress and outgoing longwave radiation), the index of agreement between the filtered signal obtained using the CNN-based filter and a conventional weight-based filter is between 95% and 99%. The advantage of the CNN-based filter over the conventional filters is its applicability to time series with the length comparable to the period of the signal being extracted. Significance StatementThis study proposes a new method for discovering hidden connections in data representative of tropical atmosphere variability. The method makes use of an artificial intelligence (AI) algorithm that combines a mathematical operation known as convolution with a mathematical model built to reflect the behavior of the human brain known as artificial neural network. Our results show that the filtered data produced by the AI-based method are consistent with the results obtained using conventional mathematical algorithms. The advantage of the AI-based method is that it can be applied to cases for which the conventional methods have limitations, such as forecast (hindcast) data or real-time monitoring of tropical variability in the 20–100-day range.more » « less
-
This paper presents a novel predictive model, MetaMorph, for metamorphic registration of images with appearance changes (i.e., caused by brain tumors). In contrast to previous learning-based registration methods that have little or no control over appearance-changes, our model introduces a new regularization that can effectively suppress the negative effects of appearance changing areas. In particular, we develop a piecewise regularization on the tangent space of diffeomorphic transformations (also known as initial velocity fields) via learned segmentation maps of abnormal regions. The geometric transformation and appearance changes are treated as joint tasks that are mutually beneficial. Our model MetaMorph is more robust and accurate when searching for an optimal registration solution under the guidance of segmentation, which in turn improves the segmentation performance by providing appropriately augmented training labels. We validate MetaMorph on real 3D human brain tumor magnetic resonance imaging (MRI) scans. Experimental results show that our model outperforms the state-of-the-art learning-based registration models. The proposed MetaMorph has great potential in various image-guided clinical interventions, e.g., real-time image-guided navigation systems for tumor removal surgery.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3934/ipi.2022010
