An official website of the United States government
3D image reconstruction from a limited number of 2D images has been a long-standing challenge in computer vision and image analysis. While deep learning-based approaches have achieved impressive performance in this area, existing deep networks often fail to effectively utilize the shape structures of objects presented in images. As a result, the topology of reconstructed objects may not be well preserved, leading to the presence of artifacts such as discontinuities, holes, or mismatched connections between different parts. In this paper, we propose a shape-aware network based on diffusion models for 3D image reconstruction, named SADIR, to address these issues. In contrast to previous methods that primarily rely on spatial correlations of image intensities for 3D reconstruction, our model leverages shape priors learned from the training data to guide the reconstruction process. To achieve this, we develop a joint learning network that simultaneously learns a mean shape under deformation models. Each reconstructed image is then considered as a deformed variant of the mean shape. We validate our model, SADIR, on both brain and cardiac magnetic resonance images (MRIs). Experimental results show that our method outperforms the baselines with lower reconstruction error and better preservation of the shape structure of objects within the images.
more »
« less
A tailor-made 3-dimensional directional Haar semi-tight framelet for pMRI reconstruction
In this paper, we propose a model for parallel magnetic resonance imaging (pMRI) reconstruction, regularized by a carefully designed tight framelet system, that can lead to reconstructed images with much less artifacts in comparison to those from existing models. Our model is motivated from the observations that each receiver coil in a pMRI system is more sensitive to the specific object nearest to the coil, and all coil images are correlated. To exploit these observations, we first stack all coil images together as a 3-dimensional (3D) data matrix, and then design a 3D directional Haar tight framelet (3DHTF) to represent it. After analyzing sparse information of the coil images provided by the high-pass filters of the 3DHTF, we separate the high-pass filters into effective ones and ineffective ones, and we then devise a 3D directional Haar semi-tight framelet (3DHSTF) from the 3DHTF by replacing its ineffective filters with only one filter. This 3DHSTF is tailor-made for coil images, meanwhile, giving a significant saving in computation comparing to the 3DHTF. With the 3DHSTF, we propose an l1-3DHSTF model for pMRI reconstruction. Numerical experiments for MRI phantom and in-vivo data sets are provided to demonstrate the superiority of our l1-3DHSTF model in terms of the efficiency of reducing aliasing artifacts in the reconstructed images.
more »
« less
- Award ID(s):
- 1913039
- PAR ID:
- 10327244
- Date Published:
- Journal Name:
- Applied and computational harmonic analysis
- Volume:
- 60
- ISSN:
- 1096-603X
- Page Range / eLocation ID:
- 446-470
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
PurposeTo develop a physics‐guided deep learning (PG‐DL) reconstruction strategy based on a signal intensity informed multi‐coil (SIIM) encoding operator for highly‐accelerated simultaneous multislice (SMS) myocardial perfusion cardiac MRI (CMR). MethodsFirst‐pass perfusion CMR acquires highly‐accelerated images with dynamically varying signal intensity/SNR following the administration of a gadolinium‐based contrast agent. Thus, using PG‐DL reconstruction with a conventional multi‐coil encoding operator leads to analogous signal intensity variations across different time‐frames at the network output, creating difficulties in generalization for varying SNR levels. We propose to use a SIIM encoding operator to capture the signal intensity/SNR variations across time‐frames in a reformulated encoding operator. This leads to a more uniform/flat contrast at the output of the PG‐DL network, facilitating generalizability across time‐frames. PG‐DL reconstruction with the proposed SIIM encoding operator is compared to PG‐DL with conventional encoding operator, split slice‐GRAPPA, locally low‐rank (LLR) regularized reconstruction, low‐rank plus sparse (L + S) reconstruction, and regularized ROCK‐SPIRiT. ResultsResults on highly accelerated free‐breathing first pass myocardial perfusion CMR at three‐fold SMS and four‐fold in‐plane acceleration show that the proposed method improves upon the reconstruction methods use for comparison. Substantial noise reduction is achieved compared to split slice‐GRAPPA, and aliasing artifacts reduction compared to LLR regularized reconstruction, L + S reconstruction and PG‐DL with conventional encoding. Furthermore, a qualitative reader study indicated that proposed method outperformed all methods. ConclusionPG‐DL reconstruction with the proposed SIIM encoding operator improves generalization across different time‐frames /SNRs in highly accelerated perfusion CMR.more » « less
-
Context. Asymptotic giant branch (AGB) stars are cool luminous evolved stars that are well observable across the Galaxy and populating Gaia data. They have complex stellar surface dynamics, which amplifies the uncertainties on stellar parameters and distances. Aims. On the AGB star CL Lac, it has been shown that the convection-related variability accounts for a substantial part of the Gaia DR2 parallax error. We observed this star with the MIRC-X beam combiner installed at the CHARA interferometer to detect the presence of stellar surface inhomogeneities. Methods. We performed the reconstruction of aperture synthesis images from the interferometric observations at different wavelengths. Then, we used 3D radiative hydrodynamics (RHD) simulations of stellar convection with CO5BOLD and the post-processing radiative transfer code O PTIM 3D to compute intensity maps in the spectral channels of MIRC-X observations. Then, we determined the stellar radius using the average 3D intensity profile and, finally, compared the 3D synthetic maps to the reconstructed ones focusing on matching the intensity contrast, the morphology of stellar surface structures, and the photocentre position at two different spectral channels, 1.52 and 1.70 μ m, simultaneously. Results. We measured the apparent diameter of CL Lac at two wavelengths (3.299 ± 0.005 mas and 3.053 ± 0.006 mas at 1.52 and 1.70 μ m, respectively) and recovered the radius ( R = 307 ± 41 and R = 284 ± 38 R ⊙ ) using a Gaia parallax. In addition to this, the reconstructed images are characterised by the presence of a brighter area that largely affects the position of the photocentre. The comparison with 3D simulation shows good agreement with the observations both in terms of contrast and surface structure morphology, meaning that our model is adequate for explaining the observed inhomogenities. Conclusions. This work confirms the presence of convection-related surface structures on an AGB star of Gaia DR2. Our result will help us to take a step forward in exploiting Gaia measurement uncertainties to extract the fundamental properties of AGB stars using appropriate RHD simulations.more » « less
-
Abstract We present a feature-selective segmentation and merging technique to achieve spatially resolved surface profiles of the parts by 3D stereoscopy and strobo-stereoscopy. A pair of vision cameras capture images of the parts at different angles, and 3D stereoscopic images can be reconstructed. Conventional filtering processes of the 3D images involve data loss and lower the spatial resolution of the image. In this study, the 3D reconstructed image was spatially resolved by automatically recognizing and segmenting the features on the raw images, locally and adaptively applying super-resolution algorithm to the segmented images based on the classified features, and then merging those filtered segments. Here, the features are transformed into masks that selectively separate the features and background images for segmentation. The experimental results were compared with those of conventional filtering methods by using Gaussian filters and bandpass filters in terms of spatial frequency and profile accuracy. As a result, the selective feature segmentation technique was capable of spatially resolved 3D stereoscopic imaging while preserving imaging features.more » « less
-
3D Point Cloud Data (PCD) is an efficient machine representation for surrounding environments and has been used in many applications. But the measured PCD is often incomplete and sparse due to the sensor occlusion and poor lighting conditions. To automatically reconstruct complete PCD from the incomplete ones, we propose DeepPCD, a deep-learning-based system that reconstructs both geometric and color information for large indoor environments. For geometric reconstruction, DeepPCD uses a novel patch based technique that splits the PCD into multiple parts, approximates, extends, and independently reconstructs the parts by 3D planes, and then merges and refines them. For color reconstruction, DeepPCD uses a conditional Generative Adversarial Network to infer the missing color of the geometrically reconstructed PCD by using the color feature extracted from incomplete color PCD. We experimentally evaluate DeepPCD with several real PCD collected from large, diverse indoor environments and explore the feasibility of PCD autocompletion in various ubiquitous sensing applications.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/https://doi.org/10.1016/j.acha.2022.04.003
