An official website of the United States government
Image restoration aims to recover a clean image given a noisy image. It has long been a topic of interest for researchers in imaging, optical science and computer vision. As the imaging environment becomes more and more deteriorated, the problem becomes more challenging. Several computational approaches, ranging from statistical to deep learning, have been proposed over the years to tackle this problem. The deep learning-based approaches provided promising image restoration results, but it’s purely data driven and the requirement of large datasets (paired or unpaired) for training might demean its utility for certain physical problems. Recently, physics informed image restoration techniques have gained importance due to their ability to enhance performance, infer some sense of the degradation process and its potential to quantify the uncertainty in the prediction results. In this paper, we propose a physics informed deep learning approach with simultaneous parameter estimation using 3D integral imaging and Bayesian neural network (BNN). An image-image mapping architecture is first pretrained to generate a clean image from the degraded image, which is then utilized for simultaneous training with Bayesian neural network for simultaneous parameter estimation. For the network training, simulated data using the physical model has been utilized instead of actual degraded data. The proposed approach has been tested experimentally under degradations such as low illumination and partial occlusion. The recovery results are promising despite training from a simulated dataset. We have tested the performance of the approach under varying levels of illumination condition. Additionally, the proposed approach also has been analyzed against corresponding 2D imaging-based approach. The results suggest significant improvements compared to 2D even training under similar datasets. Also, the parameter estimation results demonstrate the utility of the approach in estimating the degradation parameter in addition to image restoration under the experimental conditions considered.
more »
« less
Three-dimensional integral imaging-based image descattering and recovery using physics informed unsupervised CycleGAN
Image restoration and denoising has been a challenging problem in optics and computer vision. There has been active research in the optics and imaging communities to develop a robust, data-efficient system for image restoration tasks. Recently, physics-informed deep learning has received wide interest in scientific problems. In this paper, we introduce a three-dimensional integral imaging-based physics-informed unsupervised CycleGAN algorithm for underwater image descattering and recovery using physics-informed CycleGAN (Generative Adversarial Network). The system consists of a forward and backward pass. The base architecture consists of an encoder and a decoder. The encoder takes the clean image along with the depth map and the degradation parameters to produce the degraded image. The decoder takes the degraded image generated by the encoder along with the depth map and produces the clean image along with the degradation parameters. In order to provide physical significance for the input degradation parameter w.r.t a physical model for the degradation, we also incorporated the physical model into the loss function. The proposed model has been assessed under the dataset curated through underwater experiments at various levels of turbidity. In addition to recovering the original image from the degraded image, the proposed algorithm also helps to model the distribution under which the degraded images have been sampled. Furthermore, the proposed three-dimensional Integral Imaging approach is compared with the traditional deep learning-based approach and 2D imaging approach under turbid and partially occluded environments. The results suggest the proposed approach is promising, especially under the above experimental conditions.
more »
« less
- Award ID(s):
- 2141473
- PAR ID:
- 10484086
- Publisher / Repository:
- Optical Society of America
- Date Published:
- Journal Name:
- Optics Express
- Volume:
- 32
- Issue:
- 2
- ISSN:
- 1094-4087; OPEXFF
- Format(s):
- Medium: X Size: Article No. 1825
- Size(s):
- Article No. 1825
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In this paper, we present a polarimetric image restoration approach that aims to recover the Stokes parameters and the degree of linear polarization from their corresponding degraded counterparts. The Stokes parameters and the degree of linear polarization are affected due to the degradations present in partial occlusion or turbid media, such as scattering, attenuation, and turbid water. The polarimetric image restoration with corresponding Mueller matrix estimation is performed using polarization-informed deep learning and 3D Integral imaging. An unsupervised image-to-image translation (UNIT) framework is utilized to obtain clean Stokes parameters from the degraded ones. Additionally, a multi-output convolutional neural network (CNN) based branch is used to predict the Mueller matrix estimate along with an estimate of the corresponding residue. The degree of linear polarization with the Mueller matrix estimate generates information regarding the characteristics of the underlying transmission media and the object under consideration. The approach has been evaluated under different environmentally degraded conditions, such as various levels of turbidity and partial occlusion. The 3D integral imaging reduces the effects of degradations in a turbid medium. The performance comparison between 3D and 2D imaging in varying scene conditions is provided. Experimental results suggest that the proposed approach is promising under the scene degradations considered. To the best of our knowledge, this is the first report on polarization-informed deep learning in 3D imaging, which attempts to recover the polarimetric information along with the corresponding Mueller matrix estimate in a degraded environment.more » « less
-
The transmission properties of multi-degree-of-freedom mechanisms can be tuned by shaping velocity ellipses throughout their workspace. Velocity ellipses are the image of a circle in the actuator velocity space mapped by the Jacobian into end-effector velocities. In this work, two machine learning methods using convolutional neural network architectures are proposed to synthesize planar 2R mechanism designs that approximately produce the desired velocity ellipses. An ensemble of image-based regression models is trained in a supervised fashion to output multiple 2R designs that approximate the specified ellipses. As an alternative to this approach, a second physics-informed neural network is constructed to train an ensemble of encoder models without specifying the 2R link lengths. During training, a decoder model that approximates the kinematics (physics) of the 2R is used to find how well the 2R design output by the encoder approximates the specified ellipses. These models are used to obtain multiple 2R designs that produce ellipses suitable for legged locomotion tasks and some preliminary results are presented.more » « less
-
Stefan Harmeling (Ed.)Discretization invariant learning aims at learning in the infinite-dimensional function spaces with the capacity to process heterogeneous discrete representations of functions as inputs and/or outputs of a learning model. This paper proposes a novel deep learning framework based on integral autoencoders (IAE-Net) for discretization invariant learning. The basic building block of IAE-Net consists of an encoder and a decoder as integral transforms with data-driven kernels, and a fully connected neural network between the encoder and decoder. This basic building block is applied in parallel in a wide multi-channel structure, which is repeatedly composed to form a deep and densely connected neural network with skip connections as IAE-Net. IAE-Net is trained with randomized data augmentation that generates training data with heterogeneous structures to facilitate the performance of discretization invariant learning. The proposed IAE-Net is tested with various applications in predictive data science, solving forward and inverse problems in scientific computing, and signal/image processing. Compared with alternatives in the literature, IAE-Net achieves state-of-the-art performance in existingmore » « less
-
It is generally assumed that oceanic effects, such as absorption, scattering, and turbulence, deteriorate underwater optical imaging and/or signal detection. In this paper, we present an interesting observation that slight turbidity may actually improve the performance of underwater optical imaging in the presence of occlusion. We have carried out simulations and optical experiments in underwater degraded environments to investigate this hypothesis. For simulation, the Monte Carlo method was used to analyze the influence of imaging performance under varying turbidity and occlusion conditions. Additionally, optical experiments were conducted under various turbid and partially occluded environments. We considered the effects of different parameters such as varying turbidity levels, severity of partial occlusion, number of photons, propagation distances, and imaging modality. The simulation results we performed suggest that, regardless of the variation of the imaging system and degradation parameters, slight turbidity may improve underwater imaging performance in occlusion. The optical experimental results are also in agreement with the simulation results that slightly increasing the turbidity levels may boost the image quality in the scenarios we considered. To the best of our knowledge, this is the first report to theoretically analyze and experimentally validate the phenomenon that turbidity may improve underwater imaging performance in certain degraded environments.more » « less
