skip to main content
Explore Scholarly Publications and Datasets in the NSF-PAR

Title: Generative imaging and image processing via generative encoder

This paper introduces a novel generative encoder (GE) framework for generative imaging and image processing tasks like image reconstruction, compression, denoising, inpainting, deblurring, and super-resolution. GE unifies the generative capacity of GANs and the stability of AEs in an optimization framework instead of stacking GANs and AEs into a single network or combining their loss functions as in existing literature. GE provides a novel approach to visualizing relationships between latent spaces and the data space. The GE framework is made up of a pre-training phase and a solving phase. In the former, a GAN with generator \begin{document}$ G $\end{document} capturing the data distribution of a given image set, and an AE network with encoder \begin{document}$ E $\end{document} that compresses images following the estimated distribution by \begin{document}$ G $\end{document} are trained separately, resulting in two latent representations of the data, denoted as the generative and encoding latent space respectively. In the solving phase, given noisy image \begin{document}$ x = \mathcal{P}(x^*) $\end{document}, where \begin{document}$ x^* $\end{document} is the target unknown image, \begin{document}$ \mathcal{P} $\end{document} is an operator adding an addictive, or multiplicative, or convolutional noise, or equivalently given such an image \begin{document}$ x $\end{document} more » in the compressed domain, i.e., given \begin{document}$ m = E(x) $\end{document}, the two latent spaces are unified via solving the optimization problem

and the image \begin{document}$ x^* $\end{document} is recovered in a generative way via \begin{document}$ \hat{x}: = G(z^*)\approx x^* $\end{document}, where \begin{document}$ \lambda>0 $\end{document} is a hyperparameter. The unification of the two spaces allows improved performance against corresponding GAN and AE networks while visualizing interesting properties in each latent space.

« less
Authors:
;
Award ID(s):
1945029
Publication Date:
NSF-PAR ID:
10336727
Journal Name:
Inverse Problems & Imaging
Volume:
16
Issue:
3
Page Range or eLocation-ID:
525
ISSN:
1930-8337
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ( , Kinetic and Related Models)

    Consider the linear transport equation in 1D under an external confining potential \begin{document}$ \Phi $\end{document}:

    For \begin{document}$ \Phi = \frac {x^2}2 + \frac { \varepsilon x^4}2 $\end{document} (with \begin{document}$ \varepsilon >0 $\end{document} small), we prove phase mixing and quantitative decay estimates for \begin{document}$ {\partial}_t \varphi : = - \Delta^{-1} \int_{ \mathbb{R}} {\partial}_t f \, \mathrm{d} v $\end{document}, with an inverse polynomial decay rate \begin{document}$ O({\langle} t{\rangle}^{-2}) $\end{document}. In the proof, we develop a commuting vector field approach, suitably adapted to this setting. We will explain why we hope this is relevant for the nonlinear stability of the zero solution for the Vlasov–Poisson system in \begin{document}$ 1 $\end{document}D under the external potential \begin{document}$ \Phi $\end{document}.

  2. ; ( , Numerical Algebra, Control and Optimization)

    We study the convergence rate of a continuous-time simulated annealing process \begin{document}$ (X_t; \, t \ge 0) $\end{document} for approximating the global optimum of a given function \begin{document}$ f $\end{document}. We prove that the tail probability \begin{document}$ \mathbb{P}(f(X_t) > \min f +\delta) $\end{document} decays polynomial in time with an appropriately chosen cooling schedule of temperature, and provide an explicit convergence rate through a non-asymptotic bound. Our argument applies recent development of the Eyring-Kramers law on functional inequalities for the Gibbs measure at low temperatures.

  3. ; ; ( , Discrete and Continuous Dynamical Systems - S)

    We consider optimal control of fractional in time (subdiffusive, i.e., for \begin{document}$ 0<\gamma <1 $\end{document}) semilinear parabolic PDEs associated with various notions of diffusion operators in an unifying fashion. Under general assumptions on the nonlinearity we \begin{document}$\mathsf{first\;show}$\end{document} the existence and regularity of solutions to the forward and the associated \begin{document}$\mathsf{backward\;(adjoint)}$\end{document} problems. In the second part, we prove existence of optimal \begin{document}$\mathsf{controls }$\end{document} and characterize the associated \begin{document}$\mathsf{first\;order}$\end{document} optimality conditions. Several examples involving fractional in time (and some fractional in space diffusion) equations are described in detail. The most challenging obstacle we overcome is the failure of the semigroup property for the semilinear problem in any scaling of (frequency-domain) Hilbert spaces.

  4. ; ; ( , Kinetic and Related Models)

    We establish existence of finite energy weak solutions to the kinetic Fokker-Planck equation and the linear Landau equation near Maxwellian, in the presence of specular reflection boundary condition for general domains. Moreover, by using a method of reflection and the \begin{document}$ S_p $\end{document} estimate of [7], we prove regularity in the kinetic Sobolev spaces \begin{document}$ S_p $\end{document} and anisotropic Hölder spaces for such weak solutions. Such \begin{document}$ S_p $\end{document} regularity leads to the uniqueness of weak solutions.

  5. ; ; ( , Inverse Problems & Imaging)

    In two dimensions, we consider the problem of inversion of the attenuated \begin{document}$ X $\end{document}-ray transform of a compactly supported function from data restricted to lines leaning on a given arc. We provide a method to reconstruct the function on the convex hull of this arc. The attenuation is assumed known. The method of proof uses the Hilbert transform associated with \begin{document}$ A $\end{document}-analytic functions in the sense of Bukhgeim.

  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email