skip to main content

Explore Research Products in the NSF-PAR

Title: Generative imaging and image processing via generative encoder

This paper introduces a novel generative encoder (GE) framework for generative imaging and image processing tasks like image reconstruction, compression, denoising, inpainting, deblurring, and super-resolution. GE unifies the generative capacity of GANs and the stability of AEs in an optimization framework instead of stacking GANs and AEs into a single network or combining their loss functions as in existing literature. GE provides a novel approach to visualizing relationships between latent spaces and the data space. The GE framework is made up of a pre-training phase and a solving phase. In the former, a GAN with generator \begin{document}$ G $\end{document} capturing the data distribution of a given image set, and an AE network with encoder \begin{document}$ E $\end{document} that compresses images following the estimated distribution by \begin{document}$ G $\end{document} are trained separately, resulting in two latent representations of the data, denoted as the generative and encoding latent space respectively. In the solving phase, given noisy image \begin{document}$ x = \mathcal{P}(x^*) $\end{document}, where \begin{document}$ x^* $\end{document} is the target unknown image, \begin{document}$ \mathcal{P} $\end{document} is an operator adding an addictive, or multiplicative, or convolutional noise, or equivalently given such an image \begin{document}$ x $\end{document} in the compressed domain, i.e., given \begin{document}$ m = E(x) $\end{document}, the two latent spaces are unified via solving the optimization problem

and the image \begin{document}$ x^* $\end{document} is recovered in a generative way via \begin{document}$ \hat{x}: = G(z^*)\approx x^* $\end{document}, where \begin{document}$ \lambda>0 $\end{document} is a hyperparameter. The unification of the two spaces allows improved performance against corresponding GAN and AE networks while visualizing interesting properties in each latent space.

  more » « less
Award ID(s):
1945029 2244988
NSF-PAR ID:
10336727
Author(s) / Creator(s):
;
Date Published:
Journal Name:
Inverse Problems & Imaging
Volume:
16
Issue:
3
ISSN:
1930-8337
Page Range / eLocation ID:
525
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ( , Kinetic and Related Models)

    Consider the linear transport equation in 1D under an external confining potential \begin{document}$ \Phi $\end{document}:

    For \begin{document}$ \Phi = \frac {x^2}2 + \frac { \varepsilon x^4}2 $\end{document} (with \begin{document}$ \varepsilon >0 $\end{document} small), we prove phase mixing and quantitative decay estimates for \begin{document}$ {\partial}_t \varphi : = - \Delta^{-1} \int_{ \mathbb{R}} {\partial}_t f \, \mathrm{d} v $\end{document}, with an inverse polynomial decay rate \begin{document}$ O({\langle} t{\rangle}^{-2}) $\end{document}. In the proof, we develop a commuting vector field approach, suitably adapted to this setting. We will explain why we hope this is relevant for the nonlinear stability of the zero solution for the Vlasov–Poisson system in \begin{document}$ 1 $\end{document}D under the external potential \begin{document}$ \Phi $\end{document}.

     
    more » « less
  2. ; ( , Evolution Equations and Control Theory)

    We consider the linear third order (in time) PDE known as the SMGTJ-equation, defined on a bounded domain, under the action of either Dirichlet or Neumann boundary control \begin{document}$ g $\end{document}. Optimal interior and boundary regularity results were given in [1], after [41], when \begin{document}$ g \in L^2(0, T;L^2(\Gamma)) \equiv L^2(\Sigma) $\end{document}, which, moreover, in the canonical case \begin{document}$ \gamma = 0 $\end{document}, were expressed by the well-known explicit representation formulae of the wave equation in terms of cosine/sine operators [19], [17], [24,Vol â…ˇ]. The interior or boundary regularity theory is however the same, whether \begin{document}$ \gamma = 0 $\end{document} or \begin{document}$ 0 \neq \gamma \in L^{\infty}(\Omega) $\end{document}, since \begin{document}$ \gamma \neq 0 $\end{document} is responsible only for lower order terms. Here we exploit such cosine operator based-explicit representation formulae to provide optimal interior and boundary regularity results with \begin{document}$ g $\end{document} "smoother" than \begin{document}$ L^2(\Sigma) $\end{document}, qualitatively by one unit, two units, etc. in the Dirichlet boundary case. To this end, we invoke the corresponding results for wave equations, as in [17]. Similarly for the Neumann boundary case, by invoking the corresponding results for the wave equation as in [22], [23], [37] for control smoother than \begin{document}$ L^2(0, T;L^2(\Gamma)) $\end{document}, and [44] for control less regular in space than \begin{document}$ L^2(\Gamma) $\end{document}. In addition, we provide optimal interior and boundary regularity results when the SMGTJ equation is subject to interior point control, by invoking the corresponding wave equations results [42], [24,Section 9.8.2].

     
    more » « less
  3. ; ( , Numerical Algebra, Control and Optimization)

    We study the convergence rate of a continuous-time simulated annealing process \begin{document}$ (X_t; \, t \ge 0) $\end{document} for approximating the global optimum of a given function \begin{document}$ f $\end{document}. We prove that the tail probability \begin{document}$ \mathbb{P}(f(X_t) > \min f +\delta) $\end{document} decays polynomial in time with an appropriately chosen cooling schedule of temperature, and provide an explicit convergence rate through a non-asymptotic bound. Our argument applies recent development of the Eyring-Kramers law on functional inequalities for the Gibbs measure at low temperatures.

     
    more » « less
  4. ; ; ; ( , Discrete and Continuous Dynamical Systems - S)

    It is shown that for any positive integer \begin{document}$ n \ge 3 $\end{document}, there is a stable irreducible \begin{document}$ n\times n $\end{document} matrix \begin{document}$ A $\end{document} with \begin{document}$ 2n+1-\lfloor\frac{n}{3}\rfloor $\end{document} nonzero entries exhibiting Turing instability. Moreover, when \begin{document}$ n = 3 $\end{document}, the result is best possible, i.e., every \begin{document}$ 3\times 3 $\end{document} stable matrix with five or fewer nonzero entries will not exhibit Turing instability. Furthermore, we determine all possible \begin{document}$ 3\times 3 $\end{document} irreducible sign pattern matrices with 6 nonzero entries which can be realized by a matrix \begin{document}$ A $\end{document} that exhibits Turing instability.

     
    more » « less
  5. ; ; ( , Journal of Modern Dynamics)

    Let \begin{document}$ f_c(z) = z^2+c $\end{document} for \begin{document}$ c \in {\mathbb C} $\end{document}. We show there exists a uniform upper bound on the number of points in \begin{document}$ {\mathbb P}^1( {\mathbb C}) $\end{document} that can be preperiodic for both \begin{document}$ f_{c_1} $\end{document} and \begin{document}$ f_{c_2} $\end{document}, for any pair \begin{document}$ c_1\not = c_2 $\end{document} in \begin{document}$ {\mathbb C} $\end{document}. The proof combines arithmetic ingredients with complex-analytic: we estimate an adelic energy pairing when the parameters lie in \begin{document}$ \overline{\mathbb{Q}} $\end{document}, building on the quantitative arithmetic equidistribution theorem of Favre and Rivera-Letelier, and we use distortion theorems in complex analysis to control the size of the intersection of distinct Julia sets. The proofs are effective, and we provide explicit constants for each of the results.

     
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email