skip to main content

Explore Research Products in the NSF-PAR

Title: Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
We present rectified flow, a surprisingly simple approach to learning (neural) ordinary differential equation (ODE) models to transport between two empirically observed distributions π0 and π1, hence providing a unified solution to generative modeling and domain transfer, among various other tasks involving distribution transport. The idea of rectified flow is to learn the ODE to follow the straight paths connecting the points drawn from π0 and π1 as much as possible. This is achieved by solving a straightforward nonlinear least squares optimization problem, which can be easily scaled to large models without introducing extra parameters beyond standard supervised learning. The straight paths are special and preferred because they are the shortest paths between two points, and can be simulated exactly without time discretization and hence yield computationally efficient models. We show that the procedure of learning a rectified flow from data, called rectification, turns an arbitrary coupling of π0 and π1 to a new deterministic coupling with provably non-increasing convex transport costs. In addition, recursively applying rectification allows us to obtain a sequence of flows with increasingly straight paths, which can be simulated accurately with coarse time discretization in the inference phase. In empirical studies, we show that rectified flow performs superbly on image generation, image-to-image translation, and domain adaptation. In particular, on image generation and translation, our method yields nearly straight flows that give high quality results even with a single Euler discretization step.  more » « less
Award ID(s):
1846421 2037267 2322879
NSF-PAR ID:
10440561
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
International conference on learning representations (ICLR)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , Proceedings of the AAAI Conference on Artificial Intelligence)
    null (Ed.)
    A normalizing flow is an invertible mapping between an arbitrary probability distribution and a standard normal distribution; it can be used for density estimation and statistical inference. Computing the flow follows the change of variables formula and thus requires invertibility of the mapping and an efficient way to compute the determinant of its Jacobian. To satisfy these requirements, normalizing flows typically consist of carefully chosen components. Continuous normalizing flows (CNFs) are mappings obtained by solving a neural ordinary differential equation (ODE). The neural ODE's dynamics can be chosen almost arbitrarily while ensuring invertibility. Moreover, the log-determinant of the flow's Jacobian can be obtained by integrating the trace of the dynamics' Jacobian along the flow. Our proposed OT-Flow approach tackles two critical computational challenges that limit a more widespread use of CNFs. First, OT-Flow leverages optimal transport (OT) theory to regularize the CNF and enforce straight trajectories that are easier to integrate. Second, OT-Flow features exact trace computation with time complexity equal to trace estimators used in existing CNFs. On five high-dimensional density estimation and generative modeling tasks, OT-Flow performs competitively to state-of-the-art CNFs while on average requiring one-fourth of the number of weights with an 8x speedup in training time and 24x speedup in inference. 
    more » « less
  2. ; ; ( , Mathematics of Computation)
    Boundary integral numerical methods are among the most accurate methods for interfacial Stokes flow, and are widely applied. They have the advantage that only the boundary of the domain must be discretized, which reduces the number of discretization points and allows the treatment of complicated interfaces. Despite their popularity, there is no analysis of the convergence of these methods for interfacial Stokes flow. In practice, the stability of discretizations of the boundary integral formulation can depend sensitively on details of the discretization and on the application of numerical filters. We present a convergence analysis of the boundary integral method for Stokes flow, focusing on a rather general method for computing the evolution of an elastic capsule or viscous drop in 2D strain and shear flows. The analysis clarifies the role of numerical filters in practical computations. 
    more » « less
  3. ; ; ; ; ; ( , ICML 2023)
    Simulation-free methods for training continuous-time generative models construct probability paths that go between noise distributions and individual data samples. Recent works, such as Flow Matching, derived paths that are optimal for each data sample. However, these algorithms rely on independent data and noise samples, and do not exploit underlying structure in the data distribution for constructing probability paths. We propose Multisample Flow Matching, a more general framework that uses non-trivial couplings between data and noise samples while satisfying the correct marginal constraints. At very small overhead costs, this generalization allows us to (i) reduce gradient variance during training, (ii) obtain straighter flows for the learned vector field, which allows us to generate high-quality samples using fewer function evaluations, and (iii) obtain transport maps with lower cost in high dimensions, which has applications beyond generative modeling. Importantly, we do so in a completely simulation-free manner with a simple minimization objective. We show that our proposed methods improve sample consistency on downsampled ImageNet data sets, and lead to better low-cost sample generation. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al ( , Nature Communications)
    Abstract

    Thermal rectification is an exotic thermal transport phenomenon which allows heat to transfer in one direction but block the other. We demonstrate an unusual dual-mode solid-state thermal rectification effect using a heterogeneous “irradiated-pristine” polyethylene nanofiber junction as a nanoscale thermal diode, in which heat flow can be rectified in both directions by changing the working temperature. For the nanofiber samples measured here, we observe a maximum thermal rectification factor as large as ~50%, which only requires a small temperature bias of <10 K. The tunable nanoscale thermal diodes with large rectification and narrow temperature bias open up new possibilities for developing advanced thermal management, energy conversion and, potentially thermophononic technologies.

     
    more » « less
  5. ; ; ( , Advances in Neural Information Processing Systems 33 (NeurIPS 2020))
    Detecting out-of-distribution (OOD) data is crucial for robust machine learning systems. Normalizing flows are flexible deep generative models that often surprisingly fail to distinguish between in- and out-of-distribution data: a flow trained on pictures of clothing assigns higher likelihood to handwritten digits. We investigate why normalizing flows perform poorly for OOD detection. We demonstrate that flows learn local pixel correlations and generic image-to-latent-space transformations which are not specific to the target image datasets, focusing on flows based on coupling layers. We show that by modifying the architecture of flow coupling layers we can bias the flow towards learning the semantic structure of the target data, improving OOD detection. Our investigation reveals that properties that enable flows to generate high-fidelity images can have a detrimental effect on OOD detection. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email