An official website of the United States government
Video Frame Interpolation aims to recover realistic missing frames between observed frames, generating a highframe- rate video from a low-frame-rate video. However, without additional guidance, the large motion between frames makes this problem ill-posed. Event-based Video Frame Interpolation (EVFI) addresses this challenge by using sparse, high-temporal-resolution event measurements as motion guidance. This guidance allows EVFI methods to significantly outperform frame-only methods. However, to date, EVFI methods have relied on a limited set of paired eventframe training data, severely limiting their performance and generalization capabilities. In this work, we overcome the limited data challenge by adapting pre-trained video diffusion models trained on internet-scale datasets to EVFI. We experimentally validate our approach on real-world EVFI datasets, including a new one that we introduce. Our method outperforms existing methods and generalizes across cameras far better than existing approaches.
more »
« less
Unpaired Downscaling of Fluid Flows with Diffusion Bridges
Abstract We present a method to downscale idealized geophysical fluid simulations using generative models based on diffusion maps. By analyzing the Fourier spectra of fields drawn from different data distributions, we show how a diffusion bridge can be used as a transformation between a low resolution and a high resolution dataset, allowing for new sample generation of high-resolution fields given specific low resolution features. The ability to generate new samples allows for the computation of any statistic of interest, without any additional calibration or training. Our unsupervised setup is also designed to downscale fields without access to paired training data; this flexibility allows for the combination of multiple source and target domains without additional training. We demonstrate that the method enhances resolution and corrects context-dependent biases in geophysical fluid simulations, including in extreme events. We anticipate that the same method can be used to downscale the output of climate simulations, including temperature and precipitation fields, without needing to train a new model for each application and providing a significant computational cost savings.
more »
« less
- Award ID(s):
- 1835860
- PAR ID:
- 10547947
- Publisher / Repository:
- American Meteorological Society
- Date Published:
- Journal Name:
- Artificial Intelligence for the Earth Systems
- Volume:
- 3
- Issue:
- 2
- ISSN:
- 2769-7525
- Page Range / eLocation ID:
- e230039
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We introduce a new family of physics-inspired generative models termed PFGM++ that unifies diffusion models and Poisson Flow Generative Models (PFGM). These models realize generative trajectories for N dimensional data by embedding paths in N+D dimensional space while still controlling the progression with a simple scalar norm of the D additional variables. The new models reduce to PFGM when D=1 and to diffusion models when D→∞. The flexibility of choosing D allows us to trade off robustness against rigidity as increasing D results in more concentrated coupling between the data and the additional variable norms. We dispense with the biased large batch field targets used in PFGM and instead provide an unbiased perturbation-based objective similar to diffusion models. To explore different choices of D, we provide a direct alignment method for transferring well-tuned hyperparameters from diffusion models (D→∞) to any finite D values. Our experiments show that models with finite D can be superior to previous state-of-the-art diffusion models on CIFAR-10/FFHQ 64×64 datasets/LSUN Churches 256×256, with median Ds. In class-conditional setting, D=2048 yields current state-of-the-art FID of 1.74 on CIFAR-10 without additional training. Furthermore, we demonstrate that models with smaller D exhibit improved robustness against modeling errors. Code is available at https://github.com/Newbeeer/pfgmppmore » « less
-
Abstract Several new methods are proposed that can diagnose the interscale transfer (or spectral flux) of kinetic energy (KE) and other properties in oceanic and broader geophysical systems, using integrals of advective structure functions and Bessel functions (herein “Bessel methods”). The utility of the Bessel methods is evaluated using simulations of anisotropic flow within two-dimensional (2D), surface quasigeostrophic (SQG), and two-layer QG systems. The Bessel methods diagnose various spectral fluxes within all of these systems, even under strong anisotropy and complex dynamics (e.g., multiple cascaded variables, coincident and opposing spectral fluxes, and nonstationary systems). In 2D turbulence, the Bessel methods capture the inverse KE cascade at large scales and the downscale enstrophy cascade (and associated downscale energy flux) at small scales. In SQG turbulence, the Bessel methods capture the downscale buoyancy variance cascade and the coincident upscale wavenumber-dependent KE flux. In QG turbulence, the Bessel methods capture the upscale kinetic energy flux. It is shown that these Bessel methods can be applied to data with limited extent or resolution, provided the scales of interest are captured by the range of separation distances. The Bessel methods are shown to have several advantages over other flux-estimation methods, including the ability to diagnose downscale energy cascades and to identify sharp transition scales. Analogous Bessel methods are also discussed for third-order structure functions, along with some caveats due to boundary terms. Significance StatementBig ocean eddies play an important role in Earth’s energy cycle by moving energy to both larger and smaller scales, but it is difficult to measure these “eddy energy fluxes” from oceanic observations. We develop a new method to estimate eddy energy fluxes that utilizes spatial differences between pairs of points and can be applied to various ocean data. This new method accurately diagnoses key eddy energy flux properties, as we demonstrate using idealized numerical simulations of various large-scale ocean systems.more » « less
-
Abstract Extreme winds associated with tropical cyclones (TCs) can cause significant loss of life and economic damage globally, highlighting the need for accurate, high‐resolution modeling and forecasting for wind. However, due to their coarse horizontal resolution, most global climate and weather models suffer from chronic underprediction of TC wind speeds, limiting their use for impact analysis and energy modeling. In this study, we introduce a cascading deep learning framework designed to downscale high‐resolution TC wind fields given low‐resolution data. Our approach maps 85 TC events from ERA5 data (0.25° resolution) to high‐resolution (0.05° resolution) observations at 6‐hr intervals. The initial component is a debiasing neural network designed to model accurate wind speed observations using ERA5 data. The second component employs a generative super‐resolution strategy based on a conditional denoising diffusion probabilistic model (DDPM) to enhance the spatial resolution and to produce ensemble estimates. The model is able to accurately model intensity and produce realistic radial profiles and fine‐scale spatial structures of wind fields, with a percentage mean bias of −3.74% compared to the high‐resolution observations. Our downscaling framework enables the prediction of high‐resolution wind fields using widely available low‐resolution and intensity wind data, allowing for the modeling of past events and the assessment of future TC risks.more » « less
-
Abstract The rise of exascale supercomputing has motivated an increase in high‐fidelity computational fluid dynamics (CFD) simulations. The detail in these simulations, often involving shape‐dependent, time‐variant flow domains and low‐speed, complex, turbulent flows, is essential for fueling innovations in fields like wind, civil, automotive, or aerospace engineering. However, the massive amount of data these simulations produce can overwhelm storage systems and negatively affect conventional data management and postprocessing workflows, including iterative procedures such as design space exploration, optimization, and uncertainty quantification. This study proposes a novel sampling method harnessing the signed distance function (SDF) concept: SDF‐biased flow importance sampling (BiFIS) and implicit compression based on implicit neural network representations for transforming large‐size, shape‐dependent flow fields into reduced‐size shape‐agnostic images. Designed to alleviate the above‐mentioned problems, our approach achieves near‐lossless compression ratios of approximately :, reducing the size of a bridge aerodynamics forced‐vibration simulation from roughly to about while maintaining low reproduction errors, in most cases below , which is unachievable with other sampling approaches. Our approach also allows for real‐time analysis and visualization of these massive simulations and does not involve decompression preprocessing steps that yield full simulation data again. Given that image sampling is a fundamental step for any image‐based flow field prediction model, the proposed BiFIS method can significantly improve the accuracy and efficiency of such models, helping any application that relies on precise flow field predictions. The BiFIS code is available onGitHub.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1175/AIES-D-23-0039.1
