More Like this

1. Unifying radiative transfer models in computer graphics and remote sensing, Part I: A survey

   https://doi.org/10.1016/j.jqsrt.2023.108847  

   Salesin, Katherine; Knobelspiesse, Kirk D; Chowdhary, Jacek; Zhai, Peng-Wang; Jarosz, Wojciech (February 2024, Journal of Quantitative Spectroscopy and Radiative Transfer)

   The constellation of Earth-observing satellites continuously collects measurements of scattered radiance, which must be transformed into geophysical parameters in order to answer fundamental scientific questions about the Earth. Retrieval of these parameters requires highly flexible, accurate, and fast forward and inverse radiative transfer models. Existing forward models used by the remote sensing community are typically accurate and fast, but sacrifice flexibility by assuming the atmosphere or ocean is composed of plane-parallel layers. Monte Carlo forward models can handle more complex scenarios such as 3D spatial heterogeneity, but are relatively slower. We propose looking to the computer graphics community for inspiration to improve the statistical efficiency of Monte Carlo forward models and explore new approaches to inverse models for remote sensing. In Part 1 of this work, we examine the evolution of radiative transfer models in computer graphics and highlight recent advancements that have the potential to push forward models in remote sensing beyond their current periphery of realism. 

   more » « less
2. Differential Walk on Spheres

   Miller, Bailey; Sawhney, Rohan; Crane, Keenan; Gkioulekas, Ioannis (December 2024, ACM Transactions on Graphics)

   We introduce a Monte Carlo method for computing derivatives of the solution to a partial differential equation (PDE) with respect to problem parameters (such as domain geometry or boundary conditions). Derivatives can be evaluated at arbitrary points, without performing a global solve or constructing a volumetric grid or mesh. The method is hence well suited to inverse problems with complex geometry, such as PDE-constrained shape optimization. Like other walk on spheres (WoS) algorithms, our method is trivial to parallelize, and is agnostic to boundary representation (meshes, splines, implicit surfaces, etc.), supporting large topological changes. We focus in particular on screened Poisson equations, which model diverse problems from scientific and geometric computing. As in differentiable rendering, we jointly estimate derivatives with respect to all parameters—hence, cost does not grow significantly with parameter count. In practice, even noisy derivative estimates exhibit fast, stable convergence for stochastic gradient-based optimization, as we show through examples from thermal design, shape from diffusion, and computer graphics. 

   more » « less
3. Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

   Tewari, Ayush; Yin, Tianwei; Cazenavette, George; Rezchikov, Semon; Tenenbaum, Joshua B; Durand, Frédo; Freeman, William T; Sitzmann, Vincent (December 2023, Neural Information Processing Systems)

   Denoising diffusion models have emerged as a powerful class of generative models capable of capturing the distributions of complex, real-world signals. However, current approaches can only model distributions for which training samples are directly accessible, which is not the case in many real-world tasks. In inverse graphics, for instance, we seek to sample from a distribution over 3D scenes consistent with an image but do not have access to ground-truth 3D scenes, only 2D images. We present a new class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never observed directly, but instead are only measured through a known differentiable forward model that generates partial observations of the unknown signal. To accomplish this, we directly integrate the forward model into the denoising process. At test time, our approach enables us to sample from the distribution over underlying signals consistent with some partial observation. We demonstrate the efficacy of our approach on three challenging computer vision tasks. For instance, in inverse graphics, we demonstrate that our model enables us to directly sample from the distribution 3D scenes consistent with a single 2D input image. 

   more » « less
4. Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

   Tewari, Ayush; Yin, Tianwei; Cazenavette, George; Rezchikov, Semon; Tenenbaum, Joshua B; Durand, Frédo; Freeman, William T; Sitzmann, Vincent (December 2023, Neural Information Processing Systems)

   Denoising diffusion models have emerged as a powerful class of generative models capable of capturing the distributions of complex, real-world signals. However, current approaches can only model distributions for which training samples are directly accessible, which is not the case in many real-world tasks. In inverse graphics, for instance, we seek to sample from a distribution over 3D scenes consistent with an image but do not have access to ground-truth 3D scenes, only 2D images. We present a new class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never observed directly, but instead are only measured through a known differentiable forward model that generates partial observations of the unknown signal. To accomplish this, we directly integrate the forward model into the denoising process. At test time, our approach enables us to sample from the distribution over underlying signals consistent with some partial observation. We demonstrate the efficacy of our approach on three challenging computer vision tasks. For instance, in inverse graphics, we demonstrate that our model enables us to directly sample from the distribution 3D scenes consistent with a single 2D input image. 

   more » « less
5. Differential Walk on Spheres

   https://doi.org/10.1145/3687913  

   Miller, Bailey; Sawhney, Rohan; Crane, Keenan; Gkioulekas, Ioannis (December 2024, ACM Transactions on Graphics)

   We introduce a Monte Carlo method for computing derivatives of the solution to a partial differential equation (PDE) with respect to problem parameters (such as domain geometry or boundary conditions). Derivatives can be evaluated at arbitrary points, without performing a global solve or constructing a volumetric grid or mesh. The method is hence well suited to inverse problems with complex geometry, such as PDE-constrained shape optimization. Like otherwalk on spheres (WoS)algorithms, our method is trivial to parallelize, and is agnostic to boundary representation (meshes, splines, implicit surfaces,etc.), supporting large topological changes. We focus in particular on screened Poisson equations, which model diverse problems from scientific and geometric computing. As in differentiable rendering, we jointly estimate derivatives with respect to all parameters---hence, cost does not grow significantly with parameter count. In practice, even noisy derivative estimates exhibit fast, stable convergence for stochastic gradient-based optimization, as we show through examples from thermal design, shape from diffusion, and computer graphics. 

   more » « less