Search for: All records

Visual aliasing, or doppelgangers, poses severe challenges to 3D reconstruction. We propose Doppelganger++, an enhanced pairwise image classifier that excels in visual disambiguation across diverse and challenging scenes. We seamlessly integrate Doppelganger++ into SfM, successfully disambiguating each scene. (Middle) Compared to prior work (which we refer to as DG-OG), Doppelgangers++ is more robust for everyday scenes, showing improved accuracy and robustness. We show pairs that DG-OG classifies incorrectly and ours gets correct. Our new VisymScenes dataset, featuring complex daily scenes, is particularly challenging for COLMAP and DG-OG, but our method can achieve correct and complete reconstructions. 

We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods—from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps)—addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). 

Many animals exhibit structural colors, which are often iridescent, meaning that the perceived colors change with illumination conditions and viewing perspectives. Biological iridescence is usually caused by multilayers or other periodic structures in animal tissues, which selectively reflect light of certain wavelengths and often result in a shiny appearance---which almost always comes with spatially varying highlights, thanks to randomness and irregularities in the structures. Previous models for biological iridescence tend to each target one specific structure, and most models only compute large-area averages, overlooking spatial variation in iridescent appearance. In this work, we build appearance models for biological iridescence using bird feathers as our case study, investigating different types of feathers with a variety of structural coloration mechanisms. We propose an approximate wave simulation method that takes advantage of quasi-regular structures while efficiently modeling the effects of natural structural irregularities. We further propose a method to distill our simulation results into distributions of BRDFs, generated using noise functions, that preserve relevant statistical properties of the simulated BRDFs. This allows us to model the spatially varying, glittery appearance commonly seen on feathers. Our BRDFs are practical and efficient, and we present renderings of multiple types of iridescent feathers with comparisons to photographic images. 

This paper presents a method that uses neural networks as a caching mechanism to reduce the variance of Monte Carlo Partial Differential Equation solvers, such as the Walk-on-Spheres algorithm [Sawhney and Crane 2020]. While these Monte Carlo PDE solvers have the merits of being unbiased and discretization-free, their high variance often hinders real-time applications. On the other hand, neural networks can approximate the PDE solution, and evaluating these networks at inference time can be very fast. However, neural-network-based solutions may suffer from convergence difficulties and high bias. Our hybrid system aims to combine these two potentially complementary solutions by training a neural field to approximate the PDE solution using supervision from a WoS solver. This neural field is then used as a cache in the WoS solver to reduce variance during inference. We demonstrate that our neural field training procedure is better than the commonly used self-supervised objectives in the literature. We also show that our hybrid solver exhibits lower variance than WoS with the same computational budget: it is significantly better for small compute budgets and provides smaller improvements for larger budgets, reaching the same performance as WoS in the limit.