Search for: All records

We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird’s-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns to complete geometry and appearance of occluded regions. In addition, we show that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The ability to separately reconstruct movable objects enables a variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as a backbone for efficient 3D scene understanding models. 

High-resolution simulations can deliver great visual quality, but they are often limited by available memory, especially on GPUs. We present a compiler for physical simulation that can achieve both high performance and significantly reduced memory costs, by enabling flexible and aggressive quantization. Low-precision ("quantized") numerical data types are used and packed to represent simulation states, leading to reduced memory space and bandwidth consumption. Quantized simulation allows higher resolution simulation with less memory, which is especially attractive on GPUs. Implementing a quantized simulator that has high performance and packs the data tightly for aggressive storage reduction would be extremely labor-intensive and error-prone using a traditional programming language. To make the creation of quantized simulation practical, we have developed a new set of language abstractions and a compilation system. A suite of tailored domain-specific optimizations ensure quantized simulators often run as fast as the full-precision simulators, despite the overhead of encoding-decoding the packed quantized data types. Our programming language and compiler, based on Taichi , allow developers to effortlessly switch between different full-precision and quantized simulators, to explore the full design space of quantization schemes, and ultimately to achieve a good balance between space and precision. The creation of quantized simulation with our system has large benefits in terms of memory consumption and performance, on a variety of hardware, from mobile devices to workstations with high-end GPUs. We can simulate with levels of resolution that were previously only achievable on systems with much more memory, such as multiple GPUs. For example, on a single GPU, we can simulate a Game of Life with 20 billion cells (8× compression per pixel), an Eulerian fluid system with 421 million active voxels (1.6× compression per voxel), and a hybrid Eulerian-Lagrangian elastic object simulation with 235 million particles (1.7× compression per particle). At the same time, quantized simulations create physically plausible results. Our quantization techniques are complementary to existing acceleration approaches of physical simulation: they can be used in combination with these existing approaches, such as sparse data structures, for even higher scalability and performance. 

Abstract Interpretation of resolved polarized images of black holes by the Event Horizon Telescope (EHT) requires predictions of the polarized emission observable by an Earth-based instrument for a particular model of the black hole accretion system. Such predictions are generated by general relativistic radiative transfer (GRRT) codes, which integrate the equations of polarized radiative transfer in curved spacetime. A selection of ray-tracing GRRT codes used within the EHT Collaboration is evaluated for accuracy and consistency in producing a selection of test images, demonstrating that the various methods and implementations of radiative transfer calculations are highly consistent. When imaging an analytic accretion model, we find that all codes produce images similar within a pixel-wise normalized mean squared error (NMSE) of 0.012 in the worst case. When imaging a snapshot from a cell-based magnetohydrodynamic simulation, we find all test images to be similar within NMSEs of 0.02, 0.04, 0.04, and 0.12 in Stokes I , Q , U , and V , respectively. We additionally find the values of several image metrics relevant to published EHT results to be in agreement to much better precision than measurement uncertainties. 

Abstract We report on the observations of the quasar NRAO 530 with the Event Horizon Telescope (EHT) on 2017 April 5−7, when NRAO 530 was used as a calibrator for the EHT observations of Sagittarius A*. At z = 0.902, this is the most distant object imaged by the EHT so far. We reconstruct the first images of the source at 230 GHz, at an unprecedented angular resolution of ∼20 μ as, both in total intensity and in linear polarization (LP). We do not detect source variability, allowing us to represent the whole data set with static images. The images reveal a bright feature located on the southern end of the jet, which we associate with the core. The feature is linearly polarized, with a fractional polarization of ∼5%–8%, and it has a substructure consisting of two components. Their observed brightness temperature suggests that the energy density of the jet is dominated by the magnetic field. The jet extends over 60 μ as along a position angle ∼ −28°. It includes two features with orthogonal directions of polarization (electric vector position angle), parallel and perpendicular to the jet axis, consistent with a helical structure of the magnetic field in the jet. The outermost feature has a particularly high degree of LP, suggestive of a nearly uniform magnetic field. Future EHT observations will probe the variability of the jet structure on microarcsecond scales, while simultaneous multiwavelength monitoring will provide insight into the high-energy emission origin. 

Abstract The blazar J1924–2914 is a primary Event Horizon Telescope (EHT) calibrator for the Galactic center’s black hole Sagittarius A*. Here we present the first total and linearly polarized intensity images of this source obtained with the unprecedented 20 μ as resolution of the EHT. J1924–2914 is a very compact flat-spectrum radio source with strong optical variability and polarization. In April 2017 the source was observed quasi-simultaneously with the EHT (April 5–11), the Global Millimeter VLBI Array (April 3), and the Very Long Baseline Array (April 28), giving a novel view of the source at four observing frequencies, 230, 86, 8.7, and 2.3 GHz. These observations probe jet properties from the subparsec to 100 pc scales. We combine the multifrequency images of J1924–2914 to study the source morphology. We find that the jet exhibits a characteristic bending, with a gradual clockwise rotation of the jet projected position angle of about 90° between 2.3 and 230 GHz. Linearly polarized intensity images of J1924–2914 with the extremely fine resolution of the EHT provide evidence for ordered toroidal magnetic fields in the blazar compact core.