Search for: All records

Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques. 

Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques. 

Abstract Changes in the viscosity of intracellular microenvironments may indicate the onset of diseases like diabetes, blood‐based illnesses, hypertension, and Alzheimer's. To date, monitoring viscosity changes in the intracellular environment remains a challenge with prior work focusing primarily on visible light‐absorbing viscosity sensing fluorophores. Herein, a series of near‐infrared (NIR, 700–1000 nm) absorbing and emitting indolizine squaraine fluorophores (1PhSQ,2PhSQ,SO₃SQ,1DMASQ,7DMASQ, and1,7DMASQ) are synthesized and studied for NIR viscosity sensitivity.2PhSQexhibits a very high slope in its Forster‐Hoffmann plot at 0.75 which indicates this dye is a potent viscosity sensor. The properties of the squaraine fluorophores are studied computationallyviadensity functional theory (DFT) and time‐dependent (TD)‐DFT. Experimentally, both steady‐state and time‐resolved emission spectroscopy, absorption spectroscopy, and electrochemical characterization are conducted on the dyes. Precise photophysical tuning is observed within the series with emission maxima wavelengths as long as 881 nm for1,7DMASQand fluorescence quantum yields as high as 39.5 and 72.0 % for1PhSQin DCM and THF, respectively. The high tunability of this molecular scaffold renders indolizine squaraine fluorophores excellent prospects as viscosity‐sensitive biological imaging agents with2PhSQgiving a dramatically higher fluorescence quantum yield (from 0.3 to 37.1 %) as viscosity increases. 

We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. In this challenging regime, 3D scene points are regularly observed only once, requiring prior-based reconstruction of scene geometry and appearance. We find that existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry and due to the high cost of differentiable rendering that precludes their scaling to large-scale training. We take a step towards resolving these shortcomings by formulating a multi-view transformer encoder, proposing an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray, and a lightweight cross-attention-based renderer. Our contributions enable training of our method on a large-scale real-world dataset of indoor and outdoor scenes. We demonstrate that our method learns powerful multi-view geometry priors while reducing both rendering time and memory footprint. We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis. 

Neural scene representations, both continuous and discrete, have recently emerged as a powerful new paradigm for 3D scene understanding. Recent efforts have tackled unsupervised discovery of object-centric neural scene representations. However, the high cost of ray-marching, exacerbated by the fact that each object representation has to be ray-marched separately, leads to insufficiently sampled radiance fields and thus, noisy renderings, poor framerates, and high memory and time complexity during training and rendering. Here, we propose to represent objects in an object-centric, compositional scene representation as light fields. We propose a novel light field compositor module that enables reconstructing the global light field from a set of object-centric light fields. Dubbed Compositional Object Light Fields (COLF), our method enables unsupervised learning of object-centric neural scene representations, state-of-the-art reconstruction and novel view synthesis performance on standard datasets, and rendering and training speeds at orders of magnitude faster than existing 3D approaches. 

Herein, we focus on the design, synthesis, and characterization of thienothiadiazole (TTD)-based near-infrared II (NIR-II) theranostic fluorophores and their nanoparticles. 

Fluorescent organic dyes that absorb and emit in the near-infrared (NIR, 700–1000 nm) and shortwave infrared (SWIR, 1000–1700 nm) regions have the potential to produce noninvasive high-contrast biological images and videos. BODIPY dyes are well known for their high quantum yields in the visible energy region. To tune these chromophores to the NIR region, fused nitrogen-based heterocyclic indolizine donors were added to a BODIPY scaffold. The indolizine BODIPY dyes were synthesized via microwave-assisted Knoevenagel condensation with indolizine aldehydes. The non-protonated dyes showed NIR absorption and emission at longer wavelengths than an aniline benchmark. Protonation of the dyes produced a dramatic 0.35 eV bathochromic shift (230 nm shift from 797 nm to 1027 nm) to give a SWIR absorption and emission (λmaxemis = 1061 nm). Deprotonation demonstrates that material emission is reversibly switchable between the NIR and SWIR. 

Abstract Small organic molecules absorbing and emitting in the shortwave infrared (SWIR, 1000–2000 nm) region are desirable for biological imaging applications due to low auto‐fluorescence, reduce photon scattering, and good tissue penetration depth of photons which allows forin vivoimaging with high resolution and sensitivity. Si‐substituted xanthene‐based fluorophores with indolizine donors have demonstrated some of the longest wavelengths of absorption and emission from organic dyes. This work seeks to compare an indolizine heterocyclic nitrogen with dimethyl aniline nitrogen donors on otherwise identical Si‐substituted xanthene fluorophoresviaoptical spectroscopy, computational chemistry and electrochemistry. Three donors are compared including an indolizine donor, a ubiquitous dimethyl aniline donor, and a vinyl dimethyl aniline group that keeps the number of π‐bonds consistent with indolizine. Significantly higher quantum yields and molar absorptivity are observed in these studies for a dimethylamine‐based donor relative to a simple indolizine donor absorbing and emitting at similar wavelengths (~1312 nm emission). Substantially longer wavelengths are obtainable by appending aniline‐based groups to the indolizine donor (~1700 nm) indicating longer wavelengths can be accessed with indolizine donors while stronger emitters can be accessed with anilines in place of indolizine.