More Like this

1. Motion Adaptive Deblurring With Single-Photon Cameras

   Seets, Trevor; Ingle, Atul; Laurenzis, Martin; Velten, Andreas (January 2021, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV))

   null (Ed.)

   Single-photon avalanche diodes (SPADs) are a rapidly developing image sensing technology with extreme low-light sensitivity and picosecond timing resolution. These unique capabilities have enabled SPADs to be used in applications like LiDAR, non-line-of-sight imaging and fluorescence microscopy that require imaging in photon-starved scenarios. In this work we harness these capabilities for dealing with motion blur in a passive imaging setting in low illumination conditions. Our key insight is that the data captured by a SPAD array camera can be represented as a 3D spatio-temporal tensor of photon detection events which can be integrated along arbitrary spatio-temporal trajectories with dynamically varying integration windows, depending on scene motion. We propose an algorithm that estimates pixel motion from photon timestamp data and dynamically adapts the integration windows to minimize motion blur. Our simulation results show the applicability of this algorithm to a variety of motion profiles including translation, rotation and local object motion. We also demonstrate the real-world feasibility of our method on data captured using a 32x32 SPAD camera. 

   more » « less
2. Dual Infrared 2‐Photon Microscopy Achieves Minimal Background Deep Tissue Imaging in Brain and Plant Tissues

   https://doi.org/10.1002/adfm.202404709  

   Safaee, Mohammad M; Nishitani, Shoichi; McFarlane, Ian R; Yang, Sarah J; Sun, Ethan; Medina, Sebastiana M; Squire, Henry; Landry, Markita P (October 2024, Advanced Functional Materials)

   Abstract Traditional deep fluorescence imaging has primarily focused on red‐shifting imaging wavelengths into the near‐infrared (NIR) windows or implementation of multi‐photon excitation approaches. Here, the advantages of NIR and multiphoton imaging are combined by developing a dual‐infrared two‐photon microscope that enables high‐resolution deep imaging in biological tissues. This study first computationally identifies that photon absorption, as opposed to scattering, is the primary contributor to signal attenuation. A NIR two‐photon microscope is constructed next with a 1640 nm femtosecond pulsed laser and a NIR PMT detector to image biological tissues labeled with fluorescent single‐walled carbon nanotubes (SWNTs). Spatial imaging resolutions are achieved close to the Abbe resolution limit and eliminate blur and background autofluorescence of biomolecules, 300 µm deep into brain slices and through the full 120 µm thickness of aNicotiana benthamianaleaf. NIR‐II two‐photon microscopy can also measure tissue heterogeneity by quantifying how much the fluorescence power law function varies across tissues, a feature this study exploits to distinguish Huntington's Disease afflicted mouse brain tissues from wildtype. These results suggest dual‐infrared two‐photon microscopy can accomplish in‐tissue structural imaging and biochemical sensing with a minimal background, and with high spatial resolution, in optically opaque or highly autofluorescent biological tissues. 

   more » « less
3. Seeing Photons in Color

   https://doi.org/10.1145/3592438  

   Ma, Sizhuo; Sundar, Varun; Mos, Paul; Bruschini, Claudio; Charbon, Edoardo; Gupta, Mohit (August 2023, ACM Transactions on Graphics)

   Megapixel single-photon avalanche diode (SPAD) arrays have been developed recently, opening up the possibility of deploying SPADs as generalpurpose passive cameras for photography and computer vision. However, most previous work on SPADs has been limited to monochrome imaging. We propose a computational photography technique that reconstructs high-quality color images from mosaicked binary frames captured by a SPAD array, even for high-dyanamic-range (HDR) scenes with complex and rapid motion. Inspired by conventional burst photography approaches, we design algorithms that jointly denoise and demosaick single-photon image sequences. Based on the observation that motion effectively increases the color sample rate, we design a blue-noise pseudorandom RGBW color filter array for SPADs, which is tailored for imaging dark, dynamic scenes. Results on simulated data, as well as real data captured with a fabricated color SPAD hardware prototype shows that the proposed method can reconstruct high-quality images with minimal color artifacts even for challenging low-light, HDR and fast-moving scenes. We hope that this paper, by adding color to computational single-photon imaging, spurs rapid adoption of SPADs for real-world passive imaging applications. 

   more » « less
4. Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second

   https://doi.org/10.1038/s41467-021-26721-x  

   Nam, Ji_Hyun; Brandt, Eric; Bauer, Sebastian; Liu, Xiaochun; Renna, Marco; Tosi, Alberto; Sifakis, Eftychios; Velten, Andreas (November 2021, Nature Communications)

   Abstract Non-Line-Of-Sight (NLOS) imaging aims at recovering the 3D geometry of objects that are hidden from the direct line of sight. One major challenge with this technique is the weak available multibounce signal limiting scene size, capture speed, and reconstruction quality. To overcome this obstacle, we introduce a multipixel time-of-flight non-line-of-sight imaging method combining specifically designed Single Photon Avalanche Diode (SPAD) array detectors with a fast reconstruction algorithm that captures and reconstructs live low-latency videos of non-line-of-sight scenes with natural non-retroreflective objects. We develop a model of the signal-to-noise-ratio of non-line-of-sight imaging and use it to devise a method that reconstructs the scene such that signal-to-noise-ratio, motion blur, angular resolution, and depth resolution are all independent of scene depth suggesting that reconstruction of very large scenes may be possible. 

   more » « less
5. Mitigating Network Latency in Cloud-Based Teleoperation using Motion Segmentation and Synthesis

   Tian, Nan; Tanwani, Ajay Kumar; Goldberg, Ken; Sojoudi, Somayeh (October 2019, International Symposium on Robotics Research)

   null (Ed.)

   Network latency is a major problem in Cloud Robotics for human robot interactions such as teleoperation. Routing delays can be highly variable in a heterogeneous computing environment, imposing challenges to reliably teleoperate a robot with a closed-loop feedback controller. By sharing Gaussian Mixture Models (GMMs), Hidden Semi- Markov Models (HSMMs), and linear quadratic tracking (LQT) con- trollers between the cloud and the robot. We build a motion recognition, segmentation, and synthesis framework for Cloud Robotic teleoperation; and we introduce a set of latency mitigation network protocols under this framework. We use this framework in experiments with a dynamic robot arm to perform learned hand-written letter motions.We then study the motion recognition errors, motion synthesis errors, and the latency mitigation performance. 

   more » « less