skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Passive Inter-Photon Imaging
Digital camera pixels measure image intensities by converting incident light energy into an analog electrical current, and then digitizing it into a fixed-width binary representation. This direct measurement method, while conceptually simple, suffers from limited dynamic range and poor performance under extreme illumination — electronic noise dominates under low illumination, and pixel full-well capacity results in saturation under bright illumination. We propose a novel intensity cue based on measuring inter-photon timing, defined as the time delay between detection of successive photons. Based on the statistics of inter-photon times measured by a time-resolved single-photon sensor, we develop theory and algorithms for a scene brightness estimator which works over extreme dynamic range; we experimentally demonstrate imaging scenes with a dynamic range of over ten million to one. The proposed techniques, aided by the emergence of single-photon sensors such as single-photon avalanche diodes (SPADs) with picosecond timing resolution, will have implications for a wide range of imaging applications: robotics, consumer photography, astronomy, microscopy and biomedical imaging.  more » « less
Award ID(s):
1846884 1943149
PAR ID:
10314158
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV))
    null (Ed.)
    Single-photon avalanche diodes (SPADs) are a rapidly developing image sensing technology with extreme low-light sensitivity and picosecond timing resolution. These unique capabilities have enabled SPADs to be used in applications like LiDAR, non-line-of-sight imaging and fluorescence microscopy that require imaging in photon-starved scenarios. In this work we harness these capabilities for dealing with motion blur in a passive imaging setting in low illumination conditions. Our key insight is that the data captured by a SPAD array camera can be represented as a 3D spatio-temporal tensor of photon detection events which can be integrated along arbitrary spatio-temporal trajectories with dynamically varying integration windows, depending on scene motion. We propose an algorithm that estimates pixel motion from photon timestamp data and dynamically adapts the integration windows to minimize motion blur. Our simulation results show the applicability of this algorithm to a variety of motion profiles including translation, rotation and local object motion. We also demonstrate the real-world feasibility of our method on data captured using a 32x32 SPAD camera. 
    more » « less
  2. ; ; ; (, Optica)
    Time-correlated single-photon counting (TCSPC) is an enabling technology for applications such as low-light fluorescence lifetime microscopy and photon counting time-of-flight (ToF) 3D imaging. However, state-of-the-art TCSPC single-photon timing resolution (SPTR) is limited to 3–100 ps by single-photon detectors. Here, we experimentally demonstrate a time-magnified TCSPC (TM-TCSPC) that achieves an ultrashort SPTR of 550 fs with an off-the-shelf single-photon detector. The TM-TCSPC can resolve ultrashort pulses with a 130-fs pulse width difference at a 22-fs accuracy. When applied to photon counting ToF 3D imaging, the TM-TCSPC greatly suppresses the range walk error that limits all photon counting ToF 3D imaging systems by 99.2% and thus provides high depth accuracy and precision of 26 µm and 3 µm, respectively. 
    more » « less
  3. ; ; (, Optics Express)
    Single-photon lidar (SPL) is a promising technology for depth measurement at long range or from weak reflectors because of the sensitivity to extremely low light levels. However, constraints on the timing resolution of existing arrays of single-photon avalanche diode (SPAD) detectors limit the precision of resulting depth estimates. In this work, we describe an implementation of subtractively-dithered SPL that can recover high-resolution depth estimates despite the coarse resolution of the detector. Subtractively-dithered measurement is achieved by adding programmable delays into the photon timing circuitry that introduce relative time shifts between the illumination and detection that are shorter than the time bin duration. Careful modeling of the temporal instrument response function leads to an estimator that outperforms the sample mean and results in depth estimates with up to 13 times lower root mean-squared error than if dither were not used. The simple implementation and estimation suggest that globally dithered SPAD arrays could be used for high spatial- and temporal-resolution depth sensing. 
    more » « less
  4. ; ; ; ; (, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV))
    Reconstruction of high-resolution extreme dynamic range images from a small number of low dynamic range (LDR) images is crucial for many computer vision applications. Current high dynamic range (HDR) cameras based on CMOS image sensor technology rely on multiexposure bracketing which suffers from motion artifacts and signal-to-noise (SNR) dip artifacts in extreme dynamic range scenes. Recently, single-photon cameras (SPCs) have been shown to achieve orders of magnitude higher dynamic range for passive imaging than conventional CMOS sensors. SPCs are becoming increasingly available commercially, even in some consumer devices. Unfortunately, current SPCs suffer from low spatial resolution. To overcome the limitations of CMOS and SPC sensors, we propose a learning-based CMOS-SPC fusion method to recover high-resolution extreme dynamic range images. We compare the performance of our method against various traditional and state-of-the-art baselines using both synthetic and experimental data. Our method outperforms these baselines, both in terms of visual quality and quantitative metrics. 
    more » « less
  5. ; ; ; (, Optica)
    In time-correlated single-photon counting (TCSPC), photons that arrive during the detector and timing electronics dead times are missed, causing distortion of the detection time distribution. Conventional wisdom holds that TCSPC should be performed with detections in fewer than 5% of illumination cycles to avoid substantial distortion. This requires attenuation and leads to longer acquisition times if the incident flux is too high. Through the example of ranging with a single-photon lidar system, this work demonstrates that accurately modeling the sequence of detection times as a Markov chain allows for measurements at much higher incident flux without attenuation. Our probabilistic model is validated by the close match between the limiting distribution of the Markov chain and both simulated and experimental data, so long as issues of calibration and afterpulsing are minimal. We propose an algorithm that corrects for the distortion in detection histograms caused by dead times without assumptions on the form of the transient light intensity. Our histogram correction yields substantially improved depth imaging performance, and modest additional improvement is achieved with a parametric model assuming a single depth per pixel. We show results for depth and flux estimation with up to 5 photoelectrons per illumination cycle on average, facilitating an increase in time efficiency of more than two orders of magnitude. The use of identical TCSPC equipment in other fields suggests that our modeling and histogram correction could likewise enable high-flux acquisitions in fluorescence lifetime microscopy or quantum optics applications. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email