Non-Line-Of-Sight (NLOS) imaging aims at recovering the 3D geometry of objects that are hidden from the direct line of sight. One major challenge with this technique is the weak available multibounce signal limiting scene size, capture speed, and reconstruction quality. To overcome this obstacle, we introduce a multipixel time-of-flight non-line-of-sight imaging method combining specifically designed Single Photon Avalanche Diode (SPAD) array detectors with a fast reconstruction algorithm that captures and reconstructs live low-latency videos of non-line-of-sight scenes with natural non-retroreflective objects. We develop a model of the signal-to-noise-ratio of non-line-of-sight imaging and use it to devise a method that reconstructs the scene such that signal-to-noise-ratio, motion blur, angular resolution, and depth resolution are all independent of scene depth suggesting that reconstruction of very large scenes may be possible.
What You Can Learn by Staring at a Blank Wall
We present a passive non-line-of-sight method that infers the number of people or activity of a person from the observation of a blank wall in an unknown room. Our technique analyzes complex imperceptible changes in indirect illumination in a video of the wall to reveal a signal that is correlated with motion in the hidden part of a scene. We use this signal to classify between zero, one, or two moving people, or the activity of a person in the hidden scene. We train two convolutional neural networks using data collected from 20 different scenes, and achieve an accuracy of 94% for both tasks in unseen test environments and real-time online settings. Unlike other passive non-line-of-sight methods, the technique does not rely on known occluders or controllable light sources, and generalizes to unknown rooms with no recalibration. We analyze the generalization and robustness of our method with both real and synthetic data, and study the effect of the scene parameters on the signal quality.
more »
« less
- Award ID(s):
- 1816209
- NSF-PAR ID:
- 10301901
- Date Published:
- Journal Name:
- Int. Conf. Computer Vision (ICCV-2021)
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
In a built environment, wanting to see without direct line of sight is often due to being outside a doorway. The two vertical edges of the doorway provide occlusions that can be exploited for non-line-of-sight imaging by forming corner cameras. While each corner camera can separately yield a robust 1D reconstruction, joint processing suggests novelties in both forward modeling and inversion. The resulting doorway camera provides accurate and robust 2D reconstructions of the hidden scene. This work provides a novel inversion algorithm to jointly estimate two views of change in the hidden scene, using the temporal difference between photographs acquired on the visible side of the doorway. Successful reconstruction is demonstrated in a variety of real and rendered scenarios, including different hidden scenes and lighting conditions. A Cramer-Rao bound analysis is used to demonstrate the 2D resolving power of the doorway camera over other passive acquisition strategies and to motivate the novel biangular reconstruction grid.more » « less
-
more » « less
Abstract The ability to form reconstructions beyond line-of-sight view could be transformative in a variety of fields, including search and rescue, autonomous vehicle navigation, and reconnaissance. Most existing active non-line-of-sight (NLOS) imaging methods use data collection steps in which a pulsed laser is directed at several points on a relay surface, one at a time. The prevailing approaches include raster scanning of a rectangular grid on a vertical wall opposite the volume of interest to generate a collection of confocal measurements. These and a recent method that uses a horizontal relay surface are inherently limited by the need for laser scanning. Methods that avoid laser scanning to operate in a snapshot mode are limited to treating the hidden scene of interest as one or two point targets. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of foreground objects while also introducing the capability of mapping the stationary scenery behind moving objects. The ability to count, localize, and characterize the sizes of hidden objects, combined with mapping of the stationary hidden scene, could greatly improve indoor situational awareness in a variety of applications.
-
In this paper, we present a multiple concurrent occupant identification approach through footstep-induced floor vibration sensing. Identification of human occupants is useful in a variety of indoor smart structure scenarios, with applications in building security, space allocation, and healthcare. Existing approaches leverage sensing modalities such as vision, acoustic, RF, and wearables, but are limited due to deployment constraints such as line-of-sight requirements, sensitivity to noise, dense sensor deployment, and requiring each walker to wear/carry a device. To overcome these restrictions, we use footstep-induced structural vibration sensing. Footstep-induced signals contain information about the occupants' unique gait characteristics, and propagate through the structural medium, which enables sparse and passive identification of indoor occupants. The primary research challenge is that multiple-person footstep-induced vibration responses are a mixture of structurally-codependent overlapping individual responses with unknown timing, spectral content, and mixing ratios. As such, it is difficult to determine which part of the signal corresponds to each occupant. We overcome this challenge through a recursive sparse representation approach based on cosine distance that identifies each occupant in a footstep event in the order that their signals are generated, reconstructs their portion of the signal, and removes it from the mixed response. By leveraging sparse representation, our approach can simultaneously identify and separate mixed/overlapping responses, and the use of the cosine distance error function reduces the influence of structural codependency on the multiple walkers' signals. In this way, we isolate and identify each of the multiple occupants' footstep responses. We evaluate our approach by conducting real-world walking experiments with three concurrent walkers and achieve an average F1 score for identifying all persons of 0.89 (1.3x baseline improvement), and with a 10-person "hybrid" dataset (simulated combination of single-walker real-world data), we identify 2, 3, and 4 concurrent walkers with a trace-level accuracy of 100%, 93%, and 73%, respectively, and observe as much as a 2.9x error reduction over a naive baseline approach.more » « less
-
null (Ed.)Non-line-of-sight (NLOS) imaging is a rapidly advancing technology that provides asymmetric vision: seeing without being seen. Though limited in accuracy, resolution, and depth recovery compared to active methods, the capabilities of passive methods are especially surprising because they typically use only a single, inexpensive digital camera. One of the largest challenges in passive NLOS imaging is ambient background light, which limits the dynamic range of the measurement while carrying no useful information about the hidden part of the scene. In this work we propose a new reconstruction approach that uses an optimized linear transformation to balance the rejection of uninformative light with the retention of informative light, resulting in fast (video-rate) reconstructions of hidden scenes from photographs of a blank wall under high ambient light conditions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
