More Like this

1. AutoPCD: Learning-Augmented Indoor Point Cloud Completion

   Cai, Pingping; Sitar, Edward M.; Sur, Sanjib (September 2021, Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing)

   null (Ed.)

   3D Point Cloud (PCD) is an efficient machine representation for surrounding environments and has been used in many applications. But a fast reconstruction of complete PCD for large environments remains a challenge. We propose AutoPCD, a machine-learning model that reconstructs complete PCDs, under sensor occlusion and poor lighting conditions. AutoPCD splits the PCD into multiple parts, approximates them by several 3D planes, and independently learns the plane features for reconstruction. We have experimentally evaluated AutoPCD in a large indoor hallway environment. 

   more » « less
2. A Compact Handheld Sensor Package with Sensor Fusion for Comprehensive and Robust 3D Mapping

   https://doi.org/10.3390/s24082494  

   Wei, Peng; Fu, Kaiming; Villacres, Juan; Ke, Thomas; Krachenfels, Kay; Stofer, Curtis Ryan; Bayati, Nima; Gao, Qikai; Zhang, Bill; Vanacker, Eric; et al (April 2024, Sensors)

   This paper introduces an innovative approach to 3D environmental mapping through the integration of a compact, handheld sensor package with a two-stage sensor fusion pipeline. The sensor package, incorporating LiDAR, IMU, RGB, and thermal cameras, enables comprehensive and robust 3D mapping of various environments. By leveraging Simultaneous Localization and Mapping (SLAM) and thermal imaging, our solution offers good performance in conditions where global positioning is unavailable and in visually degraded environments. The sensor package runs a real-time LiDAR-Inertial SLAM algorithm, generating a dense point cloud map that accurately reconstructs the geometric features of the environment. Following the acquisition of that point cloud, we post-process these data by fusing them with images from the RGB and thermal cameras and produce a detailed, color-enriched 3D map that is useful and adaptable to different mission requirements. We demonstrated our system in a variety of scenarios, from indoor to outdoor conditions, and the results showcased the effectiveness and applicability of our sensor package and fusion pipeline. This system can be applied in a wide range of applications, ranging from autonomous navigation to smart agriculture, and has the potential to make a substantial benefit across diverse fields. 

   more » « less
3. SeaScan: An Energy-Efficient Underwater Camera for Wireless 3D Color Imaging

   https://doi.org/10.1145/3636534.3690661  

   Naeem, Nazish; Rademacher, Jack; Patnaik, Ritik; Boroushaki, Tara; Adib, Fadel (December 2024, ACM)

   We present the design, implementation, and evaluation of SeaScan, an energy-efficient camera for 3D imaging of underwater environments. At the core of SeaScan’s design is a trinocular lensing system, which employs three ultra-lowpower monochromatic image sensors to reconstruct color images. Each of the sensors is equipped with a different filter (red, green, and blue) for color capture. The design introduces multiple innovations to enable reconstructing 3D color images from the captured monochromatic ones. This includes an ML-based cross-color alignment architecture to combine the monochromatic images. It also includes a cross-refractive compensation technique that overcomes the distortion of the wide-angle imaging of the low-power CMOS sensors in underwater environments.We built an end-to-end prototype of SeaScan, including color filter integration, 3D reconstruction, compression, and underwater backscatter communication. Our evaluation in real-world underwater environments demonstrates that SeaScan can capture underwater color images with as little as 23.6 mJ, which represents 37× reduction in energy consumption in comparison to the lowest-energy state-of-the-art underwater imaging system.We also report qualitative and quantitative evaluation of SeaScan’s color reconstruction and demonstrate its success in comparison to multiple potential alternative techniques (both geometric and ML-based) in the literature. SeaScan’s ability to image underwater environments at such low energy opens up important applications in long-term monitoring for ocean climate change, seafood production, and scientific discovery. 

   more » « less
4. ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements

   https://doi.org/10.1145/3615984.3616503  

   Cao, Bryan Bo; Alali, Abrar; Liu, Hansi; Meegan, Nicholas; Gruteser, Marco; Dana, Kristin; Ashok, Ashwin; Jain, Shubham (October 2023, ACM)

   Tracking subjects in videos is one of the most widely used functions in camera-based IoT applications such as security surveillance, smart city traffic safety enhancement, vehicle to pedestrian communication and so on. In computer vision domain, tracking is usually achieved by first detecting subjects, then associating detected bounding boxes across video frames. Typically, frames are transmitted to a remote site for processing, incurring high latency and network costs. To address this, we propose ViFiT, a transformerbased model that reconstructs vision bounding box trajectories from phone data (IMU and Fine Time Measurements). It leverages a transformer’s ability of better modeling long-term time series data. ViFiT is evaluated on Vi-Fi Dataset, a large-scale multimodal dataset in 5 diverse real world scenes, including indoor and outdoor environments. Results demonstrate that ViFiT outperforms the state-of-the-art approach for cross-modal reconstruction in LSTM Encoder-Decoder architecture X-Translator and achieves a high frame reduction rate as 97.76% with IMU and Wi-Fi data. 

   more » « less
5. DeepInMiniscope: Deep learning–powered physics-informed integrated miniscope

   https://doi.org/10.1126/sciadv.adr6687  

   Tian, Feng; Mattison, Ben; Yang, Weijian (September 2025, Science Advances)

   Mask-based integrated fluorescence microscopy is a compact imaging technique for biomedical research. It can perform snapshot 3D imaging through a thin optical mask with a scalable field of view (FOV). Integrated microscopy uses computational algorithms for object reconstruction, but efficient reconstruction algorithms for large-scale data have been lacking. Here, we developed DeepInMiniscope, a miniaturized integrated microscope featuring a custom-designed optical mask and an efficient physics-informed deep learning model that markedly reduces computational demand. Parts of the 3D object can be individually reconstructed and combined. Our deep learning algorithm can reconstruct object volumes over 4 millimeters by 6 millimeters by 0.6 millimeters. We demonstrated substantial improvement in both reconstruction quality and speed compared to traditional methods for large-scale data. Notably, we imaged neuronal activity with near-cellular resolution in awake mouse cortex, representing a substantial leap over existing integrated microscopes. DeepInMiniscope holds great promise for scalable, large-FOV, high-speed, 3D imaging applications with compact device footprint. 

   more » « less