More Like this

1. FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements

   https://doi.org/10.1109/TPAMI.2020.3033882  

   Khan, Salman Siddique; Sundar, Varun; Boominathan, Vivek; Veeraraghavan, Ashok; Mitra, Kaushik (October 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence)

   null (Ed.)

   Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning-based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called FlatNet, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras. 

   more » « less
2. CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo

   https://doi.org/10.1109/CVPR46437.2021.00709  

   Tan, Shiyu; Wu, Yicheng; Yu, Shoou-I; Veeraraghavan, Ashok (June 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR))

   Conventional stereo suffers from a fundamental trade-off between imaging volume and signal-to-noise ratio (SNR) – due to the conflicting impact of aperture size on both these variables. Inspired by the extended depth of field cameras, we propose a novel end-to-end learning-based technique to overcome this limitation, by introducing a phase mask at the aperture plane of the cameras in a stereo imaging system. The phase mask creates a depth-dependent yet numerically invertible point spread function, allowing us to recover sharp image texture and stereo correspondence over a significantly extended depth of field (EDOF) than conventional stereo. The phase mask pattern, the EDOF image reconstruction, and the stereo disparity estimation are all trained together using an end-to-end learned deep neural network. We perform theoretical analysis and characterization of the proposed approach and show a 6× increase in volume that can be imaged in simulation. We also build an experimental prototype and validate the approach using real-world results acquired using this prototype system. 

   more » « less
3. Pix2HDR - A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos

   https://doi.org/10.1109/TPAMI.2024.3410140  

   Wang, Caixin; Zhang, Jie; Wilson, Matthew A; Etienne-Cummings, Ralph (January 2024, IEEE Transactions on Pattern Analysis and Machine Intelligence)

   Abstract—Accurately capturing dynamic scenes with wideranging motion and light intensity is crucial for many vision applications. However, acquiring high-speed high dynamic range (HDR) video is challenging because the camera’s frame rate restricts its dynamic range. Existing methods sacrifice speed to acquire multi-exposure frames. Yet, misaligned motion in these frames can still pose complications for HDR fusion algorithms, resulting in artifacts. Instead of frame-based exposures, we sample the videos using individual pixels at varying exposures and phase offsets. Implemented on a monochrome pixel-wise programmable image sensor, our sampling pattern captures fast motion at a high dynamic range. We then transform pixel-wise outputs into an HDR video using end-to-end learned weights from deep neural networks, achieving high spatiotemporal resolution with minimized motion blurring. We demonstrate aliasing-free HDR video acquisition at 1000 FPS, resolving fast motion under low-light conditions and against bright backgrounds — both challenging conditions for conventional cameras. By combining the versatility of pixel-wise sampling patterns with the strength of deep neural networks at decoding complex scenes, our method greatly enhances the vision system’s adaptability and performance in dynamic conditions. Index Terms—High-dynamic-range video, high-speed imaging, CMOS image sensors, programmable sensors, deep learning, convolutional neural networks. 

   more » « less
4. Spatial and axial resolution limits for mask-based lensless cameras

   https://doi.org/10.1364/OE.480025  

   Hua, Yi; Asif, M_Salman; Sankaranarayanan, Aswin_C (January 2023, Optics Express)

   One of the open challenges in lensless imaging is understanding how well they resolve scenes in three dimensions. The measurement model underlying prior lensless imagers lacks special structures that facilitate deeper analysis; thus, a theoretical study of the achievable spatio-axial resolution has been lacking. This paper provides such a theoretical framework by analyzing a generalization of a mask-based lensless camera, where the sensor captures z-stacked measurements acquired by moving the sensor relative to an attenuating mask. We show that the z-stacked measurements are related to the scene’s volumetric albedo function via a three-dimensional convolutional operator. The specifics of this convolution, and its Fourier transform, allow us to fully characterize the spatial and axial resolving power of the camera, including its dependence on the mask. Since z-stacked measurements are a superset of those made by previously-studied lensless systems, these results provide an upper bound for their performance. We numerically evaluate the theory and its implications using simulations. 

   more » « less
5. PhaseCam3D — Learning Phase Masks for Passive Single View Depth Estimation

   https://doi.org/10.1109/ICCPHOT.2019.8747330  

   Wu, Yicheng; Boominathan, Vivek; Chen, Huaijin; Sankaranarayanan, Aswin; Veeraraghavan, Ashok (May 2019, IEEE International Conference on Computational Photography)

   There is an increasing need for passive 3D scanning in many applications that have stringent energy constraints. In this paper, we present an approach for single frame, single viewpoint, passive 3D imaging using a phase mask at the aperture plane of a camera. Our approach relies on an end-to-end optimization framework to jointly learn the optimal phase mask and the reconstruction algorithm that allows an accurate estimation of range image from captured data. Using our optimization framework, we design a new phase mask that performs significantly better than existing approaches. We build a prototype by inserting a phase mask fabricated using photolithography into the aperture plane of a conventional camera and show compelling performance in 3D imaging. 

   more » « less