More Like this

1. Pix2HDR - A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos

   https://doi.org/10.1109/TPAMI.2024.3410140  

   Wang, Caixin; Zhang, Jie; Wilson, Matthew A; Etienne-Cummings, Ralph (January 2024, IEEE Transactions on Pattern Analysis and Machine Intelligence)

   Abstract—Accurately capturing dynamic scenes with wideranging motion and light intensity is crucial for many vision applications. However, acquiring high-speed high dynamic range (HDR) video is challenging because the camera’s frame rate restricts its dynamic range. Existing methods sacrifice speed to acquire multi-exposure frames. Yet, misaligned motion in these frames can still pose complications for HDR fusion algorithms, resulting in artifacts. Instead of frame-based exposures, we sample the videos using individual pixels at varying exposures and phase offsets. Implemented on a monochrome pixel-wise programmable image sensor, our sampling pattern captures fast motion at a high dynamic range. We then transform pixel-wise outputs into an HDR video using end-to-end learned weights from deep neural networks, achieving high spatiotemporal resolution with minimized motion blurring. We demonstrate aliasing-free HDR video acquisition at 1000 FPS, resolving fast motion under low-light conditions and against bright backgrounds — both challenging conditions for conventional cameras. By combining the versatility of pixel-wise sampling patterns with the strength of deep neural networks at decoding complex scenes, our method greatly enhances the vision system’s adaptability and performance in dynamic conditions. Index Terms—High-dynamic-range video, high-speed imaging, CMOS image sensors, programmable sensors, deep learning, convolutional neural networks. 

   more » « less
2. Synchronized strobed phase contrast and fluorescence microscopy: the interlaced standard reimagined

   https://doi.org/10.1364/OE.474045  

   Hosu, Basarab G.; Hill, Winfield; Samuel, Aravinthan D.; Berg, Howard C. (January 2023, Optics Express)

   We propose a simple, cost-effective method for synchronized phase contrast and fluorescence video acquisition in live samples. Counter-phased pulses of phase contrast illumination and fluorescence excitation light are synchronized with the exposure of the two fields of an interlaced camera sensor. This results in a video sequence in which each frame contains both exposure modes, each in half of its pixels. The method allows real-time acquisition and display of synchronized and spatially aligned phase contrast and fluorescence image sequences that can be separated by de-interlacing in two independent videos. The method can be implemented on any fluorescence microscope with a camera port without needing to modify the optical path. 

   more » « less
3. Efficient Video Redaction at the Edge: Human Motion Tracking for Privacy Protection

   https://doi.org/10.1145/3762994  

   Qiao, Haotian; Srinivas, Vidya; Dinda, Peter; Dick, Robert (August 2025, ACM Transactions on Embedded Computing Systems)

   Computationally efficient, camera-based, real-time human position tracking on low-end, edge devices would enable numerous applications, including privacy-preserving video redaction and analysis. Unfortunately, running most deep neural network based models in real time requires expensive hardware, making widespread deployment difficult, particularly on edge devices. Shifting inference to the cloud increases the attack surface, generally requiring that users trust cloud servers, and increases demands on wireless networks in deployment venues. Our goal is to determine the extreme to which edge video redaction efficiency can be taken, with a particular interest in enabling, for the first time, low-cost, real-time deployments with inexpensive commodity hardware. We present an efficient solution to the human detection (and redaction) problem based on singular value decomposition (SVD) background removal and describe a novel time- and energy-efficient sensor-fusion algorithm that leverages human position information in real-world coordinates to enable real-time visual human detection and tracking at the edge. These ideas are evaluated using a prototype built from (resource-constrained) commodity hardware representative of commonly used low-cost IoT edge devices. The speed and accuracy of the system are evaluated via a deployment study, and it is compared with the most advanced relevant alternatives. The multi-modal system operates at a frame rate ranging from 20 FPS to 60 FPS, achieves awIoU_0.3score (see Section 5.4) ranging from 0.71 to 0.79, and successfully performs complete redaction of privacy-sensitive pixels with a success rate of 91%–99% in human head regions and 77%–91% in upper body regions, depending on the number of individuals present in the field of view. These results demonstrate that it is possible to achieve adequate efficiency to enable real-time redaction on inexpensive, commodity edge hardware. 

   more » « less
4. Low-Rank Adaptation of Neural Fields

   https://doi.org/10.1145/3757377.3763882  

   Truong, Anh; Mahmoud, Ahmed H; Konaković_Luković, Mina; Solomon, Justin (December 2025, ACM)

   Processing visual data often involves small adjustments or sequences of changes, e.g., image filtering, surface smoothing, and animation. While established graphics techniques like normal mapping and video compression exploit redundancy to encode such small changes efficiently, the problem of encoding small changes to neural fields—neural network parameterizations of visual or physical functions—has received less attention. We propose a parameter-efficient strategy for updating neural fields using low-rank adaptations (LoRA). LoRA, a method from the parameter-efficient fine-tuning LLM community, encodes small updates to pre-trained models with minimal computational overhead. We adapt LoRA for instance-specific neural fields, avoiding the need for large pre-trained models and yielding lightweight updates. We validate our approach with experiments in image filtering, geometry editing, video compression, and energy-based editing, demonstrating its effectiveness and versatility for representing neural field updates. 

   more » « less
5. Squint: A Framework for Dynamic Voltage Scaling of Image Sensors Towards Low Power IoT Vision

   Kodukula, Venkatesh; Manetta, Mason; LiKamWa, Robert (January 2023, The 29th Annual International Conference on Mobile Computing and Networking (ACM MobiCom ’23))

   Energy-efficient visual sensing is of paramount importance to enable battery-backed low power IoT and mobile applications. Unfortunately, modern image sensors still consume hundreds of milliwatts of power, mainly due to analog readout. This is because current systems always supply a fixed voltage to the sensor’s analog circuitry, leading to higher power profiles. In this work, we propose to aggressively scale the analog voltage supplied to the camera as a means to significantly reduce sensor power consumption. To that end, we characterize the power and fidelity implications of analog voltage scaling on three off-the-shelf image sensors. Our characterization reveals that analog voltage scaling reduces sensor power but also degrades image quality. Furthermore, the degradation in image quality situationally affects the task accuracy of vision applications. We develop a visual streaming pipeline that flexibly allows application developers to dynamically adapt sensor voltage on a frame-by-frame basis. We develop a voltage controller that programmatically generates desired sensor voltage based on application request. We integrate our voltage controller into the existing RPi-based video streaming IoT pipeline. On top of this, we develop runtime support for flexible voltage specification from vision applications. Evaluating the system over a wide range of voltage scaling policies on popular vision tasks reveals that Squint imaging can deliver up to 73% sensor power savings, while maintaining reasonable task fidelity. Our artifacts are available at: https://gitlab.com/squint1/squint-ae-public 

   more » « less