Attention:The NSF Public Access Repository (PAR) system and access will be unavailable from 11:00 PM ET on Thursday, November 12th until 12:00 AM ET on Friday, November 13th due to maintenance. We apologize for the inconvenience.


Title: Object Detection Based on Raw Bayer Images
Bayer pattern is a widely used Color Filter Array (CFA) for digital image sensors, efficiently capturing different light wavelengths on different pixels without the need for a costly ISP pipeline. The resulting single-channel raw Bayer images offer benefits such as spectral wavelength sensitivity and low time latency. However, object detection based on Bayer images has been underexplored due to challenges in human observation and algorithm design caused by the discontinuous color channels in adjacent pixels. To address this issue, we propose the BayerDetect network, an end-to-end deep object detection framework that aims to achieve fast, accurate, and memory-efficient object detection. Unlike RGB color images, where each pixel encodes spectral context from adjacent pixels during ISP color interpolation, raw Bayer images lack spectral context. To enhance the spectral context, the BayerDetect network introduces a spectral frequency attention block, transforming the raw Bayer image pattern to the frequency domain. In object detection, clear object boundaries are essential for accurate bounding box predictions. To handle the challenges posed by alternating spectral channels and mitigate the influence of discontinuous boundaries, the BayerDetect network incorporates a spatial attention scheme that utilizes deformable convolutional kernels in multiple scales to explore spatial context effectively. The extracted convolutional features are then passed through a sparse set of proposal boxes for detection and classification. We conducted experiments on both public and self-collected raw Bayer images, and the results demonstrate the superb performance of the BayerDetect network in object detection tasks.  more » « less
Award ID(s):
2334246 2126643
PAR ID:
10542665
Author(s) / Creator(s):
Publisher / Repository:
IEEE
Date Published:
ISBN:
978-1-6654-9190-7
Page Range / eLocation ID:
9582 to 9589
Subject(s) / Keyword(s):
Location:
Detroit, MI, USA
Sponsoring Org:
National Science Foundation
More Like this
  1. Bayer pattern is a widely used Color Filter Array (CFA) for digital image sensors, efficiently capturing different light wavelengths on different pixels without the need for a costly ISP pipeline. The resulting single-channel raw Bayer images offer benefits such as spectral wavelength sensitivity and low time latency. However, object detection based on Bayer images has been underexplored due to challenges in human observation and algorithm design caused by the discontinuous color channels in adjacent pixels. To address this issue, we propose the BayerDetect network, an end-to-end deep object detection framework that aims to achieve fast, accurate, and memory-efficient object detection. Unlike RGB color images, where each pixel encodes spectral context from adjacent pixels during ISP color interpolation, raw Bayer images lack spectral context. To enhance the spectral context, the BayerDetect network introduces a spectral frequency attention block, transforming the raw Bayer image pattern to the frequency domain. In object detection, clear object boundaries are essential for accurate bounding box predictions. To handle the challenges posed by alternating spectral channels and mitigate the influence of discontinuous boundaries, the BayerDetect network incorporates a spatial attention scheme that utilizes deformable convolutional kernels in multiple scales to explore spatial context effectively. The extracted convolutional features are then passed through a sparse set of proposal boxes for detection and classification. We conducted experiments on both public and self-collected raw Bayer images, and the results demonstrate the superb performance of the BayerDetect network in object detection tasks. 
    more » « less
  2. Major semantic segmentation approaches are designed for RGB color images, which is interpolated from raw Bayer images. The use of RGB images on the one hand provides abundant scene color information. On the other hand, RGB images are easily observable for human users to understand the scene. The RGB color continuity also facilitates researchers to design segmentation algorithms, which becomes unnecessary in end-to-end learning. More importantly, the use of 3 channels adds extra storage and computation burden for neural networks. In contrast, the raw Bayer images can reserve the primitive color information in the largest extent with just a single channel. The compact design of Bayer pattern not only potentially increases a higher segmentation accuracy because of avoiding interpolation, but also significantly decreases the storage requirement and computation time in comparison with standard R, G, B images. In this paper, we propose BayerSeg-Net to segment single channel raw Bayer image directly. Different from RGB color images that already contain neighboring context information during ISP color interpolation, each pixel in raw Bayer images does not contain any context clues. Based on Bayer pattern properties, BayerSeg-Net assigns dynamic attention on Bayer images' spectral frequency and spatial locations to mitigate classification confusion, and proposes a re-sampling strategy to capture both global and local contextual information. We demonstrate the usability of raw Bayer images in segmentation tasks and the efficiency of BayerSeg-Net on multiple datasets. 
    more » « less
  3. Semantic segmentation methods are typically designed for RGB color images, which are interpolated from raw Bayer images. While RGB images provide abundant color information and are easily understood by humans, they also add extra storage and computational burden for neural networks. On the other hand, raw Bayer images preserve primitive color information with a single channel, potentially increasing segmentation accuracy while significantly decreasing storage and computation time. In this paper, we propose RawSeg-Net to segment single-channel raw Bayer images directly. Different from RGB images that already contain neighboring context information during ISP color interpolation, each pixel in raw Bayer images does not contain any context clues. Based on Bayer pattern properties, RawSeg-Net assigns dynamic attention on Bayer images' spectral frequency and spatial locations to mitigate classification confusion, and proposes a re-sampling strategy to capture both global and local contextual information. 
    more » « less
  4. Driver State Monitoring (DSM) is paramount for improving driving safety for both drivers of ego-vehicles and their surrounding road users, increasing public trust, and supporting the transition to autonomous driving. This paper introduces a Transformer-based classifier for DSM using an in-vehicle camera capturing raw Bayer images. Compared to traditional RGB images, we opt for the original Bayer data, further employing a Transformer-based classification algorithm. Experimental results prove that the accuracy of the Bayer Color-filled type images is only 0.61% lower than that of RGB images. Additionally, the performance of Bayer data is closely comparable to RGB images for DSM purposes. However, utilizing Bayer data can offer potential advantages, including reduced camera costs, lower energy consumption, and shortened Image Signal Processing (ISP) time. These benefits will enhance the efficacy of DSM systems and promote their widespread adoption. 
    more » « less
  5. Morph detection is of paramount significance when the integrity of Automatic Face Recognition (AFR) systems are concerned. Considering the risks incurred by morphing attacks, a robust automated morph detector is required which can distinguish authentic bona fide samples from altered morphed images. We leverage the wavelet sub-band decomposition of an input image, yielding the fine-grained spatial-frequency content of the input image. To enhance the detection of morphed images, our goal is to find the most discriminative information across frequency channels and spatial domain. To this end, we propose an end-to-end attention-based deep morph detector which assimilates the most discriminative wavelet sub-bands of a given image which are obtained by a group sparsity representation learning scheme. Specifically, our group sparsity-constrained Deep Neural Network (DNN) learns the most discriminative wavelet sub-bands (channels) of an input image while the attention mechanism captures the most discriminative spatial regions of input images for the downstream task of morph detection. To this end, we adopt three attention mechanisms to diversify our refined features for morph detection. As the first attention mechanism, we employ the Convolutional Block Attention Module (CBAM) which provides us with refined feature maps. As the second attention mechanism, compatibility scores across spatial locations and output of our DNN highlight the most discriminative regions, and lastly, the multiheaded self-attention augmented convolutions account for our third attention mechanism. We evaluate the efficiency of our proposed framework through extensive experiments using multiple morph datasets that are compiled using bona fide images available in the FERET, FRLL, FRGC, and WVU Twin datasets. Most importantly, our proposed methodology has resulted in a reduction in detection error rates when compared with state-of-the-art results. Finally, to further assess our multi-attentional morph detection, we delve into different combinations of attention mechanisms via a comprehensive ablation study. 
    more » « less