More Like this

1. The Architectural Implications of Multi-modal Detection Models for Autonomous Driving Systems

   Li, Yunge; Saha, Shaibal; Xu, Lanyu (May 2024, 2024 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST))

   Object detection plays a pivotal in autonomous driving by enabling the vehicles to perceive and comprehend their environment, thereby making informed decisions for safe navigation. Camera data provides rich visual context and object recognition, while LiDAR data offers precise distance measurements and 3D mapping. Multi-modal object detection models are gaining prominence in incorporating these data types, which is essential for the comprehensive perception and situational awareness needed in autonomous vehicles. Although graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) are promising hardware options for this application, the complex knowledge required to efficiently adapt and optimize multi-modal detection models for FPGAs presents a significant barrier to their utilization on this versatile and efficient platform. In this work, we evaluate the performance of camera and LiDARbased detection models on GPU and FPGA hardware, aiming to provide a specialized understanding for translating multi-modal detection models to suit the unique architecture of heterogeneous hardware platforms in autonomous driving systems. We focus on critical metrics from both system and model performance aspects. Based on our quantitative implications, we propose foundational insights and guidance for the design of camera and LiDAR-based multi-modal detection models on diverse hardware platforms. 

   more » « less
2. A Study of Optical Tag Detection Using Rolling Shutter Based Visible Light Communications

   https://doi.org/10.1109/GLOBECOM38437.2019.9013449  

   Pan, Zijin; Lang, Tian; Li, Zening; Chen, Gang; Wang, Albert (December 2019, Proc. IEEE Global Communications Conference (GLOBECOM))

   null (Ed.)

   In this paper, we present an in-depth study of light emitting diode (LED) based indoor visible light communication positioning system using a smart phone camera with rolling shutter effect, aiming for smart and connected hospital applications. The LED transmits periodical signals with different frequencies as its optical tags. The camera exploits the rolling shutter effect to detect the fundamental frequency of optical signals. The roles of camera parameters determining the rolling effect are studied and a technique to measure the camera readout time per column is presented. Factors limiting the detectable optical frequency range is explained based on the discussion of rolling shutter mechanism. The Fourier spectrum based frequency resolution, which determines the tracking capacity, is analyzed. 

   more » « less
3. Malicious Attacks against Multi-Sensor Fusion in Autonomous Driving

   https://doi.org/10.1145/3636534.3649372  

   Zhu, Yi; Miao, Chenglin; Xue, Hongfei; Yu, Yunnan; Su, Lu; Qiao, Chunming (May 2024, ACM)

   Multi-sensor fusion has been widely used by autonomous vehicles (AVs) to integrate the perception results from different sensing modalities including LiDAR, camera and radar. Despite the rapid development of multi-sensor fusion systems in autonomous driving, their vulnerability to malicious attacks have not been well studied. Although some prior works have studied the attacks against the perception systems of AVs, they only consider a single sensing modality or a camera-LiDAR fusion system, which can not attack the sensor fusion system based on LiDAR, camera, and radar. To fill this research gap, in this paper, we present the first study on the vulnerability of multi-sensor fusion systems that employ LiDAR, camera, and radar. Specifically, we propose a novel attack method that can simultaneously attack all three types of sensing modalities using a single type of adversarial object. The adversarial object can be easily fabricated at low cost, and the proposed attack can be easily performed with high stealthiness and flexibility in practice. Extensive experiments based on a real-world AV testbed show that the proposed attack can continuously hide a target vehicle from the perception system of a victim AV using only two small adversarial objects. 

   more » « less
4. IMU-Assisted Learning of Single-View Rolling Shutter Correction

   Mo, J; Islam, Md J.; Sattar, J. (November 2021, Conference on Robot Learning)

   Rolling shutter distortion is highly undesirable for photography and computer vision algorithms (e.g., visual SLAM) because pixels can be potentially captured at different times and poses. In this paper, we propose a deep neural network to predict depth and row-wise pose from a single image for rolling shutter correction. Our contribution in this work is to incorporate inertial measurement unit (IMU) data into the pose refinement process, which, compared to the state-of-the-art, greatly enhances the pose prediction. The improved accuracy and robustness make it possible for numerous vision algorithms to use imagery captured by rolling shutter cameras and produce highly accurate results. We also extend a dataset to have real rolling shutter images, IMU data, depth maps, camera poses, and corresponding global shutter images for rolling shutter correction training. We demonstrate the efficacy of the proposed method by evaluating the performance of Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery corrected using the proposed approach. Results show marked improvements of the DSO algorithm over using uncorrected imagery, validating the proposed approach. 

   more » « less
5. Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks

   Cao, Yulong; Wang, Ningfei; Xiao, Chaowei; Yang, Dawei; Fang, Jin; Yang, Ruigang; Chen, Qi Alfred; Liu, Mingyan; Li, Bo (January 2021, IEEE Symposium on Security and Privacy (S&P))

   null (Ed.)

   In Autonomous Driving (AD) systems, perception is both security and safety critical. Despite various prior studies on its security issues, all of them only consider attacks on cameraor LiDAR-based AD perception alone. However, production AD systems today predominantly adopt a Multi-Sensor Fusion (MSF) based design, which in principle can be more robust against these attacks under the assumption that not all fusion sources are (or can be) attacked at the same time. In this paper, we present the first study of security issues of MSF-based perception in AD systems. We directly challenge the basic MSF design assumption above by exploring the possibility of attacking all fusion sources simultaneously. This allows us for the first time to understand how much security guarantee MSF can fundamentally provide as a general defense strategy for AD perception. We formulate the attack as an optimization problem to generate a physically-realizable, adversarial 3D-printed object that misleads an AD system to fail in detecting it and thus crash into it. To systematically generate such a physical-world attack, we propose a novel attack pipeline that addresses two main design challenges: (1) non-differentiable target camera and LiDAR sensing systems, and (2) non-differentiable cell-level aggregated features popularly used in LiDAR-based AD perception. We evaluate our attack on MSF algorithms included in representative open-source industry-grade AD systems in real-world driving scenarios. Our results show that the attack achieves over 90% success rate across different object types and MSF algorithms. Our attack is also found stealthy, robust to victim positions, transferable across MSF algorithms, and physical-world realizable after being 3D-printed and captured by LiDAR and camera devices. To concretely assess the end-to-end safety impact, we further perform simulation evaluation and show that it can cause a 100% vehicle collision rate for an industry-grade AD system. We also evaluate and discuss defense strategies. 

   more » « less