An official website of the United States government
Sharing and joint processing of camera feeds and sensor measurements, known as Cooperative Perception (CP), has emerged as a new technique to achieve higher perception qualities. CP can enhance the safety of Autonomous Vehicles (AVs) where their individual visual perception quality is compromised by adverse weather conditions (haze as foggy weather), low illumination, winding roads, and crowded traffic. While previous CP methods have shown success in elevating perception quality, they often assume perfect communication conditions and unlimited transmission resources to share camera feeds, which may not hold in real-world scenarios. Also, they make no effort to select better helpers when multiple options are available.To cover the limitations of former methods, in this paper, we propose a novel approach to realize an optimized CP under constrained communications. At the core of our approach is recruiting the best helper from the available list of front vehicles to augment the visual range and enhance the Object Detection (OD) accuracy of the ego vehicle. In this two-step process, we first select the helper vehicles that contribute the most to CP based on their visual range and lowest motion blur. Next, we implement a radio block optimization among the candidate vehicles to further improve communication efficiency. We specifically focus on pedestrian detection as an exemplary scenario. To validate our approach, we used the CARLA simulator to create a dataset of annotated videos for different driving scenarios where pedestrian detection is challenging for an AV with compromised vision. Our results demonstrate the efficacy of our two-step optimization process in improving the overall performance of cooperative perception in challenging scenarios, substantially improving driving safety under adverse conditions. Finally, we note that the networking assumptions are adopted from LTE Release 14 Mode 4 side-link communication, commonly used for Vehicle-to-Vehicle (V2V) commun
more »
« less
Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals
Vehicle detection with visual sensors like lidar and camera is one of the critical functions enabling autonomous driving. While they generate fine-grained point clouds or high-resolution images with rich information in good weather conditions, they fail in adverse weather (e.g., fog) where opaque particles distort lights and significantly reduce visibility. Thus, existing methods relying on lidar or camera experience significant performance degradation in rare but critical adverse weather conditions. To remedy this, we resort to exploiting complementary radar, which is less impacted by adverse weather and becomes prevalent on vehicles. In this paper, we present Multimodal Vehicle Detection Network (MVDNet), a two-stage deep fusion detector, which first generates proposals from two sensors and then fuses region-wise features between multimodal sensor streams to improve final detection results. To evaluate MVDNet, we create a procedurally generated training dataset based on the collected raw lidar and radar signals from the open-source Oxford Radar Robotcar. We show that the proposed MVDNet surpasses other state-of-the-art methods, notably in terms of Average Precision (AP), especially in adverse weather conditions. The code and data are available at https://github.com/qiank10/MVDNet.
more »
« less
- Award ID(s):
- 1925767
- PAR ID:
- 10300639
- Date Published:
- Journal Name:
- IEEE Conference on Computer Vision and Pattern Recognition
- ISSN:
- 2163-6648
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Emerging autonomous driving systems require reliable perception of 3D surroundings. Unfortunately, current mainstream perception modalities, i.e., camera and Lidar, are vulnerable under challenging lighting and weather conditions. On the other hand, despite their all-weather operations, today's vehicle Radars are limited to location and speed detection. In this paper, we introduce MILLIPOINT, a practical system that advances the Radar sensing capability to generate 3D point clouds. The key design principle of MILLIPOINT lies in enabling synthetic aperture radar (SAR) imaging on low-cost commodity vehicle Radars. To this end, MILLIPOINT models the relation between signal variations and Radar movement, and enables self-tracking of Radar at wavelength-scale precision, thus realize coherent spatial sampling. Furthermore, MILLIPOINT solves the unique problem of specular reflection, by properly focusing on the targets with post-imaging processing. It also exploits the Radar's built-in antenna array to estimate the height of reflecting points, and eventually generate 3D point clouds. We have implemented MILLIPOINT on a commodity vehicle Radar. Our evaluation results show that MILLIPOINT effectively combats motion errors and specular reflections, and can construct 3D point clouds with much higher density and resolution compared with the existing vehicle Radar solutions.more » « less
-
Multi-sensor fusion has been widely used by autonomous vehicles (AVs) to integrate the perception results from different sensing modalities including LiDAR, camera and radar. Despite the rapid development of multi-sensor fusion systems in autonomous driving, their vulnerability to malicious attacks have not been well studied. Although some prior works have studied the attacks against the perception systems of AVs, they only consider a single sensing modality or a camera-LiDAR fusion system, which can not attack the sensor fusion system based on LiDAR, camera, and radar. To fill this research gap, in this paper, we present the first study on the vulnerability of multi-sensor fusion systems that employ LiDAR, camera, and radar. Specifically, we propose a novel attack method that can simultaneously attack all three types of sensing modalities using a single type of adversarial object. The adversarial object can be easily fabricated at low cost, and the proposed attack can be easily performed with high stealthiness and flexibility in practice. Extensive experiments based on a real-world AV testbed show that the proposed attack can continuously hide a target vehicle from the perception system of a victim AV using only two small adversarial objects.more » « less
-
This paper is on a pedestrian collision warning and avoidance system for road vehicles based on V2X communication. In cases where the presence and location of a pedestrian or group of pedestrians cannot be determined using line-of-sight sensors like camera, radar and lidar, signals from pedestrians' smartphone apps are used to detect and localize them relative to the road vehicle through the DSRC radio used for V2X communication. A hardware-in-the-loop setup using a validated automated driving vehicle model in the high fidelity vehicle dynamics simulation program Carsim Real Time with Sensors and Traffic is used along with two DSRC modems emulating the vehicle and pedestrian communications in the development and initial experimental testing of this method. The vehicle either stops or, if possible, goes around the pedestrians in a socially acceptable manner. The elastic band method is used to locally modify the vehicle trajectory in real time when pedestrians are detected on the nearby path of the vehicle. The effectiveness of the proposed method is demonstrated using hardware-in-the-loop simulations.more » « less
-
The significant advancements in autonomous vehicle applications demand detection solutions capable of swiftly recognizing and classifying objects amidst rapidly changing and low-visibility conditions. Light detection and ranging (LiDAR) has emerged as a robust solution, overcoming challenges associated with camera imaging, particularly in adverse weather conditions or low illumination. Rapid object recognition is crucial in dynamic environments, but the speed of conventional LiDARs is often constrained by the 2D scanning of the laser beam across the entire scene. In this study, we introduce a parallelization approach for the indirect time-of-flight (iToF) ranging technique. This method enables efficient and high-speed formation of 1D clouds, offering the potential to have extended range capabilities without being constrained by the laser coherence length. The application potential spans mid-range autonomous vehicles ranging to high-resolution imaging. It utilizes dual-frequency combs with slightly different repetition rates. The method leverages the topology of the target object to influence the phase of the beating signal between the comb lines in the RF domain. This approach enables parallel ranging in one direction, confining the scanning process to a single dimension, and offers the potential for high-speed LiDAR systems. A tri-comb approach will be discussed that can provide an extended unambiguous range without compromising the resolution due to the range–resolution trade-off in iToF techniques. The study starts by explaining the technique for parallel detection of distance and velocity. It then presents a theoretical estimation of phase noise for dual combs, followed by an analysis of distance and velocity detection limits, illustrating their maximum and minimum extents. Finally, a study on the mutual interference conditions between two similar LiDAR systems is presented, demonstrating the feasibility of designing simultaneously operating LiDARs to avoid mutual interference.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
