An official website of the United States government
Abstract Teleoperation can enable human intervention to help handle instances of failure in autonomy thus allowing for much safer deployment of autonomous vehicle technology. Successful teleoperation requires recreating the environment around the remote vehicle using camera data received over wireless communication channels. This paper develops a new predictive display system to tackle the significant time delays encountered in receiving camera data over wireless networks. First, a new high gain observer is developed for estimating the position and orientation of the ego vehicle. The novel observer is shown to perform accurate state estimation using only GNSS and gyroscope sensor readings. A vector field method which fuses the delayed camera and Lidar data is then presented. This method uses sparse 3D points obtained from Lidar and transforms them using the state estimates from the high gain observer to generate a sparse vector field for the camera image. Polynomial based interpolation is then performed to obtain the vector field for the complete image which is then remapped to synthesize images for accurate predictive display. The method is evaluated on real-world experimental data from the nuScenes and KITTI datasets. The performance of the high gain observer is also evaluated and compared with that of the EKF. The synthesized images using the vector field based predictive display are compared with ground truth images using various image metrics and offer vastly improved performance compared to delayed images.
more »
« less
Teleoperation Enhancement for Autonomous Vehicles Using Estimation Based Predictive Display
Teleoperation is increasingly used in the operation of delivery robots and is beginning to be utilized for certain autonomous vehicle intervention applications. This paper addresses the challenges in teleoperation of an autonomous vehicle due to latencies in wireless communication between the remote vehicle and the teleoperator station. Camera video images and Lidar data are typically delayed during wireless transmission but are critical for proper display of the remote vehicle's real-time road environment to the teleoperator. Data collected with experiments in this project show that a 0.5 second delay in real-time display makes it extremely difficult for the teleoperator to control the remote vehicle. This problem is addressed in the paper by using a predictive display (PD) system which provides intermediate updates of the remote vehicle's environment while waiting for actual camera images. The predictive display utilizes estimated positions of the ego vehicle and of other vehicles on the road computed using model-based extended Kalman filters. A crucial result presented in the paper is that vehicle motion models need to be inertial rather than relative and so tracking of other vehicles requires accurate localization of the ego vehicle itself. An experimental study using 5 human teleoperators is conducted to compare teleoperation performance with and without predictive display. A 0.5 second time-delay in camera images makes it impossible to control the vehicle to stay in its lane on curved roads, but the use of the developed predictive display system enables safe remote vehicle control with almost as accurate a performance as the delay-free case.
more »
« less
- Award ID(s):
- 2321531
- PAR ID:
- 10534536
- Publisher / Repository:
- IEEE
- Date Published:
- Journal Name:
- IEEE Transactions on Intelligent Vehicles
- ISSN:
- 2379-8904
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Driver State Monitoring (DSM) is paramount for improving driving safety for both drivers of ego-vehicles and their surrounding road users, increasing public trust, and supporting the transition to autonomous driving. This paper introduces a Transformer-based classifier for DSM using an in-vehicle camera capturing raw Bayer images. Compared to traditional RGB images, we opt for the original Bayer data, further employing a Transformer-based classification algorithm. Experimental results prove that the accuracy of the Bayer Color-filled type images is only 0.61% lower than that of RGB images. Additionally, the performance of Bayer data is closely comparable to RGB images for DSM purposes. However, utilizing Bayer data can offer potential advantages, including reduced camera costs, lower energy consumption, and shortened Image Signal Processing (ISP) time. These benefits will enhance the efficacy of DSM systems and promote their widespread adoption.more » « less
-
Multi-agent autonomous racing is a challenging problem for autonomous vehicles due to the split-second, and complex decisions that vehicles must continuously make during a race. The presence of other agents on the track requires continuous monitoring of the ego vehicle’s surroundings, and necessitates predicting the behavior of other vehicles so the ego can quickly react to a changing environment with informed decisions. In our previous work we have developed the DeepRacing AI framework for autonomous formula one racing. Our DeepRacing framework was the first implementation to use the highly photorealisitc Formula One game as a simulation testbed for autonomous racing. We have successfully demonstrated single agent high speed autonomous racing using Bezier curve trajectories. In this paper, we extend the capabilities of the DeepRacing framework towards multi-agent autonomous racing. To do so, we first develop and learn a virtual camera model from game data that the user can configure to emulate the presence of a camera sensor on the vehicle. Next we propose and train a deep recurrent neural network that can predict the future poses of opponent agents in the field of view of the virtual camera using vehicles position, velocity, and heading data with respect to the ego vehicle racecar. We demonstrate early promising results for both these contributions in the game. These added features will extend the DeepRacing framework to become more suitable for multi-agent autonomous racing algorithm developmentmore » « less
-
Autonomous vehicle trajectory tracking control is challenged by situations of varying road surface friction, especially in the scenario where there is a sudden decrease in friction in an area with high road curvature. If the situation is unknown to the control law, vehicles with high speed are more likely to lose tracking performance and/or stability, resulting in loss of control or the vehicle departing the lane unexpectedly. However, with connectivity either to other vehicles, infrastructure, or cloud services, vehicles may have access to upcoming roadway information, particularly the friction and curvature in the road path ahead. This paper introduces a model-based predictive trajectory-tracking control structure using the previewed knowledge of path curvature and road friction. In the structure, path following and vehicle stabilization are incorporated through a model predictive controller. Meanwhile, long-range vehicle speed planning and tracking control are integrated to ensure the vehicle can slow down appropriately before encountering hazardous road conditions. This approach has two major advantages. First, the prior knowledge of the desired path is explicitly incorporated into the computation of control inputs. Second, the combined transmission of longitudinal and lateral tire forces is considered in the controller to avoid violation of tire force limits while keeping performance and stability guarantees. The efficacy of the algorithm is demonstrated through an application case where a vehicle navigates a sharply curving road with varying friction conditions, with results showing that the controller can drive a vehicle up to the handling limits and track the desired trajectory accurately.more » « less
-
Zonta, Daniele; Su, Zhongqing; Glisic, Branko (Ed.)With the rapid development of smart cities, interest in vehicle automation continues growing. Autonomous vehicles are becoming more and more popular among people and are considered to be the future of ground transportation. Autonomous vehicles, either with adaptive cruise control (ACC) or cooperative adaptive cruise control (CACC), provide many possibilities for smart transportation in a smart city. However, traditional vehicles and autonomous vehicles will have to share the same road systems until autonomous vehicles fully penetrate the market over the next few decades, which leads to conflicts because of the inconsistency of human drivers. In this paper, the performance of autonomous vehicles with ACC/CACC and traditional vehicles in mixed driver environments, at a signalized intersection, were evaluated using the micro-simulator VISSIM. In the simulation, the vehicles controlled by the ACC/CACC and Wiedemann 99 (W99) model represent the behavior of autonomous vehicles and human driver vehicles, respectively. For these two different driver environments, four different transport modes were comprehensively investigated: full light duty cars, full trucks, full motorcycles, and mixed conditions. In addition, ten different seed numbers were applied to each model to avoid coincidence. To evaluate the driving behavior of the human drivers and autonomous vehicles, this paper will compare the total number of stops, average velocity, and vehicle delay of each model at the signalized traffic intersection based on a real road intersection in Minnesota.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
