skip to main content


Title: Scenario2Vector: scenario description language based embeddings for traffic situations
A popular metric for measuring progress in autonomous driving has been the "miles per intervention". This is nowhere near a sufficient metric and it does not allow for a fair comparison between the capabilities of two autonomous vehicles (AVs). In this paper we propose Scenario2Vector - a Scenario Description Language (SDL) based embedding for traffic situations that allows us to automatically search for similar traffic situations from large AV data-sets. Our SDL embedding distills a traffic situation experienced by an AV into its canonical components - actors, actions, and the traffic scene. We can then use this embedding to evaluate similarity of different traffic situations in vector space. We have also created a first of its kind, Traffic Scenario Similarity (TSS) dataset which contains human ranking annotations for the similarity between traffic scenarios. Using the TSS data, we compare our SDL embedding -with textual caption based search methods such as Sentence2Vector. We find that Scenario2Vector outperforms Sentence2Vector by 13% ; and is a promising step towards enabling fair comparisons among AVs by inspecting how they perform in similar traffic situations. We hope that Scenario2Vector can have a similar impact to the AV community that Word2Vec/Sent2Vec have had in Natural Language Processing datasets.  more » « less
Award ID(s):
2046582
NSF-PAR ID:
10320138
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
ICCPS '21: Proceedings of the ACM/IEEE 12th International Conference on Cyber-Physical Systems
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Self-driving cars and trucks, autonomous vehicles (AVs), should not be accepted by regulatory bodies and the public until they have much higher confidence in their safety and reliability --- which can most practically and convincingly be achieved by testing. But existing testing methods are inadequate for checking the end-to-end behaviors of AV controllers against complex, real-world corner cases involving interactions with multiple independent agents such as pedestrians and human-driven vehicles. While test-driving AVs on streets and highways fails to capture many rare events, existing simulation-based testing methods mainly focus on simple scenarios and do not scale well for complex driving situations that require sophisticated awareness of the surroundings. To address these limitations, we propose a new fuzz testing technique, called AutoFuzz, which can leverage widely-used AV simulators' API grammars to generate semantically and temporally valid complex driving scenarios (sequences of scenes). To efficiently search for traffic violations-inducing scenarios in a large search space, we propose a constrained neural network (NN) evolutionary search method to optimize AutoFuzz. Evaluation of our prototype on one state-of-the-art learning-based controller, two rule-based controllers, and one industrial-grade controller in five scenarios shows that AutoFuzz efficiently finds hundreds of traffic violations in high-fidelity simulation environments. For each scenario, AutoFuzz can find on average 10-39% more unique traffic violations than the best-performing baseline method. Further, fine-tuning the learning-based controller with the traffic violations found by AutoFuzz successfully reduced the traffic violations found in the new version of the AV controller software. 
    more » « less
  2. Self-driving autonomous vehicles (AVs) have recently gained popularity as a research topic. The safety of AVs is exceptionally important as failure in the design of an AV could lead to catastrophic consequences. AV systems are highly heterogeneous with many different and complex components, so it is difficult to perform end-to-end testing. One solution to this dilemma is to evaluate AVs using simulated racing competition. In this thesis, we present a simulated autonomous racing competition, Generalized RAcing Intelligence Competition (GRAIC). To compete in GRAIC, participants need to submit their controller files which are deployed on a racing ego-vehicle on different race tracks. To evaluate the submitted controller, we also developed a testing pipeline, Autonomous System Operations (AutOps). AutOps is an automated, scalable, and fair testing pipeline developed using software engineering techniques such as continuous integration, containerization, and serverless computing. In order to evaluate the submitted controller in non-trivial circumstances, we populate the race tracks with scenarios, which are pre-defined traffic situations commonly seen in the real road. We present a dynamic scenario testing strategy that generates new scenarios based on results of the ego-vehicle passing through previous scenarios. 
    more » « less
  3. Autonomous Vehicles (AVs) are an emerging and highly impactful technology on today's roads. When assessing the performance of AVs, it is useful to study their improvement relative to common metrics such as fuel economy/emissions, safety, and congestion. But metrics of the vehicle's performance alone may not be complete; an AV that is affecting and reacting to a smart traffic light, for example, may improve its own performance, but may cause the same intersection to degrade the performance of other vehicles around the AV. Similar concerns arise in nearly all AV topics: platooning, light pre-emption, lane tracking, etc. Thus, the assessment of the vehicle's impacts on surrounding traffic is important, possibly even more important than the improvements enabled on the AV alone. But what boundary, or factors, define the vehicles, equipment, etc. “surrounding” an AV? The goal of this work is to characterize the boundary of vehicles “surrounding” an AV, referred to as Region of Influence, or ROI. Specifically, this work focuses on the problem that considering a perturbation is exerted into a traffic system, how far in time and space the perturbation from an AV’s decision can influence the surrounding system’s behavior. To achieve the goal, we utilized AIMSUN, a microscopic traffic simulator, to perform baseline and perturbed simulations. The ROI was evaluated by comparing trajectories of traffic surrounding the ego vehicle using different metrics, including difference in trajectories, Euclidian distance, rate of change of Euclidian distance, total number of lane changes over the whole simulation space versus time and total number of lane changes over the whole simulation time versus distance to ego vehicle. The results show that the ROI can be viewed from different perspectives using these metrics, and it is dependent on speed variance of the traffic. 
    more » « less
  4. null (Ed.)
    In this paper we provide a proof of principle of a new method for addressing the ethics of autonomous vehicles (AVs), the Data-Theories Method, in which vehicle crash data is combined with philosophical ethical theory to provide a guide to action for AV algorithm design. We use this method to model three scenarios in which an AV is exposed to risk on the road, and determine possible actions for the AV. We then examine how different philosophical perspectives on agent partiality, or the degree to which one can act in one’s own self-interest, might address each scenario. This method shows why modelling the ethics of AVs using data is essential. First, AVs may sometimes have options that human drivers do not, and designing AVs to mimic the most ethical human driver would not ensure that they do the right thing. Second, while ethical theories can often disagree about what should be done, disagreement can be reduced and compromises found with a more complete understanding of the AV’s choices and their consequences. Finally, framing problems around thought experiments may elicit preferences that are divergent with what individuals might prefer once they are provided with information about the real risks for a scenario. Our method provides a principled and empirical approach to productively address these problems and offers guidance on AV algorithm design. 
    more » « less
  5. Self-driving technology companies and the research community are accelerating the pace of use of machine learning longitudinal motion planning (mMP) for autonomous vehicles (AVs). This paper reviews the current state of the art in mMP, with an exclusive focus on its impact on traffic congestion. The paper identifies the availability of congestion scenarios in current datasets, and summarizes the required features for training mMP. For learning methods, the major methods in both imitation learning and non-imitation learning are surveyed. The emerging technologies adopted by some leading AV companies, such as Tesla, Waymo, and Comma.ai, are also highlighted. It is found that: (i) the AV industry has been mostly focusing on the long tail problem related to safety and has overlooked the impact on traffic congestion, (ii) the current public self-driving datasets have not included enough congestion scenarios, and mostly lack the necessary input features/output labels to train mMP, and (iii) although the reinforcement learning approach can integrate congestion mitigation into the learning goal, the major mMP method adopted by industry is still behavior cloning, whose capability to learn a congestion-mitigating mMP remains to be seen. Based on the review, the study identifies the research gaps in current mMP development. Some suggestions for congestion mitigation for future mMP studies are proposed: (i) enrich data collection to facilitate the congestion learning, (ii) incorporate non-imitation learning methods to combine traffic efficiency into a safety-oriented technical route, and (iii) integrate domain knowledge from the traditional car-following theory to improve the string stability of mMP. 
    more » « less