More Like this

1. Autonomous Learning in a Pseudo-Episodic Physical Environment

   https://doi.org/10.1007/s10846-022-01577-5  

   Haughn, Kevin P.; Inman, Daniel J. (February 2022, Journal of Intelligent & Robotic Systems)

   Abstract Forpractical considerations reinforcement learning has proven to be a difficult task outside of simulation when applied to a physical experiment. Here we derive an optional approach to model free reinforcement learning, achieved entirely online, through careful experimental design and algorithmic decision making. We design a reinforcement learning scheme to implement traditionally episodic algorithms for an unstable 1-dimensional mechanical environment. The training scheme is completely autonomous, requiring no human to be present throughout the learning process. We show that the pseudo-episodic technique allows for additional learning updates with off-policy actor-critic and experience replay methods. We show that including these additional updates between periods of traditional training episodes can improve speed and consistency of learning. Furthermore, we validate the procedure in experimental hardware. In the physical environment, several algorithm variants learned rapidly, each surpassing baseline maximum reward. The algorithms in this research are model free and use only information obtained by an onboard sensor during training. 

   more » « less
2. Real-time Adversarial Image Perturbations for Autonomous Vehicles using Reinforcement Learning

   https://doi.org/10.1145/3712303  

   Yoon, Hyung-Jin; Holmes, Ryan; Jafarnejadsani, Hamidreza; Voulgaris, Petros (March 2025, ACM Transactions on Cyber-Physical Systems)

   The deep neural network (DNN) model for computer vision tasks (object detection and classification) is widely used in autonomous vehicles, such as driverless cars and unmanned aerial vehicles. However, DNN models are shown to be vulnerable to adversarial image perturbations. The generation of adversarial examples against inferences of DNNs has been actively studied recently. The generation typically relies on optimizations taking an entire image frame as the decision variable. Hence, given a new image, the computationally expensive optimization needs to start over as there is no learning between the independent optimizations. Very few approaches have been developed for attacking online image streams while taking into account the underlying physical dynamics of autonomous vehicles, their mission, and the environment. The article presents a multi-level reinforcement learning framework that can effectively generate adversarial perturbations to misguide autonomous vehicles’ missions. In the existing image attack methods against autonomous vehicles, optimization steps are repeated for every image frame. This framework removes the need for fully converged optimization at every frame. Using multi-level reinforcement learning, we integrate a state estimator and a generative adversarial network that generates the adversarial perturbations. Due to the reinforcement learning agent consisting of state estimator, actor, and critic that only uses image streams, the proposed framework can misguide the vehicle to increase the adversary’s reward without knowing the states of the vehicle and the environment. Simulation studies and a robot demonstration are provided to validate the proposed framework’s performance. 

   more » « less
3. Soaring like a bird via reinforcement learning in the field

   Reddy, G. (October 2018, 07 Nature)

   Soaring birds often rely on ascending thermal plumes in the atmosphere as they search for prey or migrate across large distances. The landscape of convective currents is turbulent and rapidly shifts on timescales of a few minutes as thermals constantly form, disintegrate, or are transported away by the wind. How soaring birds find and navigate thermals within this complex landscape is unknown. Reinforcement learning can be used to find an effective navigational strategy as a sequence of decisions taken in response to environmental cues. Reinforcement learning was applied to train gliders in the field to autonomously navigate atmospheric thermals. Gliders of two-meter wingspan were equipped with a flight controller that enabled an on-board implementation of autonomous flight policies via precise control over their bank angle and pitch. Learning is severely challenged by a multitude of physical effects and the unpredictability of the natural environment. A navigational strategy was determined solely from the experiences collected over several days in the field using exploratory behavioral policies. Bird-like performance was achieved and several viable biological mechanosensory cues were identified for soaring birds, which are also directly applicable to the development of autonomous soaring vehicles. 

   more » « less
4. Knowledge Guided Two-player Reinforcement Learning for Cyber Attacks and Defenses

   https://doi.org/10.1109/icmla55696.2022.00213  

   Piplai, Aritran; Anoruo, Mike; Fasaye, Kayode; Joshi, Anupam; Finin, Tim; Ridley, Ahmad (December 2022, 21st IEEE International Conference on Machine Learning and Applications)

   Cyber defense exercises are an important avenue to understand the technical capacity of organizations when faced with cyber-threats. Information derived from these exercises often leads to finding unseen methods to exploit vulnerabilities in an organization. These often lead to better defense mechanisms that can counter previously unknown exploits. With recent developments in cyber battle simulation platforms, we can generate a defense exercise environment and train reinforcement learning (RL) based autonomous agents to attack the system described by the simulated environment. In this paper, we describe a two-player game-based RL environment that simultaneously improves the performance of both the attacker and defender agents. We further accelerate the convergence of the RL agents by guiding them with expert knowledge from Cybersecurity Knowledge Graphs on attack and mitigation steps. We have implemented and integrated our proposed approaches into the CyberBattleSim system. 

   more » « less
5. Towards multi-agent autonomous racing with the Deepracing framework.

   Weiss, Trent; Chrosniak, John; Behl, Madhur (May 2021, International Conference on Robotics and Automation (ICRA) - Workshop on Opportunities and Challenges with Autonomous Racing)

   Multi-agent autonomous racing is a challenging problem for autonomous vehicles due to the split-second, and complex decisions that vehicles must continuously make during a race. The presence of other agents on the track requires continuous monitoring of the ego vehicle’s surroundings, and necessitates predicting the behavior of other vehicles so the ego can quickly react to a changing environment with informed decisions. In our previous work we have developed the DeepRacing AI framework for autonomous formula one racing. Our DeepRacing framework was the first implementation to use the highly photorealisitc Formula One game as a simulation testbed for autonomous racing. We have successfully demonstrated single agent high speed autonomous racing using Bezier curve trajectories. In this paper, we extend the capabilities of the DeepRacing framework towards multi-agent autonomous racing. To do so, we first develop and learn a virtual camera model from game data that the user can configure to emulate the presence of a camera sensor on the vehicle. Next we propose and train a deep recurrent neural network that can predict the future poses of opponent agents in the field of view of the virtual camera using vehicles position, velocity, and heading data with respect to the ego vehicle racecar. We demonstrate early promising results for both these contributions in the game. These added features will extend the DeepRacing framework to become more suitable for multi-agent autonomous racing algorithm development 

   more » « less