More Like this

1. Intelligent Vision-based Autonomous Ship Landing of VTOL UAVs

   Lee, B. ( December 2022 , Journal of the American Helicopter Society)

   The paper discusses an intelligent vision-based control solution for autonomous tracking and landing of Vertical Take-Off and Landing (VTOL) capable Unmanned Aerial Vehicles (UAVs) on ships without utilizing GPS signal. The central idea involves automating the Navy helicopter ship landing procedure where the pilot utilizes the ship as the visual reference for long-range tracking; however, refers to a standardized visual cue installed on most Navy ships called the ”horizon bar” for the final approach and landing phases. This idea is implemented using a uniquely designed nonlinear controller integrated with machine vision. The vision system utilizes machine learning based object detection for long-range ship tracking and classical computer vision for the estimation of aircraft relative position and orientation utilizing the horizon bar during the final approach and landing phases. The nonlinear controller operates based on the information estimated by the vision system and has demonstrated robust tracking performance even in the presence of uncertainties. The developed autonomous ship landing system was implemented on a quad-rotor UAV equipped with an onboard camera, and approach and landing were successfully demonstrated on a moving deck, which imitates realistic ship deck motions. Extensive simulations and flight tests were conducted to demonstrate vertical landing safety, tracking capability, and landing accuracy. The video of the real-world experiments and demonstrations is available at this URL. 

   more » « less
2. A Deep Reinforcement Learning Control Strategy for Vision-based Ship Landing of Vertical Flight Aircraft

   https://doi.org/10.2514/6.2021-3218  

   Lee, Bochan ; Saj, Vishnu ; Benedict, Moble ; Kalathil, Dileep ( July 2021 , AIAA AVIATION 2021 FORUM)

   The paper discusses a deep reinforcement learning (RL) control strategy for fully autonomous vision-based approach and landing of vertical take-off and landing (VTOL) capable unmanned aerial vehicles (UAVs) on ships in the presence of disturbances such as wind gusts. The automation closely follows the Navy helicopter ship landing procedure and therefore, it detects a horizon bar that is installed on most Navy ships as a visual aid for pilots by applying uniquely developed computer vision techniques. The vision system utilizes the detected corners of the horizon bar and its known dimensions to estimate the relative position and heading angle of the aircraft. A deep RL-based controller was coupled with the vision system to ensure a safe and robust approach and landing at the proximity of the ship where the airflow is highly turbulent. The vision and RL-based control system was implemented on a quadrotor UAV and flight tests were conducted where the UAV approached and landed on a sub-scale ship platform undergoing 6 degrees of freedom deck motions in the presence of wind gusts. Simulations and flight tests confirmed the superior disturbance rejection capability of the RL controller when subjected to sudden 5 m/s wind gusts in different directions. Specifically, it was observed during flight tests that the deep RL controller demonstrated a 50% reduction in lateral drift from the flight path and 3 times faster disturbance rejection in comparison to a nonlinear proportional-integral-derivative controller. 

   more » « less
3. Controls for a nonlinear system arising in vision‐based landing of airliners

   https://doi.org/10.1002/rnc.5332  

   Burlion, Laurent ; Gibert, Victor ; Malisoff, Michael ; Mazenc, Frederic ( December 2020 , International Journal of Robust and Nonlinear Control)

   We use a novel backstepping method to solve a stabilization problem for a nonlinear system with delayed sampled outputs that are not accurately measured. We provide an application to a system arising in vision‐based landing of airliners that includes coupling between the lateral and longitudinal dynamics, for which we provide performance guarantees in the presence of the delay, nonlinearity, and sampling. Our major contributions are (a) designs of lateral and longitudinal controls for our nonlinear model of an aircraft landing on an unequipped runway, (b) mathematical proofs that our controls ensure that the aircraft being modeled achieves desired alignment with the runway during its align phase, under sampling and delays that arise from image processing of visual information, and (c) comparative simulations exhibiting considerable improvement in control performance compared with previous methods that did not take the coupling of the dynamics or imprecise delayed sampled measurements into account.

    

   more » « less
4. Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis

   Kaustav Chakraborty, Somil Bansal ( October 2023 , IEEE Robotics and Automation Letters)

   Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Understanding when these controllers might lead to system safety violations is important for their integration in safety-critical applications and engineering corrective safety measures for the system. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers, i.e., the visual inputs that lead to closed-loop safety violations. However, these techniques do not scale well to the scenarios involving high-dimensional and complex visual inputs, such as RGB images. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem. Our approach blends simulation-based analysis with HJ reachability methods to compute an approximation of the backward reachable tube (BRT) of the system, i.e., the set of unsafe states for the system under vision-based controllers. Utilizing the BRT, we can tractably and systematically find the system states and corresponding visual inputs that lead to closed-loop failures. These visual inputs can be subsequently analyzed to find the input characteristics that might have caused the failure. Besides its scalability to high-dimensional visual inputs, an explicit computation of BRT allows the proposed approach to capture non-trivial system failures that are difficult to expose via random simulations. We demonstrate our framework on two case studies involving an RGB image-based neural network controller for (a) autonomous indoor navigation, and (b) autonomous aircraft taxiing. 

   more » « less
5. Fooling Detection Alone is Not Enough: First Adversarial Attack against Multiple Object Tracking

   Jia, Yunhan ; Lu, Yantao ; Shen, Junjie ; Chen, Qi A ; Zhong, Zhenyu ; Wei, Tao ( April 2020 , International Conference on Learning Representations (ICLR))

   Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%. 

   more » « less