Drones have increasingly collaborated with human workers in some workspaces, such as warehouses. The failure of a drone flight may bring potential risks to human beings' life safety during some aerial tasks. One of the most common flight failures is triggered by damaged propellers. To quickly detect physical damage to propellers, recognise risky flights, and provide early warnings to surrounding human workers, a new and comprehensive fault diagnosis framework is presented that uses only the audio caused by propeller rotation without accessing any flight data. The diagnosis framework includes three components: leverage convolutional neural networks, transfer learning, and Bayesian optimisation. Particularly, the audio signal from an actual flight is collected and transferred into time–frequency spectrograms. First, a convolutional neural network‐based diagnosis model that utilises these spectrograms is developed to identify whether there is any broken propeller involved in a specific drone flight. Additionally, the authors employ Monte Carlo dropout sampling to obtain the inconsistency of diagnostic results and compute the mean probability score vector's entropy (uncertainty) as another factor to diagnose the drone flight. Next, to reduce data dependence on different drone types, the convolutional neural network‐based diagnosis model is further augmented by transfer learning. That is, the knowledge of a well‐trained diagnosis model is refined by using a small set of data from a different drone. The modified diagnosis model has the ability to detect the broken propeller of the second drone. Thirdly, to reduce the hyperparameters' tuning efforts and reinforce the robustness of the network, Bayesian optimisation takes advantage of the observed diagnosis model performances to construct a Gaussian process model that allows the acquisition function to choose the optimal network hyperparameters. The proposed diagnosis framework is validated via real experimental flight tests and has a reasonably high diagnosis accuracy.
EVPropNet: Detecting Drones By Finding Propellers For Mid-Air Landing And Following
The rapid rise of accessibility of unmanned aerial vehicles or drones pose a threat to general security and confidentiality. Most of the commercially available or custom-built drones are multi-rotors and are comprised of multiple propellers. Since these propellers rotate at a high-speed, they are generally the fastest moving parts of an image and cannot be directly "seen" by a classical camera without severe motion blur. We utilize a class of sensors that are particularly suitable for such scenarios called event cameras, which have a high temporal resolution, low-latency, and high dynamic range. In this paper, we model the geometry of a propeller and use it to generate simulated events which are used to train a deep neural network called EVPropNet to detect propellers from the data of an event camera. EVPropNet directly transfers to the real world without any fine-tuning or retraining. We present two applications of our network: (a) tracking and following an unmarked drone and (b) landing on a near-hover drone. We successfully evaluate and demonstrate the proposed approach in many real-world experiments with different propeller shapes and sizes. Our network can detect propellers at a rate of 85.1% even when 60% of the propeller is occluded and can run at upto 35Hz on a 2W power budget. To our knowledge, this is the first deep learning-based solution for detecting propellers (to detect drones). Finally, our applications also show an impressive success rate of 92% and 90% for the tracking and landing tasks respectively.
more »
« less
- Award ID(s):
- 1824198
- NSF-PAR ID:
- 10309536
- Date Published:
- Journal Name:
- Robotics Science and Systems
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
As the drone becomes widespread in numerous crucial applications with many powerful functionalities (e.g., reconnaissance and mechanical trigger), there are increasing cases related to misused drones for unethical even criminal activities. Therefore, it is of paramount importance to identify these malicious drones and track their origins using digital forensics. Traditional drone identification techniques for forensics (e.g., RF communication, ID landmarks using a camera, etc.) require high compliance of drones. However, malicious drones will not cooperate or even spoof these identification techniques. Therefore, we present an exploration for a reliable and passive identification approach based on unique hardware traits in drones directly (e.g., analogous to the fingerprint and iris in humans) for forensics purposes. Specifically, we investigate and model the behavior of the parasitic electronic elements under RF interrogation, a particular passive parasitic response modulated by an electronic system on drones, which is distinctive and unlikely to counterfeit. Based on this theory, we design and implement DroneTrace, an end-to-end reliable and passive identification system toward digital drone forensics. DroneTrace comprises a cost-effective millimeter-wave (mmWave) probe, a software framework to extract and process parasitic responses, and a customized deep neural network (DNN)-based algorithm to analyze and identify drones. We evaluate the performance of DroneTrace with 36 commodity drones. Results show that DroneTrace can identify drones with the accuracy of over 99% and an equal error rate (EER) of 0.009, under a 0.1-second sensing time budget. Moreover, we test the reliability, robustness, and performance variation under a set of real-world circumstances, where DroneTrace maintains accuracy of over 98%. DroneTrace is resilient to various attacks and maintains functionality. At its best, DroneTrace has the capacity to identify individual drones at the scale of 104 with less than 5% error.more » « less
-
Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates and insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision 1 .more » « less
-
Existing approaches for autonomous control of pan-tilt-zoom (PTZ) cameras use multiple stages where object detection and localization are performed separately from the control of the PTZ mechanisms. These approaches require manual labels and suffer from performance bottlenecks due to error propagation across the multi-stage flow of information. The large size of object detection neural networks also makes prior solutions infeasible for real-time deployment in resource-constrained devices. We present an end-to-end deep reinforcement learning (RL) solution called Eagle1 to train a neural network policy that directly takes images as input to control the PTZ camera. Training reinforcement learning is cumbersome in the real world due to labeling effort, runtime environment stochasticity, and fragile experimental setups. We introduce a photo-realistic simulation framework for training and evaluation of PTZ camera control policies. Eagle achieves superior camera control performance by maintaining the object of interest close to the center of captured images at high resolution and has up to 17% more tracking duration than the state-of-the-art. Eagle policies are lightweight (90x fewer parameters than Yolo5s) and can run on embedded camera platforms such as Raspberry PI (33 FPS) and Jetson Nano (38 FPS), facilitating real-time PTZ tracking for resource-constrained environments. With domain randomization, Eagle policies trained in our simulator can be transferred directly to real-world scenarios2.more » « less
-
Internet of drones (IoD), employing drones as the internet of things (IoT) devices, brings flexibility to IoT networks and has been used to provision several applications (e.g., object tracking and traffic surveillance). The explosive growth of users and IoD applications injects massive traffic into IoD networks, hence causing congestions and reducing the quality of service (QoS). In order to improve the QoS, caching at IoD gateways is a promising solution which stores popular IoD data and sends them directly to the users instead of activating drones to transmit the data; this reduces the traffic in IoD networks. In order to fully utilize the storage-limited caches, appropriate content placement decisions should be made to determine which data should be cached. On the other hand, appropriate drone association strategies, which determine the serving IoD gateway for each drone, help distribute the network traffic properly and hence improve the QoS. In our work, we consider a joint optimization of drone association and content placement problem aimed at maximizing the average data transfer rate. This problem is formulated as an integer linear programming (ILP) problem. We then design the Drone Association and Content Placement (DACP) algorithm to solve this problem with low computational complexity. Extensive simulations demonstrate the performance of DACP.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.15607/RSS.2021.XVII.074
