skip to main content


Title: A Fog Robotics Approach to Deep Robot Learning: Application to Object Recognition and Grasp Planning in Surface Decluttering
The growing demand of industrial, automotive and service robots presents a challenge to the centralized Cloud Robotics model in terms of privacy, security, latency, bandwidth, and reliability. In this paper, we present a ‘Fog Robotics’ approach to deep robot learning that distributes compute, storage and networking resources between the Cloud and the Edge in a federated manner. Deep models are trained on non-private (public) synthetic images in the Cloud; the models are adapted to the private real images of the environment at the Edge within a trusted network and subsequently, deployed as a service for low-latency and secure inference/prediction for other robots in the network. We apply this approach to surface decluttering, where a mobile robot picks and sorts objects from a cluttered floor by learning a deep object recognition and a grasp planning model. Experiments suggest that Fog Robotics can improve performance by sim-to-real domain adaptation in comparison to exclusively using Cloud or Edge resources, while reducing the inference cycle time by 4 to successfully declutter 86% of objects over 213 attempts.  more » « less
Award ID(s):
1838833
NSF-PAR ID:
10111294
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Proceedings - IEEE International Conference on Robotics and Automation
ISSN:
1050-4729
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. As many robot automation applications increasingly rely on multi-core processing or deep-learning models, cloud computing is becoming an attractive and economically viable resource for systems that do not contain high computing power onboard. Despite its immense computing capacity, it is often underused by the robotics and automation community due to lack of expertise in cloud computing and cloud-based infrastructure. Fog Robotics balances computing and data between cloud edge devices. We propose a software framework, FogROS, as an extension of the Robot Operating System (ROS), the de-facto standard for creating robot automation applications and components. It allows researchers to deploy components of their software to the cloud with minimal effort, and correspondingly gain access to additional computing cores, GPUs, FPGAs, and TPUs, as well as predeployed software made available by other researchers. FogROS allows a researcher to specify which components of their software will be deployed to the cloud and to what type of computing hardware. We evaluate FogROS on 3 examples: (1) simultaneous localization and mapping (ORB-SLAM2), (2) Dexterity Network (Dex-Net) GPU-based grasp planning, and (3) multi-core motion planning using a 96-core cloud-based server. In all three examples, a component is deployed to the cloud and accelerated with a small change in system launch configuration, while incurring additional latency of 1.2 s, 0.6 s, and 0.5 s due to network communication, the computation speed is improved by 2.6x, 6.0x and 34.2x, respectively. 
    more » « less
  2. Mobility, power, and price points often dictate that robots do not have sufficient computing power on board to run contemporary robot algorithms at desired rates. Cloud computing providers such as AWS, GCP, and Azure offer immense computing power on demand, but tapping into that power from a robot is non-trivial. We present FogROS2, an open-source platform to facilitate cloud and fog robotics that is compatible with the emerging Robot Operating System 2 (ROS 2) standard. FogROS2 is completely redesigned and distinct from its predecessor FogROS1 in 9 ways, and has lower latency, overhead, and startup times; improved usability, and additional automa-tion, such as region and computer type selection. Additionally, FogROS2 was added to the official distribution of ROS 2, gaining performance, timing, and additional improvements associated with ROS 2. In examples, FogROS2 reduces SLAM latency by 50 %, reduces grasp planning time from 14 s to 1.2 s, and speeds up motion planning 28x. When compared to FogROS1, FogROS2 reduces network utilization by up to 3.8x, improves startup time by 63 %, and network round-trip latency by 97 %for images using video compression. The source code, examples, and documentation for FogROS2 are available at https://github.com/BerkeleyAutomation/FogROS2, and is available through the official ROS 2 repository at https://index.ros.org/p/fogros2/ 
    more » « less
  3. Edge machine learning can deliver low-latency and private artificial intelligent (AI) services for mobile devices by leveraging computation and storage resources at the network edge. This paper presents an energy-efficient edge processing framework to execute deep learning inference tasks at the edge computing nodes whose wireless connections to mobile devices are prone to channel uncertainties. Aimed at minimizing the sum of computation and transmission power consumption with probabilistic quality-of-service (QoS) constraints, we formulate a joint inference tasking and downlink beamforming problem that is characterized by a group sparse objective function. We provide a statistical learning based robust optimization approach to approximate the highly intractable probabilistic-QoS constraints by nonconvex quadratic constraints, which are further reformulated as matrix inequalities with a rank-one constraint via matrix lifting. We design a reweighted power minimization approach by iteratively reweighted ℓ1 minimization with difference-of-convex-functions (DC) regularization and updating weights, where the reweighted approach is adopted for enhancing group sparsity whereas the DC regularization is designed for inducing rank-one solutions. Numerical results demonstrate that the proposed approach outperforms other state-of-the-art approaches. 
    more » « less
  4. null (Ed.)
    In recent years, the addition of billions of Internet of Thing (IoT) device spawned a massive demand for computing service near the edge of the network. Due to latency, limited mobility, and location awareness, cloud computing is not capable enough to serve these devices. As a result, the focus is shifting more towards distributed platform service to put ample computing power near the edge of the networks. Thus, paradigms such as Fog and Edge computing are gaining attention from researchers as well as business stakeholders. Fog computing is a new computing paradigm, which places computing nodes in between the Cloud and the end user to reduce latency and increase availability. As an emerging technology, Fog computing also brings newer security challenges for the stakeholders to solve. Before designing the security models for Fog computing, it is better to understand the existing threats to Fog computing. In this regard, a thorough threat model can significantly help to identify these threats. Threat modeling is a sophisticated engineering process by which a computer-based system is analyzed to discover security flaws. In this paper, we applied two popular security threat modeling processes - CIAA and STRIDE - to identify and analyze attackers, their capabilities and motivations, and a list of potential threats in the context of Fog computing. We posit that such a systematic and thorough discussion of a threat model for Fog computing will help security researchers and professionals to design secure and reliable Fog computing systems. 
    more » « less
  5. In recent years, the addition of billions of Internet of Thing (IoT) device spawned a massive demand for computing service near the edge of the network. Due to latency, limited mobility, and location awareness, cloud computing is not capable enough to serve these devices. As a result, the focus is shifting more towards distributed platform service to put ample com- puting power near the edge of the networks. Thus, paradigms such as Fog and Edge computing are gaining attention from researchers as well as business stakeholders. Fog computing is a new computing paradigm, which places computing nodes in between the Cloud and the end user to reduce latency and increase availability. As an emerging technology, Fog computing also brings newer security challenges for the stakeholders to solve. Before designing the security models for Fog computing, it is better to understand the existing threats to Fog computing. In this regard, a thorough threat model can significantly help to identify these threats. Threat modeling is a sophisticated engineering process by which a computer-based system is analyzed to discover security flaws. In this paper, we applied two popular security threat modeling processes – CIAA and STRIDE – to identify and analyze attackers, their capabilities and motivations, and a list of potential threats in the context of Fog computing. We posit that such a systematic and thorough discussion of a threat model for Fog computing will help security researchers and professionals to design secure and reliable Fog computing systems. 
    more » « less