An official website of the United States government
Synchronizing decisions across multiple agents in realistic settings is problematic since it requires agents to wait for other agents to terminate and communicate about termination reliably. Ideally, agents should learn and execute asynchronously instead. Such asynchronous methods also allow temporally extended actions that can take different amounts of time based on the situation and action executed. Unfortunately, current policy gradient methods are not applicable in asynchronous settings, as they assume that agents synchronously reason about action selection at every time step. To allow asynchronous learning and decision-making, we formulate a set of asynchronous multi-agent actor-critic methods that allow agents to directly optimize asynchronous policies in three standard training paradigms: decentralized learning, centralized learning, and centralized training for decentralized execution. Empirical results (in simulation and hardware) in a variety of realistic domains demonstrate the superiority of our approaches in large multi-agent problems and validate the effectiveness of our algorithms for learning high-quality and asynchronous solutions.
more »
« less
An Asynchronous Multi-Body Simulation Framework for Real-Time Dynamics, Haptics and Learning with Application to Surgical Robots
Surgical robots for laparoscopy consist of several patient side slave manipulators that are controlled via surgeon operated master telemanipulators. Commercial surgical robots do not perform any sub-tasks - even of repetitive or noninvasive nature - autonomously or provide intelligent assistance. While this is primarily due to safety and regulatory reasons, the state of such automation intelligence also lacks the reliability and robustness for use in high-risk applications. Recent developments in continuous control using Artificial Intelligence and Reinforcement Learning have prompted growing research interest in automating mundane sub-tasks. To build on this, we present an inspired Asynchronous Framework which incorporates realtime dynamic simulation - manipulable with the masters of a surgical robot and various other input devices - and interfaces with learning agents to train and potentially allow for the execution of shared sub-tasks. The scope of this framework is generic to cater to various surgical (as well as non-surgical) training and control applications. This scope is demonstrated by examples of multi-user and multi-manual applications which allow for realistic interactions by incorporating distributed control, shared task allocation and a well-defined communication pipe-line for learning agents. These examples are discussed in conjunction with the design philosophy, specifications, system-architecture and metrics of the Asynchronous Framework and the accompanying Simulator. We show the stability of Simulator while achieving real-time dynamic simulation and interfacing with several haptic input devices and a training agent at the same time.
more »
« less
- Award ID(s):
- 1637759
- PAR ID:
- 10207707
- Date Published:
- Journal Name:
- 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Page Range / eLocation ID:
- 6268 to 6275
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Robot Dynamic Simulators offer convenient implementation and testing of physical robots, thus accelerating research and development. While existing simulators support most real-world robots with serially linked kinematic and dynamic chains, they offer limited or conditional support for complex closed-loop robots. On the other hand, many of the underlying physics computation libraries that these simulators employ support closed-loop kinematic chains and redundant mechanisms. Such mechanisms are often utilized in surgical robots to achieve constrained motions (e.g., the remote center of motion (RCM)). To deal with such robots, we propose a new simulation framework based on a front-end description format and a robust real-time dynamic simulator. Although this study focuses on surgical robots, the proposed format and simulator are applicable to any type of robot. In this manuscript, we describe the philosophy and implementation of the front-end description format and demonstrate its performance and the simulator's capabilities using simulated models of real-world surgical robots.more » « less
-
We present an open-source framework that provides a low barrier to entry for real-time simulation, visualization, and interactive manipulation of user-specifiable soft-bodies, environments, and robots (using a human-readable front-end interface). The simulated soft-bodies can be interacted by a variety of input interface devices including commercially available haptic devices, game controllers, and the Master Tele-Manipulators (MTMs) of the da Vinci Research Kit (dVRK) with real-time haptic feedback. We propose this framework for carrying out multi-user training, user-studies, and improving the control strategies for manipulation problems. In this paper, we present the associated challenges to the development of such a framework and our proposed solutions. We also demonstrate the performance of this framework with examples of soft-body manipulation and interaction with various input devices.more » « less
-
This paper proposes a low-cost interface and refined digital twin for the Raven-II surgical robot. Previous simulations of the Raven-II, e.g. via the Asynchronous Multibody Framework (AMBF), presented salient drawbacks, including control inputs inconsistent with Raven-II software, and lack of stable, high-fidelity physical contact simulations. This work bridges both of these gaps, both (1) enabling robust, simulated contact mechanics for dynamic physical interactions with the Raven-II, and (2) developing a universal input format for both simulated and physical platforms. The method furthermore proposes a low cost, commodity game-controller interface for controlling both virtual and real realizations of Raven-II, thus greatly reducing the barrier to access for Raven-II research and collaboration. Overall, this work aims to eliminate the inconsistencies between simulated and real representations of the Raven-II. Such a development can expand the reach of surgical robotics research. Namely, providing end-to-end transparency between the simulated AMBF and physical Raven-II platforms enables a software testbed previously unavailable, e.g. for training real surgeons, for creating digital synthetic datasets, or for prototyping novel architectures like shared control strategies. Experiments validate this transparency by comparing joint trajectories between digital twin and physical testbed given identical inputs. This work may be extended and incorporated into recent efforts in developing modular or common software infrastructures for both simulation and control of real robotic devices, such as the Collaborative Robotics Toolkit (CRTK).more » « less
-
null (Ed.)In the field of soft robotics, harnessing the nonlinear dynamics of soft and compliant bodies as a computational resource to enable embodied intelligence and control is known as morphological computation. Physical reservoir computing (PRC) is a true instance of morphological computation wherein; a physical nonlinear dynamic system is used as a fixed reservoir to perform complex computational tasks. These dynamic reservoirs can be used to approximate nonlinear dynamical systems and even perform machine learning tasks. By numerical simulation, this study illustrates that an origami meta-material can also be used as a dynamic reservoir for pattern generation, output modulation, and input sensing. These results could pave the way for intelligently designed origami-based robots that interact with the environment through a distributed network of sensors and actuators. This embodied intelligence will enable the next generations of soft robots to autonomously coordinate and modulate their activities, such as locomotion gait generation and limb manipulation while resisting external disturbances.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/IROS40897.2019.8968594
