An official website of the United States government
We propose a novel model-based reinforcement learning algorithm—Dynamics Learning and predictive control with Parameterized Actions (DLPA)—for Parameterized Action Markov Decision Processes (PAMDPs). The agent learns a parameterized-action-conditioned dynamics model and plans with a modified Model Predictive Path Integral control. We theoretically quantify the difference between the generated trajectory and the optimal trajectory during planning in terms of the value they achieved through the lens of Lipschitz Continuity. Our empirical results on several standard benchmarks show that our algorithm achieves superior sample efficiency and asymptotic performance than state-of-the-art PAMDP methods.
more »
« less
Stable Heteroclinic Channel-Based Movement Primitives: Tuning Trajectories Using Saddle Parameters
Dynamic systems which underlie controlled systems are expected to increase in complexity as robots, devices, and connected networks become more intelligent. While classical stable systems converge to a stable point (a sink), another type of stability is to consider a stable path rather than a single point. Such stable paths can be made of saddle points that draw in trajectories from certain regions, and then push the trajectory toward the next saddle point. These chains of saddles are called stable heteroclinic channels (SHCs) and can be used in robotic control to represent time sequences. While we have previously shown that each saddle is visualizable as a trajectory waypoint in phase space, how to increase the fidelity of the trajectory was unclear. In this paper, we hypothesized that the waypoints can be individually modified to locally vary fidelity. Specifically, we expected that increasing the saddle value (ratio of saddle eigenvalues) causes the trajectory to slow to more closely approach a particular saddle. Combined with other parameters that control speed and magnitude, a system expressed with an SHC can be modified locally, point by point, without disrupting the rest of the path, supporting their use in motion primitives. While some combinations can enable a trajectory to better reach into corners, other combinations can rotate, distort, and round the trajectory surrounding the modified saddle. Of the system parameters, the saddle value provides the most predictable tunability across 3 orders of magnitude.
more »
« less
- Award ID(s):
- 2047330
- PAR ID:
- 10611301
- Publisher / Repository:
- MDPI
- Date Published:
- Journal Name:
- Applied Sciences
- Volume:
- 14
- Issue:
- 6
- ISSN:
- 2076-3417
- Page Range / eLocation ID:
- 2523
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Stable heteroclinic channels for controlling a simulated aquatic serpentine robot in narrow crevicesStable Heteroclinic Channels (SHCs) are dynamical systems composed of connected saddle equilibria. This work demonstrates a control system that combines SHCs with movement primitives to enable swimming in a simulated six segment snake robot. We identify control system parameters for lateral undulation, where all joints oscillate with the same amplitude, and anguilliform swimming, where joint amplitudes increase linearly from the head to the tail. Swimming speed is improved by learning SHC movement primitive parameters. We also propose a method for adapting the gait amplitude and frequency with tactile sensor input to accommodate obstacles. Then, we evaluate the relationship between SHC movement primitive parameters and the resulting trajectories. The swimming speed and efficiency of SHC controllers for each gait are compared against a conventional serpenoid controller, which derives joint trajectories from sinusoids. Controllers are evaluated first in an unobstructed environment, then in straight passages of various widths, and finally in 65 randomly generated uneven channels. We find that the amplitudes of joint oscillations scale proportionally with the SHC controller parameters. Due to gait optimization, as well as adaptive amplitude and frequency in response to tactile input, the learned SHC control system exhibits an average 28.8% greater speed than a serpenoid controller that only adapts amplitude during contact. This research demonstrates that SHCs benefit from intuitive tuning like serpenoid control, while also effectively incorporating sensory information to generate smooth kinematic trajectories.more » « less
-
This paper considers the problem of understanding the exit time for trajectories of gradient-related first-order methods from saddle neighborhoods under some initial boundary conditions. Given the ‘flat’ geometry around saddle points, first-order methods can struggle to escape these regions in a fast manner due to the small magnitudes of gradients encountered. In particular, while it is known that gradient-related first-order methods escape strict-saddle neighborhoods, existing analytic techniques do not explicitly leverage the local geometry around saddle points in order to control behavior of gradient trajectories. It is in this context that this paper puts forth a rigorous geometric analysis of the gradient-descent method around strict-saddle neighborhoods using matrix perturbation theory. In doing so, it provides a key result that can be used to generate an approximate gradient trajectory for any given initial conditions. In addition, the analysis leads to a linear exit-time solution for gradient-descent method under certain necessary initial conditions, which explicitly bring out the dependence on problem dimension, conditioning of the saddle neighborhood, and more, for a class of strict-saddle functions.more » « less
-
This paper develops a control co-design (CCD) framework to simultaneously optimize the spacecraft’s trajectory and onboard system (rocket engine) and quantify its benefit. An open-loop optimal control problem (two-finite burn Mars missions) is used as the benchmark, and the engine design considers the combustion equilibrium and nozzle geometry. The objective function is the fuel burn. The design variables are the trajectory control parameters (such as burn times, burn directions, and time of flight), initial fuel mass, and engine design parameters (such as throat area, mixture ratio, and chamber pressure). The constraints include the final velocities and positions of spacecraft. Single-point optimizations are conducted for three departure dates in May, July, and September 2020. A multipoint optimization is also performed to balance the engine performance for these dates with 49 design variables and 20 constraints. It is found that the CCD optimizations exhibit 22–28% more fuel burn reduction than the trajectory-only optimization with fixed engine parameters and 16–20% more fuel burn reduction than the decoupled trajectory-engine optimization. The proposed CCD optimization framework can be extended to more spacecraft trajectory control parameters and onboard systems and has the potential to design more efficient spacecraft missions.more » « less
-
In this paper, we consider using Schur complements to design preconditioners for twofold and block tridiagonal saddle point problems. One type of the preconditioners are based on the nested (or recursive) Schur complement, the other is based on an additive type Schur complement after permuting the original saddle point systems. We analyze different preconditioners incorporating the exact Schur complements. We show that some of them will lead to positively stable preconditioned systems if proper signs are selected in front of the Schur complements. These positive-stable preconditioners outperform other preconditioners if the Schur complements are further approximated inexactly. Numerical experiments for a 3-field formulation of the Biot model are provided to verify our predictions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/app14062523
