More Like this

1. Hybrid Control for Learning Motor Skills

   Ian Abraham, Alexander Broad (June 2020, Workshop on the Algorithmic Foundations of Robotics (WAFR))

   We develop a hybrid control approach for robot learning based on combining learned predictive models with experience-based state-action policy mappings to improve the learning capabilities of robotic systems. Predictive models provide an understanding of the task and the physics (which improves sample-efficiency), while experience-based pol-icy mappings are treated as “muscle memory” that encode favorable actions as experiences that override planned actions. Hybrid control tools are used to create an algorithmic approach for combining learned predictive models with experience-based learning. Hybrid learning is presented as a method for efficiently learning motor skills by systematically combining and improving the performance of predictive models and experience-based policies. A deterministic variation of hybrid learning is derived and extended into a stochastic implementation that relaxes some of the key assumptions in the original derivation. Each variation is tested on experience-based learning methods (where the robot interacts with the environment to gain experience) as well as imitation learning methods(where experience is provided through demonstrations and tested in the environment). The results show that our method is capable of improving the performance and sample-efficiency of learning motor skills in a variety of experimental domains. 

   more » « less
2. Bootstrapping Motor Skill Learning with Motion Planning

   https://doi.org/10.1109/IROS51168.2021.9636661  

   Abbatematteo, B; Rosen, E; Tellex, S; Konidaris, G.D. (September 2021, Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems)

   Learning a robot motor skill from scratch is impractically slow; so much so that in practice, learning must typically be bootstrapped using human demonstration. However, relying on human demonstration necessarily degrades the autonomy of robots that must learn a wide variety of skills over their operational lifetimes. We propose using kinematic motion planning as a completely autonomous, sample efficient way to bootstrap motor skill learning for object manipulation. We demonstrate the use of motion planners to bootstrap motor skills in two complex object manipulation scenarios with different policy representations: opening a drawer with a dynamic movement primitive representation, and closing a microwave door with a deep neural network policy. We also show how our method can bootstrap a motor skill for the challenging dynamic task of learning to hit a ball off a tee, where a kinematic plan based on treating the scene as static is insufficient to solve the task, but sufficient to bootstrap a more dynamic policy. In all three cases, our method is competitive with human-demonstrated initialization, and significantly outperforms starting with a random policy. This approach enables robots to to efficiently and autonomously learn motor policies for dynamic tasks without human demonstration. 

   more » « less
3. Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning

   Banerjee, Arko; Rahmani, Kia; Biswas, Joydeep; Dillig, Isil (December 2024, Neurips 2024)

   Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a backup policy to ensure safety when the learned policy attempts to take risky actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies. This paper introduces Dynamic Model Predictive Shielding (DMPS), which optimizes reinforcement learning objectives while maintaining provable safety. DMPS employs a local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially, the planner and the neural policy play a synergistic role in DMPS. When planning recovery actions for ensuring safety, the planner utilizes the neural policy to estimate long-term rewards, allowing it to observe beyond its short-term planning horizon. Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging to policies that are both high-performing and safe in practice. This approach guarantees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared to several state-of-the-art baselines 

   more » « less
4. On the model-based stochastic value gradient for continuous reinforcement learning

   Amos, B; Stanton, S; Yarats, D; Wilson, A.G. (January 2021, Learning for Dynamics and Control (L4DC))

   null (Ed.)

   For over a decade, model-based reinforcement learning has been seen as a way to leverage control-based domain knowledge to improve the sample-efficiency of reinforcement learning agents. While model-based agents are conceptually appealing, their policies tend to lag behind those of model-free agents in terms of final reward, especially in non-trivial environments. In response, researchers have proposed model-based agents with increasingly complex components, from ensembles of probabilistic dynamics models, to heuristics for mitigating model error. In a reversal of this trend, we show that simple model-based agents can be derived from existing ideas that not only match, but outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. We find that a model-free soft value estimate for policy evaluation and a model-based stochastic value gradient for policy improvement is an effective combination, achieving state-of-the-art results on a high-dimensional humanoid control task, which most model-based agents are unable to solve. Our findings suggest that model-based policy evaluation deserves closer attention. 

   more » « less
5. Expediting human motor learning in high-dimensional de-novo tasks via online curriculum design

   Kamboj, Ankur; Ranganathan, Rajiv; Tan, Xiaobo; Srivastava, Vaibhav (July 2025, IEEE)

   While recent advancements in motor learning have emphasized the critical role of systematic task scheduling in enhancing task learning, the heuristic design of task schedules remains predominant. Random task scheduling can lead to sub-optimal motor learning, whereas performance-based scheduling might not be adequate for complex motor skill acquisition. This paper addresses these challenges by proposing a model-based approach for online skill estimation and individualized task scheduling in de-novo (novel) motor learning tasks. We introduce a framework utilizing a personalized human motor learning model and particle filter for skill state estimation, coupled with a stochastic nonlinear model predictive control (SNMPC) strategy to optimize curriculum design for a high-dimensional motor task. Simulation results show the effectiveness of our framework in estimating the latent skill state, and the efficacy of the framework in accelerating skill learning. Furthermore, a human subject study shows that the group with the SNMPC-based curriculum design exhibited expedited skill learning and improved task performance. Our contributions offer a pathway towards expedited motor learning across various novel tasks, with implications for enhancing rehabilitation and skill acquisition processes. 

   more » « less