More Like this

1. Robustly Learning Composable Options in Deep Reinforcement Learning

   Bagaria, A; Senthil, J; Slivinski, M; Konidaris, G (August 2021, Proceedings of the 30th International Joint Conference on Artificial Intelligence)

   null (Ed.)

   Hierarchical reinforcement learning (HRL) is only effective for long-horizon problems when high-level skills can be reliably sequentially executed. Unfortunately, learning reliably composable skills is difficult, because all the components of every skill are constantly changing during learning. We propose three methods for improving the composability of learned skills: representing skill initiation regions using a combination of pessimistic and optimistic classifiers; learning re-targetable policies that are robust to non-stationary subgoal regions; and learning robust option policies using model-based RL. We test these improvements on four sparse-reward maze navigation tasks involving a simulated quadrupedal robot. Each method successively improves the robustness of a baseline skill discovery method, substantially outperforming state-of-the-art flat and hierarchical methods. 

   more » « less
2. Skill Discovery for Exploration and Planning using Deep Skill Graphs

   Bagaria, A; Senthil, J; Konidaris, G (July 2021, Proceedings of the Thirty-Eighth International Conference on Machine Learning)

   null (Ed.)

   We introduce a new skill-discovery algorithm that builds a discrete graph representation of large continuous MDPs, where nodes correspond to skill subgoals and the edges to skill policies. The agent constructs this graph during an unsupervised training phase where it interleaves discovering skills and planning using them to gain coverage over ever-increasing portions of the state-space. Given a novel goal at test time, the agent plans with the acquired skill graph to reach a nearby state, then switches to learning to reach the goal. We show that the resulting algorithm, Deep Skill Graphs, outperforms both flat and existing hierarchical reinforcement learning methods on four difficult continuous control tasks. 

   more » « less
3. Skill Discovery for Exploration and Planning using Deep Skill Graphs

   Bagaria, A; Senthil, J.; Konidaris, G.D. (July 2021, Proceedings of the Thirty-Eighth International Conference on Machine Learning)

   We introduce a new skill-discovery algorithm that builds a discrete graph representation of large continuous MDPs, where nodes correspond to skill subgoals and the edges to skill policies. The agent constructs this graph during an unsupervised training phase where it interleaves discovering skills and planning using them to gain coverage over ever-increasing portions of the state-space. Given a novel goal at test time, the agent plans with the acquired skill graph to reach a nearby state, then switches to learning to reach the goal. We show that the resulting algorithm, Deep Skill Graphs, outperforms both flat and existing hierarchical reinforcement learning methods on four difficult continuous control tasks. 

   more » « less
4. Bootstrapping Motor Skill Learning with Motion Planning

   https://doi.org/10.1109/IROS51168.2021.9636661  

   Abbatematteo, B; Rosen, E; Tellex, S; Konidaris, G.D. (September 2021, Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems)

   Learning a robot motor skill from scratch is impractically slow; so much so that in practice, learning must typically be bootstrapped using human demonstration. However, relying on human demonstration necessarily degrades the autonomy of robots that must learn a wide variety of skills over their operational lifetimes. We propose using kinematic motion planning as a completely autonomous, sample efficient way to bootstrap motor skill learning for object manipulation. We demonstrate the use of motion planners to bootstrap motor skills in two complex object manipulation scenarios with different policy representations: opening a drawer with a dynamic movement primitive representation, and closing a microwave door with a deep neural network policy. We also show how our method can bootstrap a motor skill for the challenging dynamic task of learning to hit a ball off a tee, where a kinematic plan based on treating the scene as static is insufficient to solve the task, but sufficient to bootstrap a more dynamic policy. In all three cases, our method is competitive with human-demonstrated initialization, and significantly outperforms starting with a random policy. This approach enables robots to to efficiently and autonomously learn motor policies for dynamic tasks without human demonstration. 

   more » « less
5. Language-guided Skill Learning with Temporal Variational Inference

   Fu, H; Sharma, P; Stengel-Eskin, E; Konidaris, G; Le_Roux, N; Cote, M-A; Yuan, X (July 2024, Proceedings of the 41st International Conference on Machine Learning)

   We present an algorithm for skill discovery from expert demonstrations. The algorithm first utilizes Large Language Models (LLMs) to propose an initial segmentation of the trajectories. Following that, a hierarchical variational inference framework incorporates the LLM-generated segmentation information to discover reusable skills by merging trajectory segments. To further control the trade-off between compression and reusability, we introduce a novel auxiliary objective based on the Minimum Description Length principle that helps guide this skill discovery process. Our results demonstrate that agents equipped with our method are able to discover skills that help accelerate learning and outperform baseline skill learning approaches on new long-horizon tasks in BabyAI, a grid world navigation environment, as well as ALFRED, a household simulation environment. 

   more » « less