More Like this

1. Planning with Abstract Markov Decision Processes

   Gopalan, Nakul ; desJardins, Marie ; Littman, Michael L. ; MacGlashan, J. ; Squire, S. ; Tellex, Stefanie ; Winder, John ; Wong, Lawson L. ( July 2017 , 27th International Conference on Automated Planning and Scheduling)

   Robots acting in human-scale environments must plan under uncertainty in large state–action spaces and face constantly changing reward functions as requirements and goals change. Planning under uncertainty in large state–action spaces requires hierarchical abstraction for efficient computation. We introduce a new hierarchical planning framework called Abstract Markov Decision Processes (AMDPs) that can plan in a fraction of the time needed for complex decision making in ordinary MDPs. AMDPs provide abstract states, actions, and transition dynamics in multiple layers above a base-level “flat” MDP. AMDPs decompose problems into a series of subtasks with both local reward and local transition functions used to create policies for subtasks. The resulting hierarchical planning method is independently optimal at each level of abstraction, and is recursively optimal when the local reward and transition functions are correct. We present empirical results showing significantly improved planning speed, while maintaining solution quality, in the Taxi domain and in a mobile-manipulation robotics problem. Furthermore, our approach allows specification of a decision-making model for a mobile-manipulation problem on a Turtlebot, spanning from low-level control actions operating on continuous variables all the way up through high-level object manipulation tasks. 

   more » « less
2. Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition

   Liu, Xuan ; Fu, Jie ( July 2019 , Annual American Control Conference)

   In hierarchical planning for Markov decision processes (MDPs), temporal abstraction allows planning with macro-actions that take place at different time scale in the form of sequential composition. In this paper, we propose a novel approach to compositional reasoning and hierarchical planning for MDPs under co-safe temporal logic constraints. In addition to sequential composition, we introduce a composition of policies based on generalized logic composition: Given sub-policies for sub-tasks and a new task expressed as logic compositions of subtasks, a semi-optimal policy, which is optimal in planning with only sub-policies, can be obtained by simply composing sub-polices. Thus, a synthesis algorithm is developed to compute optimal policies efficiently by planning with primitive actions, policies for sub-tasks, and the compositions of sub-policies, for maximizing the probability of satisfying constraints specified in the fragment of co-safe temporal logic. We demonstrate the correctness and efficiency of the proposed method in stochastic planning examples with a single agent and multiple task specifications. 

   more » « less
3. Planning with Abstract Learned Models While Learning Transferable Subtasks

   https://doi.org/10.1609/aaai.v34i06.6555  

   Winder, John ; Milani, Stephanie ; Landen, Matthew ; Oh, Erebus ; Parr, Shane ; Squire, Shawn ; desJardins, Marie ; Matuszek, Cynthia ( February 2020 , Proceedings of the AAAI Conference on Artificial Intelligence)

   We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks. 

   more » « less
4. Planning with Abstract Learned Models While Learning Transferable Subtasks

   Winder, John ; Milani, Stephanie ; Landen, Matthew ; Oh, Erebus ; Parr, Shane ; Squire, Shawn ; desJardins, Marie ; Matuszek, Cynthia ( February 2020 , Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks. 

   more » « less
5. Planning with Abstract Learned Models While Learning Transferable Subtasks

   Winder, John ; Milani, Stephanie ; Landen, Matthew ; Oh, Erebus ; Parr, Shane ; Squire, Shawn ; desJardins, Marie ; Matuszek, Cynthia ( February 2020 , Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks. 

   more » « less