More Like this

1. A Human-Centered Data-Driven Planner-Actor-Critic Architecture via Logic Programming

   https://doi.org/10.4204/EPTCS.306.23  

   Lyu, D ; Yang, F ; Liu, B ; Gustafson, S. ( October 2019 , 35th International Conference on Logic Programming)

   Recent successes of Reinforcement Learning (RL) allow an agent to learn policies that surpass human experts but suffers from being time-hungry and data-hungry. By contrast, human learning is significantly faster because prior and general knowledge and multiple information resources are utilized. In this paper, we propose a Planner-Actor-Critic architecture for huMAN-centered planning and learning (PACMAN), where an agent uses its prior, high-level, deterministic symbolic knowledge to plan for goal-directed actions, and also integrates the Actor-Critic algorithm of RL to fine-tune its behavior towards both environmental rewards and human feedback. This work is the first unified framework where knowledge-based planning, RL, and human teaching jointly contribute to the policy learning of an agent. Our experiments demonstrate that PACMAN leads to a significant jump-start at the early stage of learning, converges rapidly and with small variance, and is robust to inconsistent, infrequent, and misleading feedback. 

   more » « less
2. SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

   https://doi.org/https://dl.acm.org/doi/10.5555/3463952.3464062  

   Sarathy, Vasanth ; Kasenberg, Daniel ; Goel, Shivam ; Sinapov, Jivko ; Scheutz, Matthias ( January 2021 , AAMAS Conference proceedings)

   Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support ("spot") a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator. 

   more » « less
3. Influencing Reinforcement Learning through Natural Language Guidance

   https://doi.org/10.32473/flairs.v34i1.128472  

   Tasrin, Tasmia ; Nahian, Md Sultan ; Perera, Habarakadage ; Harrison, Brent ( May 2021 , The International FLAIRS Conference Proceedings)

   null (Ed.)

   Interactive reinforcement learning (IRL) agents use human feedback or instruction to help them learn in complex environments. Often, this feedback comes in the form of a discrete signal that’s either positive or negative. While informative, this information can be difficult to generalize on its own. In this work, we explore how natural language advice can be used to provide a richer feedback signal to a reinforcement learning agent by extending policy shaping, a well-known IRL technique. Usually policy shaping employs a human feedback policy to help an agent to learn more about how to achieve its goal. In our case, we replace this human feedback policy with policy generated based on natural language advice. We aim to inspect if the generated natural language reasoning provides support to a deep RL agent to decide its actions successfully in any given environment. So, we design our model with three networks: first one is the experience driven, next is the advice generator and third one is the advice driven. While the experience driven RL agent chooses its actions being influenced by the environmental reward, the advice driven neural network with generated feedback by the advice generator for any new state selects its actions to assist the RL agent to better policy shaping. 

   more » « less
4. Critic-over-Actor-Critic Modeling: Finding Optimal Strategy in ICU Environments

   https://doi.org/10.1109/BigData55660.2022.10021125  

   Ryan, Riazat ; Shao, Ming ( December 2022 , 2022 IEEE International Conference on Big Data (Big Data))

   Reinforcement learning (RL) is mechanized to learn from experience. It solves the problem in sequential decisions by optimizing reward-punishment through experimentation of the distinct actions in an environment. Unlike supervised learning models, RL lacks static input-output mappings and the objective of minimization of a vector error. However, to find out an optimal strategy, it is crucial to learn both continuous feedback from training data and the offline rules of the experiences with no explicit dependence on online samples. In this paper, we present a study of a multi-agent RL framework which involves a Critic in semi-offline mode criticizing over an online Actor-Critic network, namely, Critic-over-Actor-Critic (CoAC) model, in finding optimal treatment plan of ICU patients as well as optimal strategy in a combative battle game. For further validation, we also examine the model in the adversarial assignment. 

   more » « less
5. Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

   Luo, Katie ; Liu, Zhenzhen ; Chen, Xiangyu ; You, Yurong ; Bainam, Sagie ; Phoo, Cheng P ; Campbell, Mark ; Sun, Wen ; Hariharan, Bharath ; Weinberger, Kilian Q ( December 2023 , Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

   Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector’s own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT. 

   more » « less