More Like this

1. Inverse Reinforcement Learning of Interaction Dynamics from Demonstrations

   Mostafa Hussein, Momotaz Begum (January 2019, IEEE International Conference on Robotics and Automation)

   This paper presents a framework to learn the reward function underlying high-level sequential tasks from demonstrations. The purpose of reward learning, in the context of learning from demonstration (LfD), is to generate policies that mimic the demonstrator’s policies, thereby enabling imitation learning. We focus on a human-robot interaction(HRI) domain where the goal is to learn and model structured interactions between a human and a robot. Such interactions can be modeled as a partially observable Markov decision process (POMDP) where the partial observability is caused by uncertainties associated with the ways humans respond to different stimuli. The key challenge in finding a good policy in such a POMDP is determining the reward function that was observed by the demonstrator. Existing inverse reinforcement learning(IRL) methods for POMDPs are computationally very expensive and the problem is not well understood. In comparison, IRL algorithms for Markov decision process (MDP) are well defined and computationally efficient. We propose an approach of reward function learning for high-level sequential tasks from human demonstrations where the core idea is to reduce the underlying POMDP to an MDP and apply any efficient MDP-IRL algorithm. Our extensive experiments suggest that the reward function learned this way generates POMDP policies that mimic the policies of the demonstrator well. 

   more » « less
2. dSLAP: Distributed Safe Learning and Planning for Multi-robot Systems

   https://doi.org/10.1109/CDC51059.2022.9992938  

   Yuan, Zhenyuan; Zhu, Minghui (December 2022, IEEE 61st Conference on Decision and Control)

   This paper considers the problem where a group of mobile robots subject to unknown external disturbances aim to safely reach goal regions. We develop a distributed safe learning and planning algorithm that allows the robots to learn about the external unknown disturbances and safely navigate through the environment via their single trajectories. We use Gaussian process regression for online learning where variance is adopted to quantify the learning uncertainty. By leveraging set-valued analysis, the developed algorithm enables fast adaptation to newly learned models while avoiding collision against the learning uncertainty. Active learning is then applied to return a control policy such that the robots are able to actively explore the unknown disturbances and reach their goal regions in time. Sufficient conditions are established to guarantee the safety of the robots. A set of simulations are conducted for evaluation. 

   more » « less
3. Learning rewards from linguistic feedback

   Sumers, Theodore R.; Ho, Mark K.; Hawkins, Robert D.; Narasimhan, K.; Griffiths, Thomas L. (January 2021, Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   We explore unconstrained natural language feedback as a learning signal for artificial agents. Humans use rich and varied language to teach, yet most prior work on interactive learning from language assumes a particular form of input (e.g., commands). We propose a general framework which does not make this assumption, instead using aspect-based sentiment analysis to decompose feedback into sentiment over the features of a Markov decision process. We then infer the teacher's reward function by regressing the sentiment on the features, an analogue of inverse reinforcement learning. To evaluate our approach, we first collect a corpus of teaching behavior in a cooperative task where both teacher and learner are human. We implement three artificial learners: sentiment-based "literal" and "pragmatic" models, and an inference network trained end-to-end to predict rewards. We then re-run our initial experiment, pairing human teachers with these artificial learners. All three models successfully learn from interactive human feedback. The inference network approaches the performance of the "literal" sentiment model, while the "pragmatic" model nears human performance. Our work provides insight into the information structure of naturalistic linguistic feedback as well as methods to leverage it for reinforcement learning. 

   more » « less
4. Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization

   Liao, Junyi; Zhu, Zihan; Fang, Ethan X; Yang, Zhuoran; Tarokh, Vahid (August 2025, ICML Proceedings)

   Estimating the unknown reward functions driving agents' behavior is a central challenge in inverse games and reinforcement learning. This paper introduces a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization. Given observed player strategies and actions, we aim to reconstruct the underlying reward functions. This task is challenging due to the inherent ambiguity of inverse problems, the non-uniqueness of feasible rewards, and limited observational data coverage. To address these challenges, we establish reward function identifiability using the quantal response equilibrium (QRE) under linear assumptions. Building on this theoretical foundation, we propose an algorithm to learn reward from observed actions, designed to capture all plausible reward parameters by constructing confidence sets. Our algorithm works in both static and dynamic settings and is adaptable to incorporate other methods, such as Maximum Likelihood Estimation (MLE). We provide strong theoretical guarantees for the reliability and sample-efficiency of our algorithm. Empirical results demonstrate the framework’s effectiveness in accurately recovering reward functions across various scenarios, offering new insights into decision-making in competitive environments. 

   more » « less
5. Learning Markov State Abstractions for Deep Reinforcement Learning

   Allen, C; Parikh, N; Gottesman, O; Konidaris, G.D. (December 2021, Neural Information Processing Systems 34)

   A fundamental assumption of reinforcement learning in Markov decision processes (MDPs) is that the relevant decision process is, in fact, Markov. However, when MDPs have rich observations, agents typically learn by way of an abstract state representation, and such representations are not guaranteed to preserve the Markov property. We introduce a novel set of conditions and prove that they are sufficient for learning a Markov abstract state representation. We then describe a practical training procedure that combines inverse model estimation and temporal contrastive learning to learn an abstraction that approximately satisfies these conditions. Our novel training objective is compatible with both online and offline training: it does not require a reward signal, but agents can capitalize on reward information when available. We empirically evaluate our approach on a visual gridworld domain and a set of continuous control benchmarks. Our approach learns representations that capture the underlying structure of the domain and lead to improved sample efficiency over state-of-the-art deep reinforcement learning with visual features—often matching or exceeding the performance achieved with hand-designed compact state information. 

   more » « less