More Like this

1. Flexible control as surrogate reward or dynamic reward maximization

   https://doi.org/10.1016/j.cognition.2022.105262  

   Liljeholm, Mimi (December 2022, Cognition)
2. Causal Confusion and Reward Misidentification in Preference-Based Reward Learning

   J. Tien, D. Brown (January 2023, International Conference on Learning Representations)
3. Bootstrapped Reward Shaping

   https://doi.org/10.1609/aaai.v39i15.33679  

   Adamczyk, Jacob; Makarenko, Volodymyr; Tiomkin, Stas; Kulkarni, Rahul V (April 2025, Proceedings of the AAAI Conference on Artificial Intelligence)

   In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, potential-based reward shaping (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required potential function must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a bootstrapped method of reward shaping, termed BS-RS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite. 

   more » « less
4. Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning

   Dohmen, T.; Topper, N.; Atia, G.; Beckus, A.; Trivedi, A.; Velasquez, A. (January 2022, International Conference on Automated Planning and Scheduling)
5. Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation

   Min, Do June; Perez-Rosas, Veronica; Resnicow, Ken; Mihalcea, Rada (August 2024, ELRA and ICCL)

   In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models. 

   more » « less