More Like this

1. Towards Off-Policy Learning for Ranking Policies with Logged Feedback

   https://doi.org/10.1609/aaai.v36i8.20849  

   Xiao, Teng; Wang, Suhang (June 2022, Proceedings of the AAAI Conference on Artificial Intelligence)

   Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods 

   more » « less
2. Learning Off-policy for Online Planning

   Sikchi, Harshit; Zhou, Wenxuan; Held, David (January 2021, Conference on Robot Learning (CoRL))

   Reinforcement learning (RL) in low-data and risk-sensitive domains requires performant and flexible deployment policies that can readily incorporate constraints during deployment. One such class of policies are the semi-parametric H-step lookahead policies, which select actions using trajectory optimization over a dynamics model for a fixed horizon with a terminal value function. In this work, we investigate a novel instantiation of H-step lookahead with a learned model and a terminal value function learned by a model-free off-policy algorithm, named Learning Off-Policy with Online Planning (LOOP). We provide a theoretical analysis of this method, suggesting a tradeoff between model errors and value function errors and empirically demonstrate this tradeoff to be beneficial in deep reinforcement learning. Furthermore, we identify the "Actor Divergence" issue in this framework and propose Actor Regularized Control (ARC), a modified trajectory optimization procedure. We evaluate our method on a set of robotic tasks for Offline and Online RL and demonstrate improved performance. We also show the flexibility of LOOP to incorporate safety constraints during deployment with a set of navigation environments. We demonstrate that LOOP is a desirable framework for robotics applications based on its strong performance in various important RL settings. 

   more » « less
3. In-Context Reinforcement Learning From Suboptimal Historical Data

   Dong, Juncheng; Guo, Moyang; Fang, Ethan X; Yang, Zhuoran; Tarokh, Vahid (August 2025, ICML Proceedings)

   Transformer models have achieved remarkable empirical successes, largely due to their in-context learning capabilities. Inspired by this, we explore training an autoregressive transformer for in-context reinforcement learning (ICRL). In this setting, we initially train a transformer on an offline dataset consisting of trajectories collected from various RL tasks, and then fix and use this transformer to create an action policy for new RL tasks. Notably, we consider the setting where the offline dataset contains trajectories sampled from suboptimal behavioral policies. In this case, standard autoregressive training corresponds to imitation learning and results in suboptimal performance. To address this, we propose the Decision Importance Transformer (DIT) framework, which emulates the actor-critic algorithm in an in-context manner. In particular, we first train a transformer-based value function that estimates the advantage functions of the behavior policies that collected the suboptimal trajectories. Then we train a transformer-based policy via a weighted maximum likelihood estimation loss, where the weights are constructed based on the trained value function to steer the suboptimal policies to the optimal ones. We conduct extensive experiments to test the performance of DIT on both bandit and Markov Decision Process problems. Our results show that DIT achieves superior performance, particularly when the offline dataset contains suboptimal historical data. 

   more » « less
4. Recorp: Receiver-Oriented Policies for Industrial Wireless Networks

   https://doi.org/10.1109/IoTDI49375.2020.00020  

   Brummet, Ryan; Chipara, Octav; Herman, Ted (April 2020, 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI))

   The next generation of Industrial Internet-of-Things (IIoT) systems will require wireless solutions to connect sensors, actuators, and controllers as part of feedback-control loops over real-time flows. A key challenge in such networks is to provide predictable performance and adaptability to variations in link quality. We address this challenge by developing Receiver Oriented Policies (RECORP), which leverages the stability of IIoT workloads to build a solution that combines offline policy synthesis and run-time adaptation. Compared to schedules that service a single flow in a slot, RECORP policies share slots among multiple flows by assigning a coordinator and a set of candidate flows in the same slot. At run-time, the coordinator will dynamically execute one of the flows depending on what flows the coordinator has already received. The net effect of this strategy is that a node can dynamically repurpose the retransmissions remaining after receiving the data of an incoming flow to service other incoming flows opportunistically. Therefore, the flows that are executed in a slot can be adapted in response to the variable link conditions observed at run-time. Furthermore, RECORP also provides predictable performance: a policy meets the end-to-end reliability and deadline constraints of flows given probabilistic link qualities. When RECORP policies and schedules are configured to meet the same end-to-end reliability target of 99%, larger-scale multihop simulations show that across typical IIoT workloads, policies provided a median improvement of 1.63 to 2.44 times in real-time capacity as well as a median reduction of 1.45 to 2.43 times in worst-case latency. 

   more » « less
5. Contextual Ranking and Selection with Gaussian Processes and Optimal Computing Budget Allocation

   https://doi.org/10.1145/3633456  

   Cakmak, Sait; Wang, Yuhao; Gao, Siyang; Zhou, Enlu (April 2024, ACM Transactions on Modeling and Computer Simulation)

   In many real-world problems, we are faced with the problem of selecting the best among a finite number of alternatives, where the best alternative is determined based on context specific information. In this work, we study the contextual Ranking and Selection problem under a finite-alternative-finite-context setting, where we aim to find the best alternative for each context. We use a separate Gaussian process to model the reward for each alternative and derive the large deviations rate function for both the expected and worst-case contextual probability of correct selection. We propose the GP-C-OCBA sampling policy, which uses the Gaussian process posterior to iteratively allocate observations to maximize the rate function. We prove its consistency and show that it achieves the optimal convergence rate under the assumption of a non-informative prior. Numerical experiments show that our algorithm is highly competitive in terms of sampling efficiency, while having significantly smaller computational overhead. 

   more » « less