skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


This content will become publicly available on June 10, 2026

Title: FRONT: Foresighted Online Policy Optimization with Interference
Contextual bandits, which leverage baseline features of sequentially arriving individuals to optimize cumulative rewards while balancing exploration and exploitation, are critical for online decision-making. Existing approaches typically assume no interference, where each individual’s action affects only their own reward. Yet, such an assumption can be violated in many practical scenarios, and the oversight of interference can lead to short-sighted policies that focus solely on maximizing the immediate outcomes for individuals, which further results in suboptimal decisions and potentially increased regret over time. To address this significant gap, we introduce the foresighted online policy with interference (FRONT) that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards.  more » « less
Award ID(s):
2401271
PAR ID:
10613416
Author(s) / Creator(s):
; ;
Publisher / Repository:
Reinforcement Learning Journal
Date Published:
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, In Proceedings of the 13th International Conference on Educational Data Mining (EDM))
    null (Ed.)
    Abstract: Identifying critical decisions is one of the most challenging decision-making problems in real-world applications. In this work, we propose a novel Reinforcement Learning (RL) based Long-Short Term Rewards (LSTR) framework for critical decisions identification. RL is a machine learning area concerned with inducing effective decision-making policies, following which result in the maximum cumulative "reward." Many RL algorithms find the optimal policy via estimating the optimal Q-values, which specify the maximum cumulative reward the agent can receive. In our LSTR framework, the "long term" rewards are defined as "Q-values" and the "short term" rewards are determined by the "reward function." Experiments on a synthetic GridWorld game and real-world Intelligent Tutoring System datasets show that the proposed LSTR framework indeed identifies the critical decisions in the sequences. Furthermore, our results show that carrying out the critical decisions alone is as effective as a fully-executed policy. 
    more » « less
  2. ; ; (, Unmanned Systems)
    This paper develops a decision framework to automate the playbook for UAS traffic management (UTM) under uncertain environmental conditions based on spatiotemporal scenario data. Motivated by the traditional air traffic management (ATM) which uses the playbook to guide traffic using pre-validated routes under convective weather, the proposed UTM playbook leverages a database to store optimal UAS routes tagged with spatiotemporal wind scenarios to automate the UAS trajectory management. Our perspective is that the UASs, and many other modern systems, operate in spatiotemporally evolving environments, and similar spatiotemporal scenarios are tied with similar management decisions. Motivated by this feature, our automated playbook solution integrates the offline operations, online operations and a database to enable real-time UAS trajectory management decisions. The solution features the use of similarity between spatiotemporal scenarios to retrieve offline decisions as the initial solution for online fine tuning, which significantly shortens the online decision time. A fast query algorithm that exploits the correlation of spatiotemporal scenarios is utilized in the decision framework to quickly retrieve the best offline decisions. The online fine tuning adapts to trajectory deviations and subject to collision avoidance among UASs. The solution is demonstrated using simulation studies, and can be utilized in other applications, where quick decisions are desired and spatiotemporal environments play a crucial role in the decision process. 
    more » « less
  3. ; (, Proceedings of the AAAI Conference on Artificial Intelligence)
    Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods 
    more » « less
  4. ; (, Proceedings of the 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP))
    We introduce a sequential Bayesian binary hypothesis testing problem under social learning, termed selfish learning, where agents work to maximize their individual rewards. In particular, each agent receives a private signal and is aware of decisions made by earlier-acting agents. Beside inferring the underlying hypothesis, agents also decide whether to stop and declare, or pass the inference to the next agent. The employer rewards only correct responses and the reward per worker decreases with the number of employees used for decision making. We characterize decision regions of agents in the infinite and finite horizon. In particular, we show that the decision boundaries in the infinite horizon are the solutions to a Markov Decision Process with discounted costs, and can be solved using value iteration. In the finite horizon, we show that team performance is enhanced upon appropriate incentivization when compared to sequential social learning. 
    more » « less
  5. ; (, American Economic Review)
    Using a laboratory experiment, we identify whether decision-makers consider it a mistake to violate canonical choice axioms. To do this, we incentivize subjects to report axioms they want their decisions to satisfy. Then, subjects make lottery choices which might conflict with their axiom preferences. In instances of conflict, we give subjects the opportunity to re-evaluate their decisions. We find that many individuals want to follow canonical axioms and revise their choices to be consistent with the axioms. In a shorter online experiment, we show correlations of mistakes with response times and measures of cognition. (JEL C91, D12, D44, D91) 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email