More Like this

1. Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

   Liu, Zhishuai; Xu, Pan (March 2024, Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS))

   We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a $$d$$-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments. 

   more » « less
2. Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

   Liu, Zhishuai; Xu, Pan (March 2024, Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS))

   We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a $$d$$-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments. 

   more » « less
3. A Quantile-based Nested Partition Algorithm for Black-box Functions on a Continuous Domain

   Linz, David; Huang, Hao; Zabinsky, Zelda B. (January 2016, Proceedings of the 2016 Winter Simulation Conference / 978-1-5090-4484-9)

   Simulation models commonly describe complex systems with no closed-form analytical representation. This paper proposes an algorithm for functions on continuous domains that fits into the nested partition framework and uses quantile estimation to rank regions and identify the most promising region. Additionally, we apply the optimal computational budget allocation (OCBA) method for allocating sample points using the normality property of quantile estimators. We prove that, for functions satisfying the Lipschitz condition, the algorithm converges in probability to a region that contains the true global optimum. The paper concludes with some numerical results. 

   more » « less
4. Algorithms for Radon Partitions with Tolerance

   https://doi.org/10.1007/978-3-030-39219-2_38  

   Bereg Sergey, Haghpanah Mohammadreza (February 2020, Lecture notes in computer science)

   Let P be a set n points in a d-dimensional space. Tverberg theorem says that, if n is at least (k − 1)(d + 1), then P can be par- titioned into k sets whose convex hulls intersect. Partitions with this property are called Tverberg partitions. A partition has tolerance t if the partition remains a Tverberg partition after removal of any set of t points from P. A tolerant Tverberg partition exists in any dimensions provided that n is sufficiently large. Let N(d,k,t) be the smallest value of n such that tolerant Tverberg partitions exist for any set of n points in R d . Only few exact values of N(d,k,t) are known. In this paper, we study the problem of finding Radon partitions (Tver- berg partitions for k = 2) for a given set of points. We develop several algorithms and found new lower bounds for N(d,2,t). 

   more » « less
5. Dynamic Geometric Set Cover and Hitting Set

   Agarwal, P; Chang, H; Suri, S; Xiao, A; Xue, J. (June 2020, 36th International Symposium on Computational Geometry (SoCG 2020).)

   We investigate dynamic versions of geometric set cover and hitting set where points and ranges may be inserted or deleted, and we want to efficiently maintain an (approximately) optimal solution for the current problem instance. While their static versions have been extensively studied in the past, surprisingly little is known about dynamic geometric set cover and hitting set. For instance, even for the most basic case of one-dimensional interval set cover and hitting set, no nontrivial results were known. The main contribution of our paper are two frameworks that lead to efficient data structures. 

   more » « less