Search for: All records

We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper, we fill in the literature gap by studying the central limit theorem behaviors of estimated Q-values and value functions under various RL settings. In particular, we explicitly identify closed-form expressions of the asymptotic variances, which allow us to efficiently construct asymptotically valid confidence regions for key RL quantities. Furthermore, we utilize these asymptotic expressions to design an effective exploration strategy, which we call Q-value-based Optimal Computing Budget Allocation (Q-OCBA). The policy relies on maximizing the relative discrepancies among the Q-value estimates. Numerical experiments show superior performances of our exploration strategy than other benchmark policies. Funding: This work was supported by the National Science Foundation (1720433). 

Consider a fractional Brownian motion (fBM) [Formula: see text] with Hurst index [Formula: see text]. We construct a probability space supporting both B H and a fully simulatable process [Formula: see text] such that[Formula: see text] with probability one for any user-specified error bound [Formula: see text]. When [Formula: see text], we further enhance our error guarantee to the α-Hölder norm for any [Formula: see text]. This enables us to extend our algorithm to the simulation of fBM-driven stochastic differential equations [Formula: see text]. Under mild regularity conditions on the drift and diffusion coefficients of Y, we construct a probability space supporting both Y and a fully simulatable process [Formula: see text] such that[Formula: see text] with probability one. Our algorithms enjoy the tolerance-enforcement feature, under which the error bounds can be updated sequentially in an efficient way. Thus, the algorithms can be readily combined with other advanced simulation techniques to estimate the expectations of functionals of fBMs efficiently. 

Fractional Brownian motions (fBM) and related processes are widely used in financial modeling to capture the complicated dependence structure of the volatility. In this paper, we analyze an infinite series representation of fBM proposed in (Dzhaparidze and Van Zanten 2004) and establish an almost sure convergence rate of the series representation. The rate is also shown to be optimal. We then demonstrate how the strong convergence rate result can be applied to construct simulation algorithms with path-by-path error guarantees. 

Abstract We present the first algorithm that samples max n ≥0 { S n − n α }, where S n is a mean zero random walk, and n α with $\alpha \in ({1 \over 2},1)$ defines a nonlinear boundary. We show that our algorithm has finite expected running time. We also apply this algorithm to construct the first exact simulation method for the steady-state departure process of a GI/GI/∞ queue where the service time distribution has infinite mean.