More Like this

1. Softmax policy gradient methods can take exponential time to converge

   https://doi.org/10.1007/s10107-022-01920-6  

   Li, Gen ; Wei, Yuting ; Chi, Yuejie ; Chen, Yuxin ( January 2023 , Mathematical Programming)

   The softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For$$\gamma $$$\gamma$-discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space$${\mathcal {S}}$$$S$and the effective horizon$$\frac{1}{1-\gamma }$$$\frac{1}{1-\gamma }$, both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that the softmax PG method with stepsize$$\eta $$$\eta$can take$$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$$\begin{array}{c}\frac{1}{\eta }{\left|S\right|}^{2^{\Omega (\frac{1}{1-\gamma })}}\text{iterations}\end{array}$to converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration (so that the distribution mismatch coefficient is not exceedingly large). This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.

    

   more » « less
2. Green function estimates on complements of low-dimensional uniformly rectifiable sets

   https://doi.org/10.1007/s00208-022-02379-8  

   David, Guy ; Feneuil, Joseph ; Mayboroda, Svitlana ( March 2022 , Mathematische Annalen)

   It has been recently established in David and Mayboroda (Approximation of green functions and domains with uniformly rectifiable boundaries of all dimensions.arXiv:2010.09793) that on uniformly rectifiable sets the Green function is almost affine in the weak sense, and moreover, in some scenarios such Green function estimates are equivalent to the uniform rectifiability of a set. The present paper tackles a strong analogue of these results, starting with the “flagship degenerate operators on sets with lower dimensional boundaries. We consider the elliptic operators$$L_{\beta ,\gamma } =- {\text {div}}D^{d+1+\gamma -n} \nabla $$$L_{\beta ,\gamma }=-\text{div}{D}^{d+1+\gamma -n}\nabla$associated to a domain$$\Omega \subset {\mathbb {R}}^n$$$\Omega \subset R^n$with a uniformly rectifiable boundary$$\Gamma $$$\Gamma$of dimension$$d < n-1$$$d<n-1$, the now usual distance to the boundary$$D = D_\beta $$$D=D_{\beta }$given by$$D_\beta (X)^{-\beta } = \int _{\Gamma } |X-y|^{-d-\beta } d\sigma (y)$$$D_{\beta }{\left(X\right)}^{-\beta }={\int }_{\Gamma }{|X-y|}^{-d-\beta }d\sigma \left(y\right)$for$$X \in \Omega $$$X\in \Omega$, where$$\beta >0$$$\beta >0$and$$\gamma \in (-1,1)$$$\gamma \in (-1,1)$. In this paper we show that the Green functionGfor$$L_{\beta ,\gamma }$$$L_{\beta ,\gamma }$, with pole at infinity, is well approximated by multiples of$$D^{1-\gamma }$$$D^{1-\gamma }$, in the sense that the function$$\big | D\nabla \big (\ln \big ( \frac{G}{D^{1-\gamma }} \big )\big )\big |^2$$$|D\nabla (ln(\frac{G}{D^{1-\gamma }})){|}^2$satisfies a Carleson measure estimate on$$\Omega $$$\Omega$. We underline that the strong and the weak results are different in nature and, of course, at the level of the proofs: the latter extensively used compactness arguments, while the present paper relies on some intricate integration by parts and the properties of the “magical distance function from David et al. (Duke Math J, to appear).

    

   more » « less
3. Numerical approximation of nonlinear SPDE’s

   https://doi.org/10.1007/s40072-022-00271-9  

   Ondreját, Martin ; Prohl, Andreas ; Walkington, Noel J. ( September 2022 , Stochastics and Partial Differential Equations: Analysis and Computations)

   The numerical analysis of stochastic parabolic partial differential equations of the form$$\begin{aligned} du + A(u)\, dt = f \,dt + g \, dW, \end{aligned}$$$\begin{array}{c}du+A\left(u\right)dt=fdt+gdW,\end{array}$is surveyed, whereAis a nonlinear partial operator andWa Brownian motion. This manuscript unifies much of the theory developed over the last decade into a cohesive framework which integrates techniques for the approximation of deterministic partial differential equations with methods for the approximation of stochastic ordinary differential equations. The manuscript is intended to be accessible to audiences versed in either of these disciplines, and examples are presented to illustrate the applicability of the theory.

    

   more » « less
4. Gaussian Fluctuation for Smoothed Local Correlations in CUE

   https://doi.org/10.1007/s10955-022-03028-9  

   Soshnikov, Alexander ( November 2022 , Journal of Statistical Physics)

   Motivated by the Rudnick-Sarnak theorem we study limiting distribution of smoothed local correlations of the form$$\begin{aligned} \sum _{j_1, j_2, \ldots , j_n} f(N(\theta _{j_2}-\theta _{j_1}), N(\theta _{j_3}-\theta _{j_1}), \ldots , N(\theta _{j_n}-\theta _{j_1})) \end{aligned}$$$\begin{array}{c}\sum _{j_1,j_2,\dots ,j_n}f(N({\theta }_{j_2}-{\theta }_{j_1}),N({\theta }_{j_3}-{\theta }_{j_1}),\dots ,N({\theta }_{j_n}-{\theta }_{j_1}))\end{array}$for the Circular United Ensemble of random matrices for sufficiently smooth test functions.

    

   more » « less
5. Multidimensional WENO-AO Reconstructions Using a Simplified Smoothness Indicator and Applications to Conservation Laws

   https://doi.org/10.1007/s10915-023-02319-x  

   Huang, Chieh-Sen ; Arbogast, Todd ; Tian, Chenyu ( August 2023 , Journal of Scientific Computing)

   Finite volume, weighted essentially non-oscillatory (WENO) schemes require the computation of a smoothness indicator. This can be expensive, especially in multiple space dimensions. We consider the use of the simple smoothness indicator$$\sigma ^{\textrm{S}}= \frac{1}{N_{\textrm{S}}-1}\sum _{j} ({\bar{u}}_{j} - {\bar{u}}_{m})^2$$${\sigma }^{\text{S}}=\frac{1}{N_{\text{S}}-1}{\sum }_j{({\overline{u}}_j-{\overline{u}}_m)}^2$, where$$N_{\textrm{S}}$$$N_{\text{S}}$is the number of mesh elements in the stencil,$${\bar{u}}_j$$${\overline{u}}_j$is the local function average over mesh elementj, and indexmgives the target element. Reconstructions utilizing standard WENO weighting fail with this smoothness indicator. We develop a modification of WENO-Z weighting that gives a reliable and accurate reconstruction of adaptive order, which we denote as SWENOZ-AO. We prove that it attains the order of accuracy of the large stencil polynomial approximation when the solution is smooth, and drops to the order of the small stencil polynomial approximations when there is a jump discontinuity in the solution. Numerical examples in one and two space dimensions on general meshes verify the approximation properties of the reconstruction. They also show it to be about 10 times faster in two space dimensions than reconstructions using the classic smoothness indicator. The new reconstruction is applied to define finite volume schemes to approximate the solution of hyperbolic conservation laws. Numerical tests show results of the same quality as standard WENO schemes using the classic smoothness indicator, but with an overall speedup in the computation time of about 3.5–5 times in 2D tests. Moreover, the computational efficiency (CPU time versus error) is noticeably improved.

    

   more » « less