More Like this

1. Decentralized optimization of energy-water nexus based on a mixed-integer boundary compatible algorithm

   https://doi.org/10.1016/j.apenergy.2023.122588  

   Sharma, Santosh; Li, Qifeng (April 2024, Applied Energy)

   The electric power distribution system (PDS) and the water distribution system (WDS) are coupled with each other through electricity-driven water facilities (EdWFs), such as pumps, water desalination plants, and wastewater treatment facilities. However, they are generally owned and operated by different utilities, and there does not exist an operator that possesses full information of both systems. As a result, centralized methods are not applicable for coordinating the operation of the two systems. This paper proposes a decentralized framework where the PDS and WDS operators solve their own operation problems, respectively, by sharing only limited information. Nevertheless, the boundary variables (i.e., the variables shared between two systems) are discontinuous due to their dependence on the on/off nature of EdWFs. Unfortunately, mature decentralized/distributed optimization algorithms like the alternating direction method of multipliers (ADMM) cannot guarantee convergence and optimality for a case like this. Therefore, this paper develops a novel algorithm that can guarantee convergence and optimality for the decentralized optimization of PDS and WDS based on a recently developed algorithm called the SD-GS-AL method. The SD-GS-AL method is a combination of the simplicial decomposition (SD), gauss–seidel (GS), and augmented Lagrangian (AL) methods, which can guarantee convergence and optimality for mixed-integer programs (MIPs) with continuous boundary variables. Nonetheless, the original SD-GS-AL algorithm does not work for the PDS-WDS coordination problem where the boundary variables are discontinuous. This paper modifies and improves the original SD-GS-AL algorithm by introducing update rules to discontinuous boundary variables (called the Auxiliary Variables Update step). The proposed mixed-integer boundary compatible (MIBC) SD-GS-AL algorithm has the following benefits: (1) it is capable of handling cases whose boundary variables are discontinuous with convergence and optimality guaranteed for mild assumptions, and (2) it only requires limited information exchange between PDS and WDS operators, which will help preserve the privacy of the two utilities and reduce the investment in building additional communication channels. Simulations on two coupled PDS and WDS test cases (Case 1: IEEE-13 node PDS and 11-node WDS, and Case 2: IEEE-37 node PDS and 36-node WDS) show that the proposed MIBC algorithm converges to the optimal solutions while the original SD-GS-AL does not converge for both test cases. The ADMM does not converge for the first test case while it converges to a sub-optimal solution, 63 % more than the optimal solution for the second test case. 

   more » « less
2. Policy Gradient in Robust MDPs with Global Convergence Guarantee

   Qiuhao Wang, Chin Pang (January 2023, International Conference on Machine Learning)

   Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties. 

   more » « less
3. Distributed Data-Driven Learning-Based Optimal Dynamic Resource Allocation for Multi-RIS-Assisted Multi-User Ad-Hoc Network

   https://doi.org/10.3390/a17010045  

   Zhang, Yuzhu; Xu, Hao (January 2024, Algorithms)

   This study investigates the problem of decentralized dynamic resource allocation optimization for ad-hoc network communication with the support of reconfigurable intelligent surfaces (RIS), leveraging a reinforcement learning framework. In the present context of cellular networks, device-to-device (D2D) communication stands out as a promising technique to enhance the spectrum efficiency. Simultaneously, RIS have gained considerable attention due to their ability to enhance the quality of dynamic wireless networks by maximizing the spectrum efficiency without increasing the power consumption. However, prevalent centralized D2D transmission schemes require global information, leading to a significant signaling overhead. Conversely, existing distributed schemes, while avoiding the need for global information, often demand frequent information exchange among D2D users, falling short of achieving global optimization. This paper introduces a framework comprising an outer loop and inner loop. In the outer loop, decentralized dynamic resource allocation optimization has been developed for self-organizing network communication aided by RIS. This is accomplished through the application of a multi-player multi-armed bandit approach, completing strategies for RIS and resource block selection. Notably, these strategies operate without requiring signal interaction during execution. Meanwhile, in the inner loop, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm has been adopted for cooperative learning with neural networks (NNs) to obtain optimal transmit power control and RIS phase shift control for multiple users, with a specified RIS and resource block selection policy from the outer loop. Through the utilization of optimization theory, distributed optimal resource allocation can be attained as the outer and inner reinforcement learning algorithms converge over time. Finally, a series of numerical simulations are presented to validate and illustrate the effectiveness of the proposed scheme. 

   more » « less
4. An Exponentially Convergent Primal-Dual Algorithm for Nonsmooth Composite Minimization

   https://doi.org/10.1109/CDC.2018.8619760  

   Ding, Dongsheng; Hu, Bin; Dhingra, Neil K.; Jovanovic, Mihailo R. (December 2018, 2018 IEEE Conference on Decision and Control (CDC))

   We consider a class of nonsmooth convex composite optimization problems, where the objective function is given by the sum of a continuously differentiable convex term and a potentially non-differentiable convex regularizer. In [1], the authors introduced the proximal augmented Lagrangian method and derived the resulting continuous-time primal-dual dynamics that converge to the optimal solution. In this paper, we extend these dynamics from continuous to discrete time via the forward Euler discretization. We prove explicit bounds on the exponential convergence rates of our proposed algorithm with a sufficiently small step size. Since a larger step size can improve the convergence speed, we further develop a linear matrix inequality (LMI) condition which can be numerically solved to provide rate certificates with general step size choices. In addition, we prove that a large range of step size values can guarantee exponential convergence. We close the paper by demonstrating the performance of the proposed algorithm via computational experiments. 

   more » « less
5. A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization

   https://doi.org/10.1007/s10208-021-09513-z  

   Berahas, Albert S.; Cao, Liyuan; Choromanski, Krzysztof; Scheinberg, Katya (May 2021, Foundations of Computational Mathematics)

   null (Ed.)

   In this paper, we analyze several methods for approximating gradients of noisy functions using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing, and smoothing on a sphere. The methods differ in the number of functions sampled, the choice of the sample points, and the way in which the gradient approximations are derived. For each method, we derive bounds on the number of samples and the sampling radius which guarantee favorable convergence properties for a line search or fixed step size descent method. To this end, we use the results in Berahas et al. (Global convergence rate analysis of a generic line search algorithm with noise, arXiv:​1910.​04055, 2019) and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere. Finally, we present numerical results evaluating the quality of the gradient approximations as well as their performance in conjunction with a line search derivative-free optimization algorithm. 

   more » « less