More Like this

1. End-to-End Constrained Optimization Learning: A Survey

   https://doi.org/10.24963/ijcai.2021/610  

   Kotary, James ; Fioretto, Ferdinando ; Van Hentenryck, Pascal ; Wilder, Bryan ( August 2021 , International Joint Conference on Artificial Intelligence)

   This paper surveys the recent attempts at leveraging machine learning to solve constrained optimization problems. It focuses on surveying the work on integrating combinatorial solvers and optimization methods with machine learning architectures.These approaches hold the promise to develop new hybrid machine learning and optimization methods to predict fast, approximate, solutions to combinatorial problems and to enable structural logical inference. This paper presents a conceptual review of the recent advancements in this emerging area. 

   more » « less
2. From Optimization Dynamics to Generalization Bounds via {\L}ojasiewicz Gradient Inequality

   Fusheng Liu ; Haizhao Yang ; Soufiane Hayou ; Qianxiao Li ( October 2022 , Transactions on machine learning research)

   Yiming Ying (Ed.)

   Optimization and generalization are two essential aspects of statistical machine learning. In this paper, we propose a framework to connect optimization with generalization by analyz- ing the generalization error based on the optimization trajectory under the gradient flow algorithm. The key ingredient of this framework is the Uniform-LGI, a property that is generally satisfied when training machine learning models. Leveraging the Uniform-LGI, we first derive convergence rates for gradient flow algorithm, then we give generalization bounds for a large class of machine learning models. We further apply our framework to three distinct machine learning models: linear regression, kernel regression, and two-layer neural networks. Through our approach, we obtain generalization estimates that match or extend previous results. 

   more » « less
3. Wasserstein Distributionally Robust Optimization and Variation Regularization

   https://doi.org/10.1287/opre.2022.2383  

   Gao, Rui ; Chen, Xi ; Kleywegt, Anton J. ( November 2022 , Operations Research)

   This paper builds a bridge between two areas in optimization and machine learning by establishing a general connection between Wasserstein distributional robustness and variation regularization. It helps to demystify the empirical success of Wasserstein distributionally robust optimization and devise new regularization schemes for machine learning. 

   more » « less
4. ASYNC: A Cloud Engine with Asynchrony and History for Distributed Machine Learning

   https://doi.org/10.1109/IPDPS47924.2020.00052  

   Soori, Saeed ; Can, Bugra ; Gurbuzbalaban, Mert ; Dehnavi, Maryam Mehri ( May 2020 , 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS))

   null (Ed.)

   ASYNC is a framework that supports the implementation of asynchrony and history for optimization methods on distributed computing platforms. The popularity of asynchronous optimization methods has increased in distributed machine learning. However, their applicability and practical experimentation on distributed systems are limited because current bulk-processing cloud engines do not provide a robust support for asynchrony and history. With introducing three main modules and bookkeeping system-specific and application parameters, ASYNC provides practitioners with a framework to implement asynchronous machine learning methods. To demonstrate ease-of-implementation in ASYNC, the synchronous and asynchronous variants of two well-known optimization methods, stochastic gradient descent and SAGA, are demonstrated in ASYNC. 

   more » « less
5. Entropy-Penalized Semidefinite Programming

   https://doi.org/10.24963/ijcai.2019/157  

   Krechetov, Mikhail ; Marecek, Jakub ; Maximov, Yury ; Takac, Martin ( August 2019 , Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence)

   Low-rank methods for semi-definite programming (SDP) have gained a lot of interest recently, especially in machine learning applications. Their analysis often involves determinant-based or Schatten-norm penalties, which are difficult to implement in practice due to high computational efforts. In this paper, we propose Entropy-Penalized Semi-Definite Programming (EP-SDP), which provides a unified framework for a broad class of penalty functions used in practice to promote a low-rank solution. We show that EP-SDP problems admit an efficient numerical algorithm, having (almost) linear time complexity of the gradient computation; this makes it useful for many machine learning and optimization problems. We illustrate the practical efficiency of our approach on several combinatorial optimization and machine learning problems.

    

   more » « less