More Like this

1. End-to-End Constrained Optimization Learning: A Survey

   https://doi.org/10.24963/ijcai.2021/610  

   Kotary, James; Fioretto, Ferdinando; Van Hentenryck, Pascal; Wilder, Bryan (August 2021, International Joint Conference on Artificial Intelligence)

   This paper surveys the recent attempts at leveraging machine learning to solve constrained optimization problems. It focuses on surveying the work on integrating combinatorial solvers and optimization methods with machine learning architectures.These approaches hold the promise to develop new hybrid machine learning and optimization methods to predict fast, approximate, solutions to combinatorial problems and to enable structural logical inference. This paper presents a conceptual review of the recent advancements in this emerging area. 

   more » « less
2. From Optimization Dynamics to Generalization Bounds via {\L}ojasiewicz Gradient Inequality

   Fusheng Liu; Haizhao Yang; Soufiane Hayou; Qianxiao Li (October 2022, Transactions on machine learning research)

   Yiming Ying (Ed.)

   Optimization and generalization are two essential aspects of statistical machine learning. In this paper, we propose a framework to connect optimization with generalization by analyz- ing the generalization error based on the optimization trajectory under the gradient flow algorithm. The key ingredient of this framework is the Uniform-LGI, a property that is generally satisfied when training machine learning models. Leveraging the Uniform-LGI, we first derive convergence rates for gradient flow algorithm, then we give generalization bounds for a large class of machine learning models. We further apply our framework to three distinct machine learning models: linear regression, kernel regression, and two-layer neural networks. Through our approach, we obtain generalization estimates that match or extend previous results. 

   more » « less
3. Wasserstein Distributionally Robust Optimization and Variation Regularization

   https://doi.org/10.1287/opre.2022.2383  

   Gao, Rui; Chen, Xi; Kleywegt, Anton J. (November 2022, Operations Research)

   This paper builds a bridge between two areas in optimization and machine learning by establishing a general connection between Wasserstein distributional robustness and variation regularization. It helps to demystify the empirical success of Wasserstein distributionally robust optimization and devise new regularization schemes for machine learning. 

   more » « less
4. Learning From Mistakes: A Multilevel Optimization Framework

   Zhang, Li; Garg, Bhanu; Sridhara, Pradyumna; Hosseini, Ramtin; Xie, Pengtao (January 2025, IEEE transactions on artificial intelligence)

   Bi-level optimization methods in machine learning are popularly effective in subdomains of neural architecture search, data reweighting, etc. However, most of these methods do not factor in variations in learning difficulty, which limits their performance in real-world applications. To address the above problems, we propose a framework that imitates the learning process of humans. In human learning, learners usually focus more on the topics where mistakes have been made in the past to deepen their understanding and master the knowledge. Inspired by this effective human learning technique, we propose a multilevel optimization framework, learning from mistakes (LFM), for machine learning. We formulate LFM as a three-stage optimization problem: 1) the learner learns, 2) the learner relearns based on the mistakes made before, and 3) the learner validates his learning. We develop an efficient algorithm to solve the optimization problem. We further apply our method to differentiable neural architecture search and data reweighting. Extensive experiments on CIFAR-10, CIFAR-100, ImageNet, and other related datasets powerfully demonstrate the effectiveness of our approach. The code of LFM is available at: https://github.com/importZL/LFM. 

   more » « less
5. ASYNC: A Cloud Engine with Asynchrony and History for Distributed Machine Learning

   https://doi.org/10.1109/IPDPS47924.2020.00052  

   Soori, Saeed; Can, Bugra; Gurbuzbalaban, Mert; Dehnavi, Maryam Mehri (May 2020, 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS))

   null (Ed.)

   ASYNC is a framework that supports the implementation of asynchrony and history for optimization methods on distributed computing platforms. The popularity of asynchronous optimization methods has increased in distributed machine learning. However, their applicability and practical experimentation on distributed systems are limited because current bulk-processing cloud engines do not provide a robust support for asynchrony and history. With introducing three main modules and bookkeeping system-specific and application parameters, ASYNC provides practitioners with a framework to implement asynchronous machine learning methods. To demonstrate ease-of-implementation in ASYNC, the synchronous and asynchronous variants of two well-known optimization methods, stochastic gradient descent and SAGA, are demonstrated in ASYNC. 

   more » « less