More Like this

1. An efficient algorithm for community detection in complex weighted networks

   https://doi.org/10.1002/aic.17205  

   Samandari_Masooleh, Leila; Arbogast, Jeffrey_E; Seider, Warren_D; Oktem, Ulku; Soroush, Masoud (February 2021, AIChE Journal)

   Abstract Community detection decomposes large‐scale, complex networks “optimally” into sets of smaller sub‐networks. It finds sub‐networks that have the least inter‐connections and the most intra‐connections. This article presents an efficient community detection algorithm that detects community structures in a weighted network by solving a multi‐objective optimization problem. The whale optimization algorithm is extended to enable it to handle multi‐objective optimization problems with discrete variables and to solve the problems on parallel processors. To this end, the population's positions are discretized using a transfer function that maps real variables to discrete variables, the initialization steps for the algorithm are modified to prevent generating unrealistic connections between variables, and the updating step of the algorithm is redefined to produce integer numbers. To identify the community configurations that are Pareto optimal, the non‐dominated sorting concept is adopted. The proposed algorithm is tested on the Tennessee Eastman process and several benchmark community‐detection problems. 

   more » « less
2. Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approach

   Jin Qiujiang.; Mokhtari, Aryan (January 2021, Advances in Neural Information Processing Systems (NeurIPS 2021))

   In this paper, we study the application of quasi-Newton methods for solving empirical risk minimization (ERM) problems defined over a large dataset. Traditional deterministic and stochastic quasi-Newton methods can be executed to solve such problems; however, it is known that their global convergence rate may not be better than first-order methods, and their local superlinear convergence only appears towards the end of the learning process. In this paper, we use an adaptive sample size scheme that exploits the superlinear convergence of quasi-Newton methods globally and throughout the entire learning process. The main idea of the proposed adaptive sample size algorithms is to start with a small subset of data points and solve their corresponding ERM problem within its statistical accuracy, and then enlarge the sample size geometrically and use the optimal solution of the problem corresponding to the smaller set as an initial point for solving the subsequent ERM problem with more samples. We show that if the initial sample size is sufficiently large and we use quasi-Newton methods to solve each subproblem, the subproblems can be solved superlinearly fast (after at most three iterations), as we guarantee that the iterates always stay within a neighborhood that quasi-Newton methods converge superlinearly. Numerical experiments on various datasets confirm our theoretical results and demonstrate the computational advantages of our method. 

   more » « less
3. Safe and Efficient Model Predictive Control Using Neural Networks: An Interior Point Approach

   Tabas, D; Zhang, B. (January 2022, Proceedings of the IEEE Conference on Decision and Control)

   Model predictive control (MPC) provides a useful means for controlling systems with constraints, but suffers from the computational burden of repeatedly solving an optimization problem in real time. Offline (explicit) solutions for MPC attempt to alleviate real time computational challenges using either multiparametric programming or machine learning. The multiparametric approaches are typically applied to linear or quadratic MPC problems, while learning-based approaches can be more flexible and are less memory-intensive. Existing learning-based approaches offer significant speedups, but the challenge becomes ensuring constraint satisfaction while maintaining good performance. In this paper, we provide a neural network parameterization of MPC policies that explicitly encodes the constraints of the problem. By exploring the interior of the MPC feasible set in an unsupervised learning paradigm, the neural network finds better policies faster than projection-based methods and exhibits substantially shorter solve times. We use the proposed policy to solve a robust MPC problem, and demonstrate the performance and computational gains on a standard test system. 

   more » « less
4. Distributed block-diagonal approximation methods for regularized empirical risk minimization

   https://doi.org/10.1007/s10994-019-05859-2  

   Lee, Ching-pei; Chang, Kai-Wei (January 2019, Machine Learning)

   In recent years, there is a growing need to train machine learning models on a huge volume of data. Therefore, designing efficient distributed optimization algorithms for empirical risk minimization (ERM) has become an active and challenging research topic. In this paper, we propose a flexible framework for distributed ERM training through solving the dual problem, which provides a unified description and comparison of existing methods. Our approach requires only approximate solutions of the sub-problems involved in the optimization process, and is versatile to be applied on many large-scale machine learning problems including classification, regression, and structured prediction. We show that our framework enjoys global linear convergence for a broad class of non-strongly-convex problems, and some specific choices of the sub-problems can even achieve much faster convergence than existing approaches by a refined analysis. This improved convergence rate is also reflected in the superior empirical performance of our method. 

   more » « less
5. slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

   Newman, Elizabeth; Chung, Julianne; Chung, Matthias; Ruthotto, Lars (July 2022, SIAM journal on scientific computing)

   Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include non-convexity, non-smoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving many training problems, which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with non-standard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, slimTrain does not require a learning rate and automatically adapts the regularization parameter. Since our method operates on mini-batches, its computational overhead per iteration is modest. In our numerical experiments, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters. 

   more » « less