More Like this

1. Convex Geometry and Duality of Over-parameterized Neural Networks

   Ergen, Tolga; Pilanci, Mert (January 2000, International Conference on Artificial Intelligence and Statistics (AISTATS 2020))

   null (Ed.)

   We develop a convex analytic framework for ReLU neural networks which elucidates the inner workings of hidden neurons and their function space characteristics. We show that neural networks with rectified linear units act as convex regularizers, where simple solutions are encouraged via extreme points of a certain convex set. For one dimensional regression and classification, as well as rank-one data matrices, we prove that finite two-layer ReLU networks with norm regularization yield linear spline interpolation. We characterize the classification decision regions in terms of a closed form kernel matrix and minimum L1 norm solutions. This is in contrast to Neural Tangent Kernel which is unable to explain neural network predictions with finitely many neurons. Our convex geometric description also provides intuitive explanations of hidden neurons as auto encoders. In higher dimensions, we show that the training problem for two-layer networks can be cast as a finite dimensional convex optimization problem with infinitely many constraints. We then provide a family of convex relaxations to approximate the solution, and a cutting-plane algorithm to improve the relaxations. We derive conditions for the exactness of the relaxations and provide simple closed form formulas for the optimal neural network weights in certain cases. We also establish a connection to ℓ0-ℓ1 equivalence for neural networks analogous to the minimal cardinality solutions in compressed sensing. Extensive experimental results show that the proposed approach yields interpretable and accurate models. 

   more » « less
2. Submodularity in Conic Quadratic Mixed 0–1 Optimization

   https://doi.org/10.1287/opre.2019.1888  

   Atamtürk, Alper; Gómez, Andrés (January 2020, Operations Research)

   We describe strong convex valid inequalities for conic quadratic mixed 0–1 optimization. These inequalities can be utilized for solving numerous practical nonlinear discrete optimization problems from value-at-risk minimization to queueing system design, from robust interdiction to assortment optimization through appropriate conic quadratic mixed 0–1 relaxations. The inequalities exploit the submodularity of the binary restrictions and are based on the polymatroid inequalities over binaries for the diagonal case. We prove that the convex inequalities completely describe the convex hull of a single conic quadratic constraint as well as the rotated cone constraint over binary variables and unbounded continuous variables. We then generalize and strengthen the inequalities by incorporating additional constraints of the optimization problem. Computational experiments on mean-risk optimization with correlations, assortment optimization, and robust conic quadratic optimization indicate that the new inequalities strengthen the convex relaxations substantially and lead to significant performance improvements. 

   more » « less
3. Convexification of Bilinear Terms over Network Polytopes

   https://doi.org/10.1287/moor.2023.0001  

   Khademnia, Erfan; Davarnia, Danial (April 2024, Mathematics of Operations Research)

   It is well-known that the McCormick relaxation for the bilinear constraint z = xy gives the convex hull over the box domains for x and y. In network applications where the domain of bilinear variables is described by a network polytope, the McCormick relaxation, also referred to as linearization, fails to provide the convex hull and often leads to poor dual bounds. We study the convex hull of the set containing bilinear constraints [Formula: see text] where x_irepresents the arc-flow variable in a network polytope, and y_jis in a simplex. For the case where the simplex contains a single y variable, we introduce a systematic procedure to obtain the convex hull of the above set in the original space of variables, and show that all facet-defining inequalities of the convex hull can be obtained explicitly through identifying a special tree structure in the underlying network. For the generalization where the simplex contains multiple y variables, we design a constructive procedure to obtain an important class of facet-defining inequalities for the convex hull of the underlying bilinear set that is characterized by a special forest structure in the underlying network. Computational experiments conducted on different applications show the effectiveness of the proposed methods in improving the dual bounds obtained from alternative techniques. Funding: This work was supported by Air Force Office of Scientific Research [Grant FA9550-23-1-0183]; National Science Foundation, Division of Civil, Mechanical and Manufacturing Innovation [Grant 2338641]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2023.0001 . 

   more » « less
4. Sparse and smooth signal estimation: Convexification of L0 formulations

   Atamturk, A; Gomez, A; Han, S (February 2021, Journal of machine learning research)

   Mirrokni, V (Ed.)

   Signal estimation problems with smoothness and sparsity priors can be naturally modeled as quadratic optimization with L0-“norm” constraints. Since such problems are non-convex and hard-to-solve, the standard approach is, instead, to tackle their convex surrogates based on L1-norm relaxations. In this paper, we propose new iterative (convex) conic quadratic relaxations that exploit not only the L0-“norm” terms, but also the fitness and smoothness functions. The iterative convexification approach substantially closes the gap between the L0-“norm” and its L1 surrogate. These stronger relaxations lead to significantly better estimators than L1-norm approaches and also allow one to utilize affine sparsity priors. In addition, the parameters of the model and the resulting estimators are easily interpretable. Experiments with a tailored Lagrangian decomposition method indicate that the proposed iterative convex relaxations yield solutions within 1% of the exact L0-approach, and can tackle instances with up to 100,000 variables under one minute. 

   more » « less
5. Totally Symmetric Self-Complementary Plane Partition Matrices and Related Polytopes

   https://doi.org/10.1007/s00026-024-00723-0  

   Holmlund, Vincent; Striker, Jessica (November 2024, Annals of Combinatorics)

   Abstract Plane partitions in the totally symmetric self-complementary symmetry class (TSSCPPs) are known to be equinumerous with$$n\times n$$ $n\times n$alternating sign matrices, but no explicit bijection is known. In this paper, we give a bijection from these plane partitions to$$\{0,1,-1\}$$ $\{0,1,-1\}$-matrices we call magog matrices, some of which are alternating sign matrices. We explore enumerative properties of these matrices related to natural statistics such as inversion number and number of negative ones. We then investigate the polytope defined as their convex hull. We show that all the magog matrices are extreme and give a partial inequality description. Finally, we define another TSSCPP polytope as the convex hull of TSSCPP boolean triangles and determine its dimension, inequalities, vertices, and facets. 

   more » « less