An official website of the United States government
Many classical examples of models of self-organized dynamics, including the Cucker–Smale, Motsch–Tadmor, multi-species, and several others, include an alignment force that is based upon density-weighted averaging protocol. Those protocols can be viewed as special cases of “environmental averaging”. In this paper we formalize this concept and introduce a unified framework for systematic analysis of alignment models.A series of studies are presented including the mean-field limit in deterministic and stochastic settings, hydrodynamic limits in the monokinetic and Maxwellian regimes, hypocoercivity and global relaxation for dissipative kinetic models, several general alignment results based on chain connectivity and spectral gap analysis. These studies cover many of the known results and reveal new ones, which include asymptotic alignment criteria based on connectivity conditions, new estimates on the spectral gap of the alignment force that do not rely on the upper bound of the macroscopic density, uniform gain of positivity for solutions of the Fokker–Planck-alignment model based on smooth environmental averaging. As a consequence, we establish unconditional relaxation result for global solutions to the Fokker–Planck-alignment model, which presents a substantial improvement over previously known perturbative results.
more »
« less
A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.
more »
« less
- Award ID(s):
- 1922658
- PAR ID:
- 10219212
- Date Published:
- Journal Name:
- Journal of machine learning research
- ISSN:
- 1533-7928
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We present a new framework to analyze accelerated stochastic mirror descent through the lens of continuous-time stochastic dynamic systems. It enables us to design new algorithms, and perform a unified and simple analysis of the convergence rates of these algorithms. More specifically, under this framework, we provide a Lyapunov function based analysis for the continuous-time stochastic dynamics, as well as several new discrete-time algorithms derived from the continuous-time dynamics. We show that for general convex objective functions, the derived discrete-time algorithms attain the optimal convergence rate. Empirical experiments corroborate our theory.more » « less
-
Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate this problem. In this paper, we propose new surrogate losses to cast the problem of learning the optimal stepsizes for the stochastic optimization of a non-convex smooth objective function onto an online convex optimization problem. This allows the use of no-regret online algorithms to compute optimal stepsizes on the fly. In turn, this results in a SGD algorithm with self-tuned stepsizes that guarantees convergence rates that are automatically adaptive to the level of noise.more » « less
-
Stress, edge crossings, and crossing angles play an important role in the quality and readability of graph drawings. Most standard graph drawing algorithms optimize one of these criteria which may lead to layouts that are deficient in other criteria. We introduce an optimization framework, Stress-Plus-X (SPX), that simultaneously optimizes stress together with several other criteria: edge crossings, minimum cross- ing angle, and upwardness (for directed acyclic graphs). SPX achieves results that are close to the state-of-the-art algorithms that optimize these metrics individually. SPX is flexible and extensible and can optimize a subset or all of these criteria simultaneously. Our experimental analysis shows that our joint optimization approach is successful in drawing graphs with good performance across readability criteria.more » « less
-
We present a computational study of several preconditioning techniques for the GMRES algorithm applied to the stochastic diffusion equation with a lognormal coefficient discretized with the stochastic Galerkin method. The clear block structure of the system matrix arising from this type of discretization motivates the analysis of preconditioners designed according to a field-splitting strategy of the stochastic variables. This approach is inspired by a similar procedure used within the framework of physics based preconditioners for deterministic problems, and its application to stochastic PDEs represents the main novelty of this work. Our numerical investigation highlights the superior properties of the field-split type preconditioners over other existing strategies in terms of computational time and stochastic parameter dependence.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
