skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks
Abstract We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Starting from a dynamical mean field theory description of infinite width deep neural network kernel and prediction dynamics, we provide a characterization of the O ( 1 / width ) fluctuations of the dynamical mean field theory order parameters over random initializations of the network weights. Our results, while perturbative in width, unlike prior analyses, are non-perturbative in the strength of feature learning. We find that once the mean field/µP parameterization is adopted, the leading finite size effect on the dynamics is to introduce initialization variance in the predictions and feature kernels of the networks. In the lazy limit of network training, all kernels are random but static in time and the prediction variance has a universal form. However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently. In two layer networks, we show how feature learning can dynamically reduce the variance of the final tangent kernel and final network predictions. We also show how initialization variance can slow down online learning in wide but finite networks. In deeper networks, kernel variance can dramatically accumulate through subsequent layers at large feature learning strengths, but feature learning continues to improve the signal-to-noise ratio of the feature kernels. In discrete time, we demonstrate that large learning rate phenomena such as edge of stability effects can be well captured by infinite width dynamics and that initialization variance can decrease dynamically. For convolutional neural networks trained on CIFAR-10, we empirically find significant corrections to both the bias and variance of network dynamics due to finite width.  more » « less
Award ID(s):
2134157 2239780
PAR ID:
10561215
Author(s) / Creator(s):
;
Publisher / Repository:
IOP Publishing Ltd
Date Published:
Journal Name:
Journal of Statistical Mechanics: Theory and Experiment
Volume:
2024
Issue:
10
ISSN:
1742-5468
Page Range / eLocation ID:
104021
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract A search for resonances in top quark pair ( t t ) production in final states with two charged leptons and multiple jets is presented, based on proton–proton collision data collected by the CMS experiment at the CERN LHC at s = 13 TeV , corresponding to 138 fb−1. The analysis explores the invariant mass of the t t system and two angular observables that provide direct access to the correlation of top quark and antiquark spins. A significant excess of events is observed near the kinematic t t threshold compared to the non-resonant production predicted by fixed-order perturbative quantum chromodynamics (pQCD). The observed enhancement is consistent with the production of a color-singlet pseudoscalar ( 1 S 0 [ 1 ] ) quasi-bound toponium state, as predicted by non-relativistic quantum chromodynamics. Using a simplified model for 1 S 0 [ 1 ] toponium, the cross section of the excess above the pQCD prediction is measured to be 8.8 1.4 + 1.2 pb
    more » « less
  2. Abstract A test of lepton flavor universality in B ± K ± μ + μ and B ± K ± e + e decays, as well as a measurement of differential and integrated branching fractions of a nonresonant B ± K ± μ + μ decay are presented. The analysis is made possible by a dedicated data set of proton-proton collisions at s = 13 TeV recorded in 2018, by the CMS experiment at the LHC, using a special high-rate data stream designed for collecting about 10 billion unbiased b hadron decays. The ratio of the branching fractions B ( B ± K ± μ + μ ) to B ( B ± K ± e + e ) is determined from the measured double ratio R ( K ) of these decays to the respective branching fractions of the B ± J / ψ K ± with J / ψ μ + μ and e + e decays, which allow for significant cancellation of systematic uncertainties. The ratio R ( K ) is measured in the range 1.1 < q 2 < 6.0 GeV 2 , whereqis the invariant mass of the lepton pair, and is found to be R ( K ) = 0.78 0.23 + 0.47 , in agreement with the standard model expectation R ( K ) 1 . This measurement is limited by the statistical precision of the electron channel. The integrated branching fraction in the sameq2range, B ( B ± K ± μ + μ ) = ( 12.42 ± 0.68 ) × 10 8 , is consistent with the present world-average value and has a comparable precision. 
    more » « less
  3. Abstract Event-by-event fluctuations of the event-wise mean transverse momentum,$$\langle p_{\textrm{T}}\rangle $$ p T , of charged particles produced in proton–proton (pp) collisions at$$\sqrt{s}$$ s = 5.02 TeV, Xe–Xe collisions at$$\sqrt{s_{\textrm{NN}}}$$ s NN = 5.44 TeV, and Pb–Pb collisions at$$\sqrt{s_{\textrm{NN}}}$$ s NN = 5.02 TeV are studied using the ALICE detector based on the integral correlator$$\langle \!\langle \Delta p_\textrm{T}\Delta p_\textrm{T}\rangle \!\rangle $$ Δ p T Δ p T . The correlator strength is found to decrease monotonically with increasing produced charged-particle multiplicity measured at midrapidity in all three systems. In Xe–Xe and Pb–Pb collisions, the multiplicity dependence of the correlator deviates significantly from a simple power-law scaling as well as from the predictions of the HIJING and AMPT models. The observed deviation from power-law scaling is expected from transverse radial flow in semicentral to central Xe–Xe and Pb–Pb collisions. In pp collisions, the correlation strength is also studied by classifying the events based on the transverse spherocity,$$S_0$$ S 0 , of the particle production at midrapidity, used as a proxy for the presence of a pronounced back-to-back jet topology. Low-spherocity (jetty) events feature a larger correlation strength than those with high spherocity (isotropic). The strength and multiplicity dependence of jetty and isotropic events are well reproduced by calculations with the PYTHIA 8 and EPOS LHC models. 
    more » « less
  4. Abstract The production cross section of inclusive isolated photons has been measured by the ALICE experiment at the CERN LHC in pp collisions at centre-of-momentum energy of$$\sqrt{s} =13$$ s = 13  TeV collected during the LHC Run 2 data-taking period. The measurement is performed by combining the measurements of the electromagnetic calorimeter EMCal and the central tracking detectors ITS and TPC, covering a pseudorapidity range of$$|\eta ^{\gamma }|<0.67$$ | η γ | < 0.67 and a transverse momentum range of$$7 7 < p T γ < 200 GeV/$$c$$ c . The result extends to lower$$p_\textrm{T}^{\gamma }$$ p T γ and$$x_\textrm{T}^{\gamma } = 2p_\textrm{T}^{\gamma }/\sqrt{s} $$ x T γ = 2 p T γ / s ranges, the lowest$$x_\textrm{T}^{\gamma }$$ x T γ of any isolated photon measurements to date, extending significantly those measured by the ATLAS and CMS experiments towards lower$$p_\textrm{T}^{\gamma }$$ p T γ at the same collision energy with a small overlap between the measurements. The measurement is compared with next-to-leading order perturbative QCD calculations and the results from the ATLAS and CMS experiments as well as with measurements at other collision energies. The measurement and theory prediction are in agreement with each other within the experimental and theoretical uncertainties. 
    more » « less
  5. Abstract We consider a process of noncollidingq-exchangeable random walks on Z making steps 0 (‘straight’) and −1 (‘down’). A single random walk is calledq-exchangeable if under an elementary transposition of the neighboring steps ( down , straight ) ( straight , down ) the probability of the trajectory is multiplied by a parameter q ( 0 , 1 ) . Our process ofmnoncollidingq-exchangeable random walks is obtained from the independentq-exchangeable walks via the Doob’sh-transform for a nonnegative eigenfunctionh(expressed via theq-Vandermonde product) with the eigenvalue less than 1. The system ofmwalks evolves in the presence of an absorbing wall at 0. The repulsion mechanism is theq-analogue of the Coulomb repulsion of random matrix eigenvalues undergoing Dyson Brownian motion. However, in our model, the particles are confined to the positive half-line and do not spread as Brownian motions or simple random walks. We show that the trajectory of the noncollidingq-exchangeable walks started from an arbitrary initial configuration forms a determinantal point process, and express its kernel in a double contour integral form. This kernel is obtained as a limit from the correlation kernel ofq-distributed random lozenge tilings of sawtooth polygons. In the limit as m , q = e γ / m withγ > 0 fixed, and under a suitable scaling of the initial data, we obtain a limit shape of our noncolliding walks and also show that their local statistics are governed by the incomplete beta kernel. The latter is a distinguished translation invariant ergodic extension of the two-dimensional discrete sine kernel. 
    more » « less