We consider learning two layer neural networks using stochastic gradient descent. The mean-field description of this learning dynamics approximates the evolution of the network weights by an evolution in the space of probability distributions in @math (where @math is the number of parameters associated to each neuron). This evolution can be defined through a partial differential equation or, equivalently, as the gradient flow in the Wasserstein space of probability distributions. Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension @math . In this paper we establish stronger and more general approximation guarantees. First of all, we show that the number of hidden units only needs to be larger than a quantity dependent on the regularity properties of the data, and independent of the dimensions. Next, we generalize this analysis to the case of unbounded activation functions, which was not covered by earlier bounds. We extend our results to noisy stochastic gradient descent. Finally, we show that kernel ridge regression can be recovered as a special limit of the mean field analysis.
The mean-field description was exploited in several papers to establish global convergence results. In @cite_16 global convergence was proved in special examples, and in a general setting for noisy SGD. The papers @cite_10 @cite_20 studied global convergence by exploiting the homogeneity properties of Eq. . In particular, @cite_20 proves a general global convergence result. For initial conditions @math with full support, the PDE converges to a global minimum provided activations are homogeneous in the parameters. Notice that the presence of unbounded second layer coefficients is crucial in order to achieve homogeneity. Unfortunately, the results of @cite_20 do not provide quantitative approximation bounds relating the PDE to finite- @math SGD. The present paper fills this gap by establishing approximation bounds that apply to the setting of @cite_20 .
@cite_16 @cite_10 @cite_20
2107101459 2121999212 2085186189 1968517161
We study quantitatively the effective large-scale behavior of discrete elliptic equations on the lattice ( Z^d ) with random coefficients. The theory of stochastic homogenization relates the random, stationary, and ergodic field of coefficients with a deterministic matrix of effective coefficients. This is done via the corrector problem, which can be viewed as a highly degenerate elliptic equation on the infinite-dimensional space of admissible coefficient fields. In this contribution we develop new quantitative methods for the corrector problem based on the assumption that ergodicity holds in the quantitative form of a Spectral Gap Estimate w.r.t. a Glauber dynamics on coefficient fields—as it is the case for independent and identically distributed coefficients. As a main result we prove an optimal decay in time of the semigroup associated with the corrector problem (i.e. of the generator of the process called “random environment as seen from the particle”). As a corollary we recover existence of stationary correctors (in dimensions (d>2 )) and prove new optimal estimates for regularized versions of the corrector (in dimensions (d 2 )). We also give a self-contained proof of a new estimate on the gradient of the parabolic, variable-coefficient Green’s function, which is a crucial analytic ingredient in our approach. As an application of these results, we prove the first (and optimal) estimates for the approximation of the homogenized coefficients by the popular periodization method in case of independent and identically distributed coefficients. This article deals with the numerical approximation of effective coefficients in stochastic homogenization of discrete linear elliptic equations. The originality of this work is the use of a well-known abstract spectral representation formula to design and analyze effective and computable approximations of the homogenized coefficients. In particular, we show that information on the edge of the spectrum of the generator of the environment viewed by the particle projected on the local drift yields bounds on the approximation error, and conversely. Combined with results by Otto and the first author in low dimension, and results by the second author in high dimension, this allows us to prove that for any dimension d ≥ 2, there exists an explicit numerical strategy to approximate homogenized coefficients which converges at the rate of the central limit theorem. We consider a random, uniformly elliptic coefficient field (a(x) ) on the (d )-dimensional integer lattice ( Z ^d ). We are interested in the spatial decay of the quenched elliptic Green function (G(a;x,y) ). Next to stationarity, we assume that the spatial correlation of the coefficient field decays sufficiently fast to the effect that a logarithmic Sobolev inequality holds for the ensemble ( ). We prove that all stochastic moments of the first and second mixed derivatives of the Green function, that is, ( | _x G(x,y)|^p ) and ( | _x _y G(x,y)|^p ), have the same decay rates in (|x-y| 1 ) as for the constant coefficient Green function, respectively. This result relies on and substantially extends the one by Delmotte and Deuschel (Probab Theory Relat Fields 133:358–390, 2005), which optimally controls second moments for the first derivatives and first moments of the second mixed derivatives of (G ), that is, ( | _x G(x,y)|^2 ) and ( | _x _y G(x,y)| ). As an application, we are able to obtain optimal estimates on the random part of the homogenization error even for large ellipticity contrast. Performance evaluation of the 802.11 MAC protocol is classically based on the decoupling assumption, which hypothesizes that the backoff processes at different nodes are independent. This decoupling assumption results from mean field convergence and is generally true in transient regime in the asymptotic sense (when the number of wireless nodes tends to infinity), but, contrary to widespread belief, may not necessarily hold in stationary regime. The issue is often related with the existence and uniqueness of a solution to a fixed point equation; however, it was also recently shown that this condition is not sufficient; in contrast, a sufficient condition is a global stability property of the associated ordinary differential equation. In this paper, we give a simple condition that establishes the asymptotic validity of the decoupling assumption for the homogeneous case (all nodes have the same parameters). We also discuss the heterogeneous and the differentiated service cases and formulate a new ordinary differential equation. We show that the uniqueness of a solution to the associated fixed point equation is not sufficient; we exhibit one case where the fixed point equation has a unique solution but the decoupling assumption is not valid in the asymptotic sense in stationary regime.