Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates
We consider optimizing two-layer neural networks in the mean-field regime
where the learning dynamics of network weights can be approximated by the
evolution in the space of probability measures over the weight parameters
associated with the neurons. The mean-field regime is a theoretically
attractive alternative to the NTK (lazy training) regime which is only
restricted locally in the so-called neural tangent kernel space around
specialized initializations. Several prior works (\cite{mei2018mean,
chizat2018global}) establish the asymptotic global optimality of the mean-field
regime, but it is still challenging to obtain a quantitative convergence rate
due to the complicated nonlinearity of the training dynamics. This work
establishes a new linear convergence result for two-layer neural networks
trained by continuous-time noisy gradient descent in the mean-field regime. Our
result relies on a novelty logarithmic Sobolev inequality for two-layer neural
networks, and uniform upper bounds on the logarithmic Sobolev constants for a
family of measures determined by the evolving distribution of hidden neurons.