here is a vast of literature addressing the overparameterization of DNN, its associated nonconvex optimization problem, and the application of the first order (gradient descent) optimization methods.
Three major perspectives have been taken.
One approach is to analyze the landscape of the objective functions, identifying properties that a nonconvex optimization problem needs to satisfy for a first order method to converge to an optimizer.
Different NN structures are then analyzed to fit these properties [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}. The second approach largely depends on the insensibility of the Gram matrix as the number of neurons goes to infinity [11]}, [12]}, [13]}, [14]}, [15]}, as a way to analyze Neural Tangent Kernel [16]}, [17]}. The analysis crucially depends on the fact that GD confines the iteration in a small neighborhood if the initial conditions are random. One key requirement for the NTK approach, however, is that the last layer needs to be linear. The third approach, which we adopt in this paper, involves the mean-field limit.
The idea is to let the number of neurons \(M\)  and/or the number of layers \(L\)  go to infinity, and translate the gradient descent for the parameters in a finite dimensional space to a gradient flow in a (infinite-dimensional) function space [18]}, [19]}, [20]}, [11]}, [12]}, [13]}, [14]}, [25]}, [26]}, [27]}, [28]}, [29]}, [30]}.
Some PDE analysis techniques are borrowed to show the convergence of the limiting gradient flow equation.
In particular, in [25]}, [26]}, [27]}, the authors study the two-layer (one hidden layer) NN in the regime where \(M\)  is infinite.
In these works, both the mean-field limit and the convergence of the gradient flow are proved, giving a complete answer about the convergence to the zero-loss when NN has two layers.
The problem is significantly more complicated for NNs that have multiple layers, since the convexity of the objective function is lost. In [20]} the author derived the mean-field limit, but could not show convergence of the limiting gradient-flow equation. In [28]} the authors showed the convergence of a PDE that can be viewed as a modified version of the true gradient flow, hinting at convergence of the real mean-field limit. To the best of our knowledge, the current paper is the first that both rigorously proves the mean-field limit that translates the optimization problem into a gradient flow, and demonstrates why the gradient flow leads to a global minimizer.
