Recent advances in the global convergence theory of large width networks [1]}, [2]}, [3]}, [4]}, [5]} have shown that under certain assumptions, the gradient descent dynamics of a network converge to that of kernel gradient descent as the widths approach infinity, where the kernel is the NTK [1]}, denoted by \(\Theta \) .
As such, in the infinite width limit, when the loss is the mean squared error (MSE), the mean prediction (marginarized over random initialization) can be solved analytically. In particular, when the training time \(t=\infty \) , the prediction is given by
\(&f(\mathcal {X}_*) =\Theta \left(\mathcal {X}_*, \mathcal {X}_T\right)\Theta ^{-1}(\mathcal {X}_T, \mathcal {X}_T)\mathcal {Y}_T \,,\) 
