The surprising and singular finding of [1]}
is that for a range of infinitely wide neural network architectures with linear output layer, \(\phi _{\mathbf {w}}({{\mathbf {x}}})\)  is independent of \({\mathbf {w}}\)  in a ball around a random “initialization” point \({\mathbf {w}}_0\) . That can be shown to be equivalent to the linearity of \(f({\mathbf {w}},{\mathbf {x}})\)  in \({\mathbf {w}}\)  (and hence transition to linearity in the limit of infinite width):
\(f({\mathbf {w}},{\mathbf {x}}) = \langle {\mathbf {w}}- {\mathbf {w}}_0,\phi _{{\mathbf {w}}_0}({\mathbf {x}})\rangle + f({\mathbf {w}}_0,{\mathbf {x}})\) 
