As the width \(n\rightarrow \infty \) , at initialization the output \(f_0(\mathcal {X})\)  forms a Gaussian Process \(f_0(\mathcal {X})\sim \mathcal {GP}(0, \mathcal {K}(\mathcal {X}, \mathcal {X}))\) , known as the NNGP [1]}, [2]}, [3]}. Here \(\mathcal {K}\)  is the GP kernel and can be computed in closed-form for a variety of architectures [4]}. By treating this infinite width network as a Bayesian model (aka Bayesian Neural Networks) and marginalizing over the training set \((\mathcal {X}_T, \mathcal {Y}_T)\) , the posterior is also a GP whose mean is given by \(\mathcal {K}(\mathcal {X}_*, \mathcal {X}_T)\mathcal {K}^{-1}(\mathcal {X}_T, \mathcal {X}_T)\mathcal {Y}_T\) .
