While [1]} first pointed out the general correspondence between an infinite neural network and Gaussian process (GP), [2]} demonstrated that neural networks with iid Gaussian weights and sigmoidal activation units are a representation of random functions drawn from a GP with arcsine covariance function. Later, [3]} obtained the arccosine kernel from computing the covariance of outputs from the ReLu neural networks. With the closed form kernels, one is able to transform the prediction task based on a wide Bayesian neural network [4]} to a GP regression problem.
Moreover, the correspondence holds beyond the shallow neural networks. [5]} and [6]} studied the deep and wide neural networks and obtained a recursive relation for the emergent kernels. Similar techniques appeared in earlier work [7]} on derivation of analytic kernels with which the authors identified a transition from a smooth into a chaotic phase.
