The study of infinite networks dates back to the seminal work of [1]} who showed the convergence of single hidden-layer networks to Gaussian Processes (GPs). Recently, there has been renewed interest in studying random, infinite networks starting with concurrent work on “conjugate kernels” [2]}, [3]} and “mean-field theory” [4]}, [5]},
taking a statistical learning and statistical physics view of points, respectively. Since then, this analysis has been extended to include a wide range of architectures  [6]}, [7]}, [8]}, [9]}, [10]}, [11]}. The inducing kernel is often referred to as the Neural Network Gaussian Process (NNGP) kernel. The neural tangent kernel (NTK), first introduced in  [12]}, along with follow-up work [13]}, [14]} showed that the distribution of functions induced by gradient descent for infinite-width networks is a Gaussian Process with NTK as the kernel. Since then, NNGP and NTK have become extremely useful and popular tools to study various properties of neural networks  [15]}, [16]}, [17]}, [18]}, [19]}, [20]}, [21]} and many others.
