As the outputs of neural network are not independent given the shared parameters, another perspective of studying the inductive bias is to investigate the distribution over the function values, i.e. \(p(f({\bf x}_1),f({\bf x}_2),\cdots ,f({\bf x}_N))\) , indexed by the set of inputs. This is a more challenging task than the above marginal distribution over the function at single input. Fortunately, the central limit theorem applies when the number of activation units becomes infinity, the multivariate distribution converges to Gaussian, and the limiting statistics only depends on the mean \(\operatorname{\mathbb {E}}[f({\bf x})]\)  and covariance \(\operatorname{\mathbb {E}}[f({\bf x})f({\bf y})]\) . Closed form covariance functions can be derived for shallow networks with sigmoidal and ReLu activations [1]}, [2]}, but the same techniques do not seem to carry to the deeper networks. As for the deep networks of finite width, various techniques from statistical physics [3]}, [4]}, [5]} have been employed to compute the corrections.
