The connections between deep random networks and GPs have been studied extensively in literature [1]}, [2]}, [3]}, but the link between deep random networks and DGP was lacking.
In fact, deep neural networks can have some narrow layers connecting their neighboring wide ones. [4]} demonstrated that such deep neural networks with bottlenecks are equivalent to DGPs. [5]} exploited the convexity of exponential function, showing bounded characteristic function of DGPs, and its convergence to that of a GP with some limiting kernel as the bottleneck width approaches infinity. Moreover, heavier statistical tails occur in the deeper models, and their conclusion is very general, not depending on the kernels of DGP or activation units of deep neural network.
The deep trigonometriconometric network with bottleneck latent layer was proposed by [6]} as a proxy for the DGPs with SE kernels, based on the random Fourier expansion of SE kernels [7]}, [8]}. Improvements in generalization over expectation propagation [9]} were demonstrated by employing a dedicated variational inference scheme [10]}, [6]}.
