in which the random weight matrices \({\bf w}_2\in \mathbb {R}^{1\times 2n_2}\) , \({\bf W}_1\in \mathbb {R}^{H\times 2n_1}\)  and the feature matrices \(\Omega _2\in \mathbb {R}^{n_2\times H}\) , \(\Omega _1\in \mathbb {R}^{n_1\times D}\) . Here, the hidden output \({\bf h}={\bf W}_1\Phi (\Omega _1{\bf x})\)  has bottleneck [1]} dimension \(H\)  collecting the \(n_1\)  postactivations. Besides the compositional hierarchy which makes the function more expressive than its shallow counterpart, one can also adopt different prior distribution over the weight and feature matrices. In the following three subsections, we shall discuss the cases of (i) the entries in \({\bf W}_1\) , \({\bf w}_2\) , \(\Omega _1\) , and \(\Omega _2\)  are all independent normal, which corresponds to the zero-mean two-layer DGP with SE kernels [2]}, (ii) same as in (i) but in the first layer the weight entries in \({\bf W}_1\sim p({\bf w}_1|{\bf Z,U})\)  are not independent, which corresponds to the two-layer DGP for multi-fidelity regression [3]} and hyper-data learning [4]} with \({\bf Z, U}\)  acting as the support in the latent function, and (iii) same as in (ii) but the feature matrix \(\Omega _2\)  consists of samples from the mixture of Gaussians, which corresponds to the two-layer DGP with outer GP using the SM kernel.
