The inductive bias associated with neural networks is connected to the prior distributions from which the random parameters in the computational graph are sampled. How well a model can generalize in Bayesian learning is critically related to its inductive bias [1]}.
While it is usually difficult to describe the inductive bias of neural networks quantitatively, some special cases do permit analytic investigation. [2]} analytically investigated the marginal distribution over the output of deep linear and ReLu networks. The following remark is about a particular shallow linear network.

Consider the linear network \(f({\bf x})={\bf W}{\Omega }{\bf x}\) , a special case of Eq. (REF ) with \(\Phi \)  being identity mapping, and the entries in the random matrices \({\Omega }\in \mathbb {R}^{2\times D}\)  and \({\bf W}\in \mathbb {R}^{1\times 2}\)  are independent normal, i.e. \(\Omega _{ij}\sim \mathcal {N}(0,\sigma _1^2)\)  and \({\bf W}_{ij}\sim \mathcal {N}(0,\sigma _2^2)\) . Then, the marginal distribution over the output is
a Laplace distribution \(p(f({\bf x}))=\exp (-|f({\bf x})|/\kappa )/2\kappa \)  with \(\kappa :=\sigma _1\sigma _2|{\bf x}|\) . The heavy-tailed character is consistent with the findings in [3]}.

 It is easy to observe that the latent output \({\bf h}=\Omega {\bf x}\)  has independent components \(h_i\sim \mathcal {N}(0,\sigma _1^2|{\bf x}|^2)\) . Similarly, conditional on \({\bf h}\) , the output has \(f|{\bf h}\sim \mathcal {N}(0,\sigma _2^2|{\bf h}|^2)\) . To obtain the marginal distribution \(p(f):=\operatorname{\mathbb {E}}_{{\bf W},\Omega }[p(f|{\bf W},\Omega )]=\operatorname{\mathbb {E}}_{\bf h}[p(f|{\bf h})]\) , one can integrate out \({\bf h}\)  during the Fourier transformation and then apply the inverse transform [2]}. Namely, in this particular case with hidden dimension \(n={\rm dim}({\bf h})=2\) , we can get,
\(p(f)=\int \frac{dq}{2\pi }d{\bf h}\ e^{iqf} \tilde{p}(q|{\bf h})p({\bf h})=\int \frac{dq}{2\pi }\frac{e^{iqf}}{1+\sigma _1^2\sigma _2^2|{\bf x}|^2q^2}=\frac{e^{-|f|/\kappa }}{2\kappa }\:.\) 
