It was suggested that the superior expressive power of deep linear network and ReLu network is related to their non-Gaussian prior function distribution [1]}, [2]}, [3]}. Besides the network with finite width which lifts the Gaussianity [4]}, the following shallow network \(f_{\psi }:\mathbb {R}^D\mapsto \mathbb {R}\)  with modified preactivation,
\(f_{\psi }({\bf x}) = \frac{1}{\sqrt{n}}\sum _{i=1}^{n} w^c_i\cos [\omega _i\cdot {\bf x}+\psi ({\bf x})] +w^s_i\sin [\omega _i\cdot {\bf x}-\psi ({\bf x})]\:,\) 
