[1]} demonstrated that zero-mean DGPs, \(f_L\circ {\bf f}_{L-1}\circ \cdots \circ {\bf f}_1\) , with any depth \(L\)  have heavy-tailed statistics. Their argument is based on the fact that the characteristic function for DGP has a lower bound which corresponds to the characteristic function of a GP with the limiting kernel when the width in each layer approaches infinity. It is not clear whether we can directly apply their argument to the deep trigonometric network from the weight space view. Focusing on the network defined in Eq. (REF ) with iid \({\bf w}_2\)  and \({\bf W}_1\) , the characteristic function for a set of \(N\)  inputs can be computed as the followings,
\(\operatorname{\mathbb {E}}[\exp (i{\bf t}\cdot {\bf f})]&=\operatorname{\mathbb {E}}\lbrace \exp [it_1f({\bf x}_1)+\cdots +it_Nf({\bf x}_N)]\rbrace \\&=\operatorname{\mathbb {E}}\bigg \lbrace i{\bf w}_2\big [t_1\Phi (\Omega _2{\bf h}({\bf x}_1))+\cdots t_N\Phi (\Omega _2{\bf h}({\bf x}_N))\big ]\bigg \rbrace \\&=\operatorname{\mathbb {E}}[\exp ( -\frac{1}{2}{\bf t}K{\bf t} )]\\&\ge \exp \bigg [-\frac{1}{2}{\bf t}^T\operatorname{\mathbb {E}}_{{\bf W}_1}(K){\bf t}\bigg ] = \exp \big (-\frac{1}{2} {\bf t}^TK_{\rm DGP}{\bf t}\big )\) 
