In this paper, we analytically investigate the deep trigonometriconometric network, an approximate representation of the infinite dimensional DGPs [1]}, [2]}, and make three main contributions. (i) Covariance: we show that the deep trigonometric networks are equivalent to the DGPs with squared exponential kernels as the former in the wide limit yield the same covariance as the latter [3]}. In weight space view, the derivation of covariance relies on the spectrum of a unique class of random matrices. We also study the multivariate characteristic function for understanding its heavy statistical tails. (ii) Expressivity: we show the trig networks can approximate an expressive GP with spectra mixture kernel [4]} if the features are samples from mixture of Gaussians. A deep kernel learning model [5]} is connected with a deep trig network containing a correlated weight layer. In addition, marginal prior distribution [6]} of a shallow trig net can be non-Gaussian if an embedding phase shift network is incorporated. (iii) NTK: translating DGPs to the deep trigonometric network representation we derive a closed form NTK for the corresponding DGPs. Exact DGP inference is impossible due to intractable marginalization in function space, and the NTK can shed light on the predictive mean from weight space view. By connecting DGP with the deep trigonometric networks from covariance perspective, the latter representation can be more flexible, expressive, and transparent than the former through our analytical results including the NTK.
