Our analysis crucially uses Mercer's theorem and the spectral decomposition of the NT kernel in the basis of spherical harmonics (which are special functions on the surface of the hypersphere; see Section ). We derive bounds on the decay rate of the Mercer eigenvalues (referred to hereafter as eigendecay) based on the differentiability of the activation functions. These eigendecays are obtained by careful characterization of a recent result
relating the eigendecay to the asymptotic endpoint expansions of the kernel, given in [1]}. In particular, we consider \(s-1\)  times differentiable activation functions \(a_s(u)=(\max (0,u))^s, s\in \mathbb {N}\) . We show that, with these activation functions and a \(d\)  dimensional hyperspherical support, the eigenvalues \(\lambda _i\)  of the corresponding NT kernel decay at the rate \(\lambda _i\sim i^{-\frac{d+2s-2}{d-1}}\)The general notations used in this paper are defined in Appendix . (up to certain parity constraints on \(i\)  in special cases; see Section  for details).
