In the special case of ReLU activation function, several recent works [1]}, [2]}, [3]} have proven a \(\tilde{\lambda }_{i}\sim i^{-d}\)  eigendecay for the NT kernel on the hypersphere \(\mathbb {S}^{d-1}\) . In this work, we use a recent result from [3]} to derive the eigendecay based on the differentiability properties of the activation functions.
