Spectral bias of DNNs have been the subject of several other empirical and theoretical works [1]}. Approximations to the Fourier transform of two-layer [2]} and multi-layer [3]} ReLU networks show that these networks have a learning bias towards low frequency functions [4]}. To improve the limitations of DNNs in learning high frequency components, empirical works use Fourier features explicitly as part of the input [5]}. A parametrization of polynomial DNNs has also been shown to speed up the learning of higher frequency components in two-layer networks [6]}. Different notions of spectral priors have also been empirically investigated in graph neural networks [7]} and elsewhere [8]}. These results support the implicit bias of DNNs towards dense and low-frequency spectral representation and only motivate our analysis of sparsity as an explicit spectral prior.
