The challenge of proving Theorem REF  is to bound the error of the lazy neural network trained using data without a certain variable – note that we are bounding the estimation error (\(\Vert h_{\theta _f + \Delta \theta _j} - f_{0,-j}\Vert \) ) instead of the prediction error (\(\Vert h_{\theta _f + \Delta \theta _j} - f_{0}\Vert \) ) that is the focus of much of the deep learning community, since the predictive
skill
of the reduced model is expected to decrease when an important variable is removed. At a high level, our proof reduces the estimation error of the neural network from lazy training to the error between the NTK estimation and the target function, where we use techniques from kernel ridge regression. The difference here is that most NTK papers (see e.g. [1]}) use random initialization for the parameters and optimization without penalty, while our method starts from a specific initialization (the full model), and requires the penalty parameter \(\lambda \)  to be large (\(\lambda = O(n^{1/2})\) ) to ensure convergence.
