In this work, we propose a computationally efficient variable importance estimation procedure for model-agnostic and distribution-free settings with theoretical guarantees that leverages a lazy retraining framework inspired by [1]}.
The key idea is to train a new model on the transformed training data, akin to retraining, but on a linearized version of the model centered around model parameters learned from the original (unreduced) training data. We perform ridge regression on this linearized model in the gradient feature space, meaning that our lazy retraining procedure can be computed very quickly. The resulting method, when applied to wide neural network models, admits error bounds that show it is nearly as accurate as full retraining, while computationally it is nearly as fast as dropout. Our theoretical bounds are complemented by a collection of simulations that explore the limitations of dropout and benefits of lazy retraining under a variety of conditions and an application to understanding the importance of various climate indices in a seasonal forecasting task.
