Dataset Distillation using Neural Feature Regression
Dataset distillation aims to learn a small synthetic dataset that preserves
most of the information from the original dataset. Dataset distillation can be
formulated as a bi-level meta-learning problem where the outer loop optimizes
the meta-dataset and the inner loop trains a model on the distilled data.
Meta-gradient computation is one of the key challenges in this formulation, as
differentiating through the inner loop learning procedure introduces
significant computation and memory costs. In this paper, we address these
challenges using neural Feature Regression with Pooling (FRePo), achieving the
state-of-the-art performance with an order of magnitude less memory requirement
and two orders of magnitude faster training than previous methods. The proposed
algorithm is analogous to truncated backpropagation through time with a pool of
models to alleviate various types of overfitting in dataset distillation. FRePo
significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and
ImageNet-1K. Furthermore, we show that high-quality distilled data can greatly
improve various downstream applications, such as continual learning and
membership inference defense.