Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization. We evaluate our method in various initialization settings.   Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods.   Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images. proposed network distillation as a way to transfer the knowledge from an ensemble of many separately-trained networks into a single, typically compact network, performing a type of model compression. In this paper, we are considering a related but orthogonal task: rather than distilling the model, we propose to distill the dataset. Unlike network distillation, we keep the model fixed but encapsulate the knowledge of the entire training dataset, which typically contains thousands to millions of images, into a small number of synthetic training images. We show that we can go as low as one synthetic image per category, training the same model to reach surprisingly good performance on these synthetic images. For example, in Figure 1a , we compress 60, 000 training images of MNIST digit dataset into only 10 synthetic images (one per category), given a fixed network initialization. Training the standard LENET on these 10 images yields test-time MNIST recognition performance of 94%, compared to 99% for the original dataset. For networks with unknown random weights, 100 synthetic images train to 89%. We name our method Dataset Distillation and these images distilled images. But why is dataset distillation interesting? First, there is the purely scientific question of how much data is encoded in a given training set and how compressible it is? Second, we wish to know whether it is possible to "load up" a given network with an entire dataset-worth of knowledge by a handful of images. This is in contrast to traditional training that often requires tens of thousands of data samples. Finally, on the practical side, dataset distillation enables applications that require compressing data with its task. We demonstrate that under the continual learning setting, storing distilled images as memory of past task and data can alleviate catastrophic forgetting (McCloskey and Cohen, 1989) . A key question is whether it is even possible to compress a dataset into a small set of synthetic data samples. For example, is it possible to train an image classification model on synthetic images that are not on the manifold of natural images? Conventional wisdom would suggest that the answer is no, as the synthetic training data may not follow the same distribution of the real test data. Yet, in this work, we show that this is indeed possible. We present an optimization algorithm for synthesizing a small number of synthetic data samples not only capturing much of the original training data but also tailored explicitly for fast model training with only a few data point. To achieve our goal, we first derive the network weights as a We distill the knowledge of tens of thousands of images into a few synthetic training images called distilled images. On MNIST, 100 distilled images can train a standard LENET with a random initialization to 89% test accuracy, compared to 99% when fully trained. On CIFAR10, 100 distilled images can train a network with a random initialization to 41% test accuracy, compared to 80% when fully trained. In Section 3.6, we show that these distilled images can efficiently store knowledge of previous tasks for continual learning. differentiable function of our synthetic training data. Given this connection, instead of optimizing the network weights for a particular training objective, we optimize the pixel values of our distilled images. However, this formulation requires access to the initial weights of the network. To relax this assumption, we develop a method for generating distilled images for randomly initialized networks. To further boost performance, we propose an iterative version, where the same distilled images are reused over multiple gradient descent steps so that the knowledge can be fully transferred into the model. Finally, we study a simple linear model, deriving a lower bound on the size of distilled data required to achieve the same performance as training on the full dataset. We demonstrate that a handful of distilled images can be used to train a model with a fixed initialization to achieve surprisingly high performance. For networks pre-trained on other tasks, our method can find distilled images for fast model fine-tuning. We test our method on several initialization settings: fixed initialization, random initialization, fixed pre-trained weights, and random pre-trained weights. Extensive experiments on four publicly available datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, show that our approach often outperforms existing methods. Finally, we demonstrate that for continual learning methods that store limited-size past data samples as episodic memory (Lopez-Paz and Ranzato, 2017; Kirkpatrick et al., 2017) , storing our distilled data instead is much more effective. Our distilled images contain richer information about the past data and tasks, and we show experimental evidence on standard continual learning benchmarks. Our code, data, and models will be available upon publication. In this paper, we have presented dataset distillation for compressing the knowledge of entire training data into a few synthetic training images. We demonstrate how to train a network to reach surprisingly good performance with only a small number of distilled images. Finally, the distilled images can efficiently store the memory of previous tasks in the continual learning setting. Many challenges remain for knowledge distillation of data. Although our method generalizes well to random initializations, it is still limited to a particular network architecture. Since loss surfaces for different architectures might be drastically different, a more flexible method of applying the distilled data may overcome this difficulty. Another limitation is the increasing computation and memory requirements for finding the distilled data as the number of images and steps increases. To compress large-scale datasets such as ImageNet, we may need first-order gradient approximations to make the optimization computationally feasible. Nonetheless, we are encouraged by the findings in this paper on the possibilities of training large models with a few distilled data, leading to potential applications such as accelerating network evaluation in neural architecture search (Zoph and Le, 2017) . We believe that the ideas developed in this work might give new insights into the quantity and type of data that deep networks are able to process, and hopefully inspire others to think along this direction.
We propose to distill a large dataset into a small set of synthetic data that can train networks close to original performance. 
{"src": "scitldr-AIC_train", "paper_id": "ryxO3gBtPB"}