TITLE: Dataset Distillation with Infinitely Wide Convolutional Networks 

ABSTRACT: The effectiveness of machine learning algorithms arises from being able to
extract useful features from large amounts of data. As model and dataset sizes
increase, dataset distillation methods that compress large datasets into
significantly smaller yet highly performant ones will become valuable in terms
of training efficiency and useful feature extraction. To that end, we apply a
novel distributed kernel based meta-learning framework to achieve
state-of-the-art results for dataset distillation using infinitely wide
convolutional neural networks. For instance, using only 10 datapoints (0.02% of
original dataset), we obtain over 65% test accuracy on CIFAR-10 image
classification task, a dramatic improvement over the previous best test
accuracy of 40%. Our state-of-the-art results extend across many other settings
for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we
perform some preliminary analyses of our distilled datasets to shed light on
how they differ from naturally occurring data.
The effectiveness of machine learning algorithms arises from being able to
extract useful features from large amounts of data. As model and dataset sizes
increase, dataset distillation methods that compress large datasets into
significantly smaller yet highly performant ones will become valuable in terms
of training efficiency and useful feature extraction. To that end, we apply a
novel distributed kernel based meta-learning framework to achieve
state-of-the-art results for dataset distillation using infinitely wide
convolutional neural networks. For instance, using only 10 datapoints (0.02% of
original dataset), we obtain over 65% test accuracy on CIFAR-10 image
classification task, a dramatic improvement over the previous best test
accuracy of 40%. Our state-of-the-art results extend across many other settings
for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we
perform some preliminary analyses of our distilled datasets to shed light on
how they differ from naturally occurring data.
Dataset Distillation with Infinitely Wide Convolutional Networks
http://arxiv.org/abs/2107.13034v3