Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
We propose an algorithm that compresses the critical information of a large
dataset into compact addressable memories. These memories can then be recalled
to quickly re-train a neural network and recover the performance (instead of
storing and re-training on the full original dataset).
  Building upon the dataset distillation framework, we make a key observation
that a shared common representation allows for more efficient and effective
distillation. Concretely, we learn a set of bases (aka "memories") which are
shared between classes and combined through learned flexible addressing
functions to generate a diverse set of training examples. This leads to several
benefits: 1) the size of compressed data does not necessarily grow linearly
with the number of classes; 2) an overall higher compression rate with more
effective distillation is achieved; and 3) more generalized queries are allowed
beyond recalling the original classes.
  We demonstrate state-of-the-art results on the dataset distillation task
across five benchmarks, including up to 16.5% and 9.7% in retained accuracy
improvement when distilling CIFAR10 and CIFAR100 respectively. We then leverage
our framework to perform continual learning, achieving state-of-the-art results
on four benchmarks, with 23.2% accuracy improvement on MANY.