Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we propose Dream Distillation, a data-independent model compression framework. Our experiments show that Dream Distillation can achieve 88.5 accuracy on the CIFAR-10 test set without actually training on the original data!
Despite its significance, the literature on model compression in absence of real data is very sparse. A relevant prior work is @cite_4 where the authors propose a Data-Free KD (DFKD) framework. However, there are major differences between DFKD and the present work: DFKD requires significantly more metadata than our approach. Specifically, @cite_4 argue that using metadata from only the final layer under-constrains the image generation problem, and results in very poor student accuracy. Consequently, DFKD assumes access to metadata at layers. In contrast, Dream Distillation assumes that metadata is available only at one layer of the teacher network. Hence, in this paper, we precisely demonstrate that metadata from a layer is sufficient to achieve high student accuracy, something that DFKD failed to accomplish. When using metadata from only one layer, DFKD achieves only @math -$77 DFKD also proposes a spectral method-based metadata for synthetic image generation. However, both spectral methods and all-layer metadata can be computationally very expensive and do not scale for larger networks. Compared to these, we follow a clustering-based approach which helps generate diverse images while using significantly less computation.
@cite_4
2766966408
Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.