@inproceedings{
zhou2024derd,
title={{DERD}: Data-free Adversarial Robustness Distillation through Self-adversarial Teacher Group},
author={Yuhang Zhou and Yushu Zhang and Leo Yu Zhang and Zhongyun Hua},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=CJKVSlCoRA}
}
Computer vision models based on deep neural networks are proven to be vulnerable to adversarial attacks. Robustness distillation, as a countermeasure, takes both robustness challenges and efficiency challenges of edge models into consideration. However, most existing robustness distillations are data-driven, which can hardly be deployed in the data-privacy scenarios. Also, the trade-off between robustness and accuracy tends to transfer from the teacher to the student along with the robustness, and there has been no discussion on mitigating this trade-off in the data-free scenario yet. 
In this paper, we propose a Data-free Experts-guided Robustness Distillation (DERD) to extend robustness distillation to data-free paradigm, which offers three advantages: 
(1) Dual-level adversarial learning strategy achieves robustness distillation without the real data.
(2) Expert-guided distillation strategy brings a better trade-off to the student model. 
(3) A novel stochastic gradient aggregation module reconciles the task conflicts of the multi-teacher from a consistency perspective. Extensive experiments demonstrate that the proposed DERD can even achieve comparable results to data-driven methods.
DERD: Data-free Adversarial Robustness Distillation through Self-adversarial Teacher Group
Yuhang Zhou Yushu Zhang Leo Yu Zhang Zhongyun Hua