Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search
Knowledge Distillation (KD) has recently emerged as a popular method for
compressing neural networks. In recent studies, generalized distillation
methods that find parameters and architectures of student models at the same
time have been proposed. Still, this search method requires a lot of
computation to search for architectures and has the disadvantage of considering
only convolutional blocks in their search space. This paper introduces a new
algorithm, coined as Trust Region Aware architecture search to Distill
knowledge Effectively (TRADE), that rapidly finds effective student
architectures from several state-of-the-art architectures using trust region
Bayesian optimization approach. Experimental results show our proposed TRADE
algorithm consistently outperforms both the conventional NAS approach and
pre-defined architecture under KD training.