Deep neural networks have achieved state-of-the-art performance on many applications such as computer vision, natural language processing, and speech recognition in recent years. The remarkable performance of deep learning relies on designing deeper or wider network architectures with many layers and millions of parameters to enhance the learning capacity. However, it is almost impossible to deploy the large-scale networks on platforms with limited computation and storage resources, e.g., mobile devices and embedded systems. Thus, the model compression and acceleration techniques mainly including network pruning [1]}, [2]}, model quantization [3]}, [4]} and knowledge distillation [5]}, [6]}, [7]}, [8]}, [9]} are proposed for training lightweight deep models. Among compressing methods, knowledge distillation, which carries out knowledge transfer from a high-capacity teacher network to a low-capacity student one, has received increasing interest recently since it was first introduced in [5]}.
