Recent years have witnessed the great success of applying convolutional neural networks (CNN) in various tasks and the remarkable performance can be owing to the deeper and wider design of network structures [1]}, [2]}, [3]}.
Whereas, it is hard to deploy such heavy networks in practice where computational resource and memory space is limited.
Knowledge Distillation [4]} has been proposed to decrease the model size by training a small and shallow network (student) under the supervision of a larger and deeper network (teacher).
Benefit by knowledge distillation, knowledge in a heavy teacher network can be compressed into a lightweight student network and the performance of the student can be improved.
<FIGURE>