The heavy computation cost of deep neural network limits its usefulness in online serving or mobile applications. Many studies [1]} have shown that, network weights might be redundant and do not convey much information. In some cases [2]}, some large-scale models tend to memorize the dataset instead of learning some generic capability, suffering from serious over-fitting. To boost the efficiency and suppress over-fitting, recently, many efficient and compact neural network architecture are developed, achieving very promising performance in both effectiveness and efficiency. One of most widely used strategy is training quantized neural networks [3]}, [4]}, [5]}, [6]}, [7]}, [8]}. For instance, [3]}, [4]} design the binary neural network where the value of weights are chosen from two constants. In parallel, some efficient neural networks [1]}, [12]}, [13]} are obtained by compressing already-trained neural networks. Inspired by the recent success achieved by knowledge distillation [14]}, some methods [15]}, [16]} propose to find a more compact student network through knowledge distillation. To be specific, [15]} jointly learn the weight quantization and the model distillation. [16]} designs a Tiny-BERT by distilling the knowledge from the large-scale BERT model. Following the spirit of Tiny-BERT, in our work, we also exploit the knowledge distillation for model compression.
