To be deployed on resources-constrained hardware for real-time applications, the efficiency of deep convolutional neural networks has been improved significantly by various model compression techniques [1]}, [2]}, [3]}, [4]}. Without altering the model architecture, quantized neural networks [3]} use a low bit width representation instead of full-precision floating-point, saving expensive multiplications. Pruning [1]} is an approach to remove the weights or neurons based on certain criteria. In terms of efficient neural network architectures, the MobileNet [2]}, ShuffleNet, and ESPNet [8]} series make use of depthwise-separable convolution, grouped convolution with shuffle operation, and efficient spatial pyramid. The knowledge distillation paradigm [4]} transfers the information from a pre-trained teacher network to a portable student network.
