We propose a method of training quantization thresholds (TQT) for uniform symmetric quantizers using standard backpropagation and gradient descent. Contrary to prior work, we show that a careful analysis of the straight-through estimator for threshold gradients allows for a natural range-precision trade-off leading to better optima. Our quantizers are constrained to use power-of-2 scale-factors and per-tensor scaling of weights and activations to make it amenable for hardware implementations. We present analytical support for the general robustness of our methods and empirically validate them on various CNNs for ImageNet classification. We are able to achieve near-floating-point accuracy on traditionally difficult networks such as MobileNets with less than 5 epochs of quantized (8-bit) retraining. Finally, we present Graffitist, a framework that enables automatic quantization of TensorFlow graphs for TQT. Available at https: github.com Xilinx graffitist .
Independently of our work, IBM's LSQ @cite_31 found very similar gradient definition for the quantizer and uses backpropagation to train them. However, our works differ in several interesting ways. For example, they learn the scale-factors directly and do not restrict them to power-of-2. Besides the evident implications for accuracy and hardware implementation, we show in that this also has major implications for training stability due to scale dependence of learning rate. As a workaround to these stability issues, they require careful fine-tuning of hyperparameters and consequently retrain for 90 epochs compared to 5 epochs in our case. They also use high precision in the first and last layers to retain performance, as is common in the field. We suspect the high precision and lack of power-of-2 limitations allow for very high accuracy in their low bit-width experiments. Further, they do not explore quantization on more difficult networks such as MobileNets @cite_24 @cite_42 . We address these issues with a different gradient formulation in and justify it analytically in .
@cite_24 @cite_31 @cite_42
2612445135 2916954108
We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization. We present here Learned Step Size Quantization, a method for training deep networks such that they can run at inference time using low precision integer matrix multipliers, which offer power and space advantages over high precision alternatives. The essence of our approach is to learn the step size parameter of a uniform quantizer by backpropagation of the training loss, applying a scaling factor to its learning rate, and computing its associated loss gradient by ignoring the discontinuity present in the quantizer. This quantization approach can be applied to activations or weights, using different levels of precision as needed for a given system, and requiring only a simple modification of existing training code. As demonstrated on the ImageNet dataset, our approach achieves better accuracy than all previous published methods for creating quantized networks on several ResNet network architectures at 2-, 3- and 4-bits of precision.