Deep neural networks (DNNs) dominate current research in machine learning. Due to massive GPU parallelization DNN training is no longer a bottleneck, and large models with many parameters and high computational effort lead common benchmark tables. In contrast, embedded devices have a very limited capability. As a result, both model size and inference time must be significantly reduced if DNNs are to achieve suitable performance on embedded devices.
 We propose a soft quantization approach to train DNNs that can be evaluated using pure fixed-point arithmetic. By exploiting the bit-shift mechanism, we derive fixed-point quantization constraints for all important components, including batch normalization and ReLU. Compared to floating-point arithmetic, fixed-point calculations significantly reduce computational effort whereas low-bit representations immediately decrease memory costs. We evaluate our approach with different architectures on common benchmark data sets and compare with recent quantization approaches. We achieve new state of the art performance using 4-bit fixed-point models with an error rate of 4.98% on CIFAR-10. Deep neural networks (DNNs) are state of the art in many machine learning challenges, pushing recent progress in computer vision, speech recognition and object detection (Deng & Yu (2014) ; Lecun et al. (2015) ; Karki et al. (2019) ). However, the greatest results have been accomplished by training large models with many parameters using large amounts of training data. As a result, modern DNNs show an extensive memory footprint and high-precision floating-point multiplications are especially expensive in terms of computation time and power consumption. When deployed on embedded devices, the complexity of DNNs is necessarily restricted by the computational capability. Therefore, efforts have been made to modify DNNs to better suit specific hardware instructions. This includes both the transfer from floating-point to fixed-point arithmetic and the reduction in bit-size. This process is termed fixed-point quantization and especially low-bit representations simultanouesly reduce memory cost, inference time, and energy consumption. A survey is given in Sze et al. (2017) . Furthermore, ternary-valued weights or even binary-valued weights allow replacement of many multiplications with additions 1 . However, most quantization approaches do not fit to the common structure in modern DNNs. State of the art architectures (such as ResNet, DenseNet, or MobileNetV2) consist of interconnected blocks that combine a convolution or fully-connected layer, a batch normalization layer and a ReLU activation function. Each block can be optionally extended by a pooling layer, as shown in Figure  1 . Since both convolution and fully-connected layers perform weighted sums, we summarize the two as a Linear component. In contrast to the block structure, recent quantization approaches focus on the Linear component while preserving floating-point batch normalization (BN) layers. This is crucial, since BN layers are folded into the preceding layer after the training and consequently destroy its fixed-point representation. Even when performed separately, channel-wise floating-point multiplications make a pure fixed-point representation impossible. Furthermore, many quantization methods strictly binarize activations which only works for very large models. In this paper, we propose a soft quantization approach to learn pure fixed-point representations of state of the art DNN architectures. Thereby, we follow the block structure and transfer all individual components into fixed-point representations before combining them appropriately. We follow the same approach as Enderich et al. (2019) and formulate bit-size dependent fixed-point constraints for each component before transferring these constraints into regularization terms. To the best of our knowledge, we are the first to provide a soft quantization approach to learn pure fixed-point representations of DNNs. We extensively validate our approach on several benchmark data sets and with state of the art DNN architectures. Although our approach is completely flexible in bit-size, we test two special cases: • A pure fixed-point model with 4-bit weights and 4-bit activations which performs explicitly well, outperforming the floating-point baseline in many cases. • A model with ternary-valued weights and 4-bit activations that can be evaluated using additions, bit shifts and clipping operations alone (no multiplications needed). Soft quantization aims to reduce the complexity of DNNs at test time rather than at training time. Therefore, training remains in floating-point precision, but maintains consideration of dedicated quantization constraints. In this paper, we propose a novel soft quantization approach to learn pure fixed-point representations of state of the art DNN architectures. With exponentially increasing fixed-point priors and weight clipping, our approach provides self-reliant weight adaptation. In detailed experiments, we achieve new state of the art quantization results. Especially the combination of 4-bit weights, 4-bit activations and fixed-point batch normalization layers seems quite promising
Soft quantization approach to learn pure fixed-point representations of deep neural networks
{"src": "scitldr-AIC_train", "paper_id": "rJgKzlSKPH"}