Deep neural networks (DNN) are widely used in many applications. However, their deployment on edge devices has been difficult because they are resource hungry. Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit. We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values. In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions. Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation. These additions are based on linear operations that are easily implementable into the binary training framework. We show experimental results on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network. On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively. Deep neural networks (DNNs) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection BID26 BID11 . The focus has been on increasing accuracy, in particular for image tasks, where deep convolutional neural networks (CNNs) are widely used. However, their increasing complexity poses a new challenge, and has become an impediment to widespread deployment in many applications; specifically when trying to deploy such models to resource constrained and lower-power devices. A typical DNN architecture contains tens to thousands of layers, resulting in millions of parameters. As an example, AlexNet BID16 requires 200MB of memory, VGGNet BID26 requires 500MB memory. Large model sizes are further exacerbated by their computational cost, requiring GPU implementation to allow real-time inference. Such requirements evidently cannot be accustomed by edge devices as they have limited memory, computation power, and battery. This motivated the community to investigate methods for compressing and reducing computation cost of DNNs.To make DNNs compatible with the resource constraints of low power devices, there have been several approaches developed, such as network pruning BID17 , architecture design BID25 , and quantization BID0 BID4 . In particular, weight compression using quantization can achieve very large savings in memory, where binary (1-bit), and ternary (2-bit) approaches have been shown to obtain competitive accuracy BID10 BID31 BID29 . Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations. In addition to this, the speed by quantizing the activation layers. In this way, both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations. This reduction in bit-width benefits hardware accelerators such as FPGAs and neural network chips.An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart, and this is made even more severe upon quantizing the activations. This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation BID19 . Although, quantizing the weights and activations have been attracting large interests thanks to their computational benefits, closing the gap in accuracy between the full precision and the quantized version remains a challenge. Indeed, quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign fluctuations in the weights. Therefore, how to control the stability of this training procedure is of high importance. In theory, it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete. Instead, heuristics and approximations are proposed to match the forward and backward passes. Often weights at different layers of DNNs follow a certain structure. How to quantize the weights locally, and maintaining a global structure to minimize a common cost function is important BID18 .Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by BID10 to improve convergence and generalization accuracy of binary networks. First , the activation function is modified to better approximate the sign function in the backward pass, second we propose two regularization functions that encourage training weights around binary values, and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization. Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods. We show accuracy gains to traditional binary training. We proposed two regularization terms (4) and (5) and an activation term (2) with a trainable parameter β. We run several experiments to better understand the effect of the different modifications to the training method, especially using different regularization and asymptote parameters β. The parameter β is trainable and would add one equation through back-propagation. However, we fixed β throughout our experiments to explicit values. The results are summarized in TAB1 .Through our experiments, we found that adding regularizing term with heavy penalization degrades the networks ability to converge, as the term would result in total loss be largely due to the regu- larizing term and not the target cross entropy loss. Similarly , the regularizing term was set to small values in BID29 . As a result , we set λ with a reasonably small value 10 −5 − 10 −7 , so that the scales move slowly as the weights gradually converge to stable values. Some preliminary experimentation was to gradually increase the regularization with respect to batch iterations updates done in training, though this approach requires careful tuning and was not pursued further.From TAB1 , and referring to networks without regularization, we see the benefit of using SwishSign approximation versus the STE. This was also noted in , where their second approximation provided better results. There is not much difference between using R 1 versus R 2 towards model generalization although since the loss metric used was the cross-entropy loss, the order of R 1 better matches the loss metric. Lastly, it seems moderate values of β is better than small or large values. Intuitively, this happens because for small values of β, the gradient approximation is not good enough and as β increases the gradients become too large, hence small noise could cause large fluctuations in the sign of the weights.We did not compare our network with that of as they introduce a shortcut connection that proves to help even the full precision network. As a final remark, we note that the learning rate is of great importance and properly tuning this is required to achieve convergence. Table 3 summarizes the best results of the ablation study and compares with BinaryNet, XNOR-Net, and ABC-Net. Table 3 : Comparison of top-1 and top-5 accuracies of our method BNN+ with BinaryNet, XNORNet and ABC-Net on ImageNet, summarized from TAB1 . The results of BNN, XNOR, & ABC-Net are reported from the corresponding papers BID23 BID10 BID29 . Results for ABC-NET on AlexNet were not available, and so is not reported. To summarize we propose three incremental ideas that help binary training: i) adding a regularizer to the objective function of the network, ii) trainable scale factors that are embedded in the regularizing term and iii) an improved approximation to the derivative of the sign activation function. We obtain competitive results by training AlexNet and Resnet-18 on the ImageNet dataset. For future work, we plan on extending these to efficient models such as CondenseNet BID9 , MobileNets BID8 , MnasNet BID28 and on object recognition tasks.
The paper presents an improved training mechanism for obtaining binary networks with smaller accuracy drop that helps close the gap with it's full precision counterpart
{"src": "scitldr-AIC_train", "paper_id": "SJfHg2A5tQ"}