Recent efforts on training light-weight binary neural networks offer promising execution/memory efficiency. This paper introduces ResBinNet, which is a composition of two interlinked methodologies aiming to address the slow convergence speed and limited accuracy of binary convolutional neural networks. The first method, called residual binarization, learns a multi-level binary representation for the features within a certain neural network layer. The second method, called temperature adjustment, gradually binarizes the weights of a particular layer. The two methods jointly learn a set of soft-binarized parameters that improve the convergence rate and accuracy of binary neural networks. We corroborate the applicability and scalability of ResBinNet by implementing a prototype hardware accelerator. The accelerator is reconfigurable in terms of the numerical precision of the binarized features, offering a trade-off between runtime and inference accuracy.
 Convolutional Neural Networks (CNNs) have shown promising inference accuracy for learning applications in various domains. These models are generally over-parameterized to facilitate the convergence during the training phase BID7 ; BID4 ). A line of optimization methodologies such as tensor decomposition BID9 ; BID16 ), parameter quantization ; BID5 ), sparse convolutions BID10 ; ), and binary networks ; BID11 ) have been proposed to reduce the complexity of neural networks for efficient execution. Among these works, binary neural networks result in two particular benefits: (i) They reduce the memory footprint by a factor of 32 compared to the full-precision model; this is specifically important since memory access plays an essential role in the execution of CNNs on resource-constrained devices. (ii) Binary networks replace the costly multiplications with simple XNOR operations BID11 ; BID12 ), reducing the execution time and energy consumption significantly.Considering the prior art, there exist two major challenges associated with binary neural networks. First, the convergence rate of the existing solutions for training binary CNNs is considerably slower than their full-precision counterparts. Second, in order to achieve comparable classification accuracy, binarized neural networks often compensate for the numerical precision loss by employing high dimensional feature maps in a wide CNN topology, which in turn reduces the effective compression rate. As a result, full-precision networks often surpass binary networks in terms of convergence rate and final achievable accuracy.In this paper, we propose ResBinNet, a novel solution for increasing the convergence rate and the final accuracy of binary networks. The global flow of ResBinNet is depicted in FIG0 . The first phase, which we call Soft Binarization, includes two methodologies that we propose to address the aforementioned challenges for training binary CNNs. First, we introduce a Residual Binarization scheme which allows the number of possible values for activation units to be reconfigurable at runtime. To this purpose, we learn a multi-level residual representation for the features within the CNN to adaptively increase the numerical precision of the activation units. Second, we introduce a novel weight binarization approach, called Tempreture Adjustment, which aims to gradually enforce binarization constraints over the weight parameters throughout the training phase. The two interlinked methods significantly improve both the convergence rate and the final accuracy of ResBinNet compared to prior art. Once the soft training phase is finished, we convert the weights to actual binary values (0,1). Fine-tuning of the model is then performed in Hard Binarization phase using existing training algorithms (e.g. BinaryNets )) in few epochs (e.g. one epoch). ResBinNet is designed to fulfill certain goals: (i) It should enable reconfigurability for binary neural networks; in other words, the number of residual binary representatives should be adjustable to offer a trade-off between inference accuracy and computation time.(ii ) The multi-level binarized features should be compatible with the XNOR multiplication approach proposed in the existing literature.(iii ) ResBinNet should speed up the convergence rate of binarized CNNs. (iv ) Current hardware accelerators for binary CNNs should be able to benefit from ResBinNet with minimum modification in their design. In summary, the contributions of this paper are as follows:• Proposing residual binarization, a methodology for learning multi-level residual representations for each feature map in binary CNNs.• Introducing temperature adjustment as a practical approach for gradual (soft) binarization of CNN weights.• Analyzing the trade-off between accuracy and execution time of ResBinNet on a real hardware design.• Evaluating ResBinNet convergence rate and accuracy on three datasets: MNIST, SVHN, and CIFAR-10.• Development of an open-source Application Program Interface (API) for ResBinNet 1 .The remainder of the paper is organized as follows: In Section 2, we describe the residual binarization method for binarizing activations. Section 3 explains the temperature adjustment technique for binarizing weights. In Section 4, we discuss how particular ResBinNet operations (e.g. multi-level XNOR-popcount) can be efficiently implemented on existing hardware accelerators. Experiments are discussed in Section 5. Finally, we discuss the related work and conclusion in Sections 6 and 7. This paper introduces ResBinNet, a novel reconfigurable binarization scheme which aims to improve the convergence rate and the final accuracy of binary CNNs. The proposed training is twofold: (i) In the first phase, called soft binarization, we introduce two distinct methodologies designed for binarizing weights and feature within CNNs, namely residual binarization, and temperature adjustment. Residual binarization learns a multi-level representation for features of CNN to provide an arbitrary numerical precision during inference. Temperature adjustment gradually imposes binarization constraints on the weights. (ii) In the second phase, which we call hard binarization, the model is fine-tuned in few training epochs. Our experiments demonstrate that the joint use of residual binarization and temperature adjustment improves the convergence rate and the accuracy of the binarized CNN. We argue that ResBinNet methodology can be adopted by current CNN hardware accelerators as it requires minimal modification to existing binarized CNN solutions. Developers can integrate the approaches proposed in this paper into their deep learning systems to provide users with a trade-off between application latency and inference accuracy.
Residual Binary Neural Networks significantly improve the convergence rate and inference accuracy of the binary neural networks.
{"src": "scitldr-AIC_train", "paper_id": "SJtfOEn6-"}