For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware,  each learned weight parameter should ideally be represented and stored using a single bit.   Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply  scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1%  respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision weights. For networks that overfit, we also show significant improvements in error rate by not learning batch normalization scale and offset parameters. This applies to both full precision and 1-bit-per-weight networks. Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR-10/100. For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/ . Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks. However, GPUs are both expensive and wasteful in terms of their energy requirements. They typically compute using single-precision floating point (32 bits), which has now been recognized as providing far more precision than needed for deep neural networks. Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM. If deep-learning methods are to become embedded in resourceconstrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.To this end, there has been increasing interest in finding methods that drive down the resource burden of modern deep neural networks. Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on important benchmarks.In this paper, we report a significant reduction in the gap (see Figure 1 and Results) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit floating point) and networks deployed using weights represented by a single-bit each.In the process of developing our methods, we also obtained significant improvements in error-rates obtained by full-precision versions of the CNNs we used.In addition to having application in custom hardware deploying deep networks, networks deployed using 1-bit-per-weight have previously been shown BID21 to enable significant speedups on regular GPUs, although doing so is not yet possible using standard popular libraries.Aspects of this work was first communicated as a subset of the material in a workshop abstract and talk BID19 . Figure 1: Our error-rate gaps between using full-precision and 1-bit-per-weight. All points except black crosses are data from some of our best results reported in this paper for each dataset. Black points are results on the full ImageNet dataset, in comparison with results of BID22 (black crosses). The notation 4x, 10x and 15x corresponds to network width (see Section 4).1.1 RELATED WORK
We train wide residual networks that can be immediately deployed using only a single bit for each convolutional weight, with signficantly better accuracy than past methods.
{"src": "scitldr-AIC_train", "paper_id": "rytNfI1AZ"}