Recent work has explored reduced numerical precision for parameters, activations, and gradients during neural network training as a way to reduce the computational cost of training (Na & Mukhopadhyay, 2016) (, 2014). We present a novel dynamic precision scaling (DPS) scheme. Using stochastic fixed-point rounding, a quantization-error based scaling scheme, and dynamic bit-widths during training, we achieve 98.8 test accuracy on the MNIST dataset using an average bit-width of just 16 bits for weights and 14 bits for activations, compared to the standard 32-bit floating point values used in deep learning frameworks.
In order to relate our work to prior research in this area, here we compare our methodology to the work of Na and Mukhopadhyay, who published results for dynamic precision scaling during training, specifically targeting their custom flexible multiply-accumulate unit that can speed up reduced precision arithmetic @cite_1 . Further comparison can be made to the work of , who investigate an overflow-based dynamic fixed point precision scaling technique with fixed bit-width during neural network training @cite_7 . Finally, inspiration can be drawn from implementations by , who study the effect of limited precision data representation and computation on neural network training @cite_2 . By using the Stochastic rounding method, as compared to the Round-to-nearest method for fixed point rounding, they achieve far superior accuracy and convergence rate. Since our work builds the work of Na and Mukhopadhyay, , as well as , we will discuss the contributions of each, as well as the differences as compared to our implementation.
@cite_1 @cite_7 @cite_2
2475884125 2114508814 1841592590
Training convolutional neural network is a major bottleneck when developing a new neural network topology. This paper presents a dynamic precision scaling (DPS) algorithm and flexible multiplier-accumulator (MAC) to speed up convolutional neural network training. The DPS algorithm utilizes dynamic fixed point and finds good enough numerical precision for target network while training. The precision information from DPS is used to configure our proposed MAC. The proposed MAC can perform fixed point computation with variable precision mode providing differentiated computation time which enables speeding up training for lower precision computation. Simulation results show that our work can achieve 5.7x speed-up while consuming 31 energy compared to baseline for modified Alexnet on Flickr image style recognition task. Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.