8-bit Numerical Formats for Deep Neural Networks
Given the current trend of increasing size and complexity of machine learning
architectures, it has become of critical importance to identify new approaches
to improve the computational efficiency of model training. In this context, we
address the advantages of floating-point over fixed-point representation, and
present an in-depth study on the use of 8-bit floating-point number formats for
activations, weights, and gradients for both training and inference. We explore
the effect of different bit-widths for exponents and significands and different
exponent biases. The experimental results demonstrate that a suitable choice of
these low-precision formats enables faster training and reduced power
consumption without any degradation in accuracy for a range of deep learning
models for image classification and language processing.