Training with larger number of parameters while keeping fast iterations is an increasingly
 adopted strategy and trend for developing better performing Deep Neural
 Network (DNN) models. This necessitates increased memory footprint and
 computational requirements for training. Here we introduce a novel methodology
 for training deep neural networks using 8-bit floating point (FP8) numbers.
 Reduced bit precision allows for a larger effective memory and increased computational
 speed. We name this method Shifted and Squeezed FP8 (S2FP8). We
 show that, unlike previous 8-bit precision training methods, the proposed method
 works out of the box for representative models: ResNet50, Transformer and NCF.
 The method can maintain model accuracy without requiring fine-tuning loss scaling
 parameters or keeping certain layers in single precision. We introduce two
 learnable statistics of the DNN tensors - shifted and squeezed factors that are used
 to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in
 information due to quantization. Deep neural networks have achieved state-of-the-art performance on a wide variety of computer vision, audio, and natural language processing (NLP) tasks. This has resulted in an explosion of interest around techniques to reduce the memory footprint and energy consumption of neural network training and inference (Guo, 2018) . Although there are a number of methods to address some of these issues for inference, the most effective method for training is using reduced precision numerical formats. While 32-bit floating point (FP32) is the most common data format for neural network training, recent hardware have leveraged techniques that allow for training with 16-bit data formats (Köster et al., 2017; Micikevicius et al., 2018) . However, 8-bit precision training remains an open challenge (Johnson, 2018; Kalamkar et al., 2019) . Current FP8 training methodologies (Wang et al., 2018; require either specialized chunk-based accumulation, stochastic rounding techniques, loss scaling or maintaining some layers of the network in higher precision. Tuning these knobs is non-intuitive and requires significant experimentation for each individual network. Accelerating the adoption of 8-bit data in training DNNs requires a hardware-friendly and out-ofthe-box implementation of FP8. Due to the reduced number of mantissa bits, 8-bit multipliers are smaller and consume less power compared to higher bit representations. In this work we describe a novel 8-bit floating point (FP8) format -shifted and squeezed FP8 (S2FP8) -which has the following advantages compared to previously proposed 8-bit training methodologies: • S2FP8 eliminates the need for loss scaling, which requires significant tuning of the loss scale values and schedule for individual topologies • Leveraged by the forward and backward passes of model training, S2FP8 is effective in adjusting the range of gradients and also of activations and weights • S2FP8 does not require keeping the first and last layer in FP32 precision, which is needed for other approaches , however maintains the master weights and accumulations inside the matrix multipliers in FP32 We demonstrate across image classification, translation, and recommendation models that S2FP8 outperforms previous 8-bit approaches, and reaches the accuracy of FP32 models without any additional hyperparameter tuning. We introduce a novel 8-bit floating point data type (S2FP8), that gives competitive performance in comparison to state-of-the-art FP32 baselines over a range of representative networks. S2FP8 makes use of shifted and squeezed factors to shift and rescale the range of tensors prior to truncation. S2FP8 allows training of neural networks with an 8-bit format while eliminating the need for loss scaling tuning, hardware-complex rounding techniques. In addition, compared to existing FP8 implementations we also eliminate the restriction of maintaining the first and last layers in FP32. Decreasing Movielens 1 million FP32 S2FP8 ∆ FP8 NCF 0.666 0.663 0.003 0.633 Figure A1 : The range and precision of FP8. Bar indicate the number density between each power of 2. Since FP8 has 2 mantissa bit, the density is 4 (except in the denormals), and the associated machine epsilon is 2 −3 = 1/8. The normal representable range goes from 2 −14 to (1 − 2 −3 )2 16 , with denormals from 2 −16 to 2 −14 .
We propose a novel 8-bit format that eliminates the need for loss scaling, stochastic rounding, and other low precision techniques
{"src": "scitldr-AIC_train", "paper_id": "Bkxe2AVtPS"}