is another common way to reduce the computational complexity and memory consumption of training.
Gupta’s work [1]} maintains the accuracy by training the model in the precision of 16-bit fixed-point number with stochastic rounding.
DoReFaNet [2]} derived from AlexNet [3]} utilizes 1-bit, 2-bit and 6-bit fixed-point number to represent weights, activations and gradients respectively, but brings visible accuracy drop.
Park [4]} proposed a value-aware quantization method by using low-precision on small values, which can significantly reduce memory consumption when training ResNet-152 [5]} and Inception-V3 [6]} with \(98\%\)  activations quantified to 3-bit.
Micikevicius [7]} keeps an FP32 copy for weight update and adopts FP16 for computation, which is efficient for training acceleration. Our approach can be regarded as gradients sparsification, and can be also integrated with gradients quantization methods.
