For on-chip training and inference, the most recent studies show that training needs at least 8 bits of precision to ensure accuracy[1]}, [2]}, [3]} although precision can be extremely low at the inference phase. Most of the related work focuses on training models from scratch and building a software framework [4]} for quantized training to reduce computational resources. Yang et al.[2]} observes the requirement of a large bit width to realize model convergence, and thus introduce a layer-wise scaling factor and an extra flag bit to solve the problem. Instead, this paper uses a more straightforward scaling method, as mentioned later. Zhu et al.[3]} suggests clipping the gradient value for integer training, but the clipping value is based on the cosine distance between the floating point gradient and its quantize-dequantized counterpart, which is impractical in the edge device.
Banner et al.[1]} proposes the Range BN for a higher tolerance to quantization noise, but collecting BN statistics has a high overhead on hardware. Furthermore, since a small training data set has too much variation, changing the BN statistics may make training unstable. Therefore, this paper decides to freeze the BN statistics during training.
