Quantization.
Quantizing neural networks (QNNs) often possess low-bit (1\(\sim \)  4-bit) weights and activations to accelerate the model inference and save the memory usage. Specifically, ternary weights are introduced to reduce the quantization error in TWN [1]}. DoReFa-Net [2]} exploits convolution kernels with low bit-width parameters and gradients to accelerate both the training and inference.
TTQ [3]} uses two full-precision scaling coefficients to quantize the weights to ternary values.
 [4]} presented a \(2\!\sim \!4\) -bit quantization scheme using a two-stage approach to alternately quantize the weights and activations, which provides an optimal trade-off among memory, efficiency, and performance. [5]} parameterizes the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network and also the accuracy degeneration with further bit-width reduction.
 [6]} introduces transfer learning into network quantization to obtain an accurate low-precision model by utilizing the Kullback-Leibler (KL) divergence.
 [7]} enables accurate approximation for tensor values that have bell-shaped distributions with long tails and finds the entire range by minimizing the quantization error.
In our Q-ViT, we aim to implement an accurate, fully quantized vision transformer under the QAT paradigm.
<FIGURE>