Because SALO does not introduce any modification to attention mechanisms, there will not be an approximation error when using SALO to accelerate the inference. But SALO uses a low precision fixed point arithmetic to achieve higher performance that quantizes the Query, Key, Value matrix to 8 bits (4 bits for fraction part). As a result, the output of SALO is in 16 bits, which may potentially influence the accuracy. Thus we conduct experiments on Longformer and ViL to measure the impact of quantization. We slightly modified the self attention layers in Longformer and ViL by inserting quantization layers after the operators, to simulate the precision on SALO. These quantization layers are implemented by QPyTorch[1]}, a low-precision arithmetic simulation package. We perform quantization-aware finetuning on both pretrained models.
<TABLE>