We consider now low-bit weight quantization and activation quantization. To fully benefit from low precision arithmetic, one should also tackle the problem of gradient quantization [1]}, [2]} and accumulation. We keep the last fully connected layer in full-precision, following [3]}, [1]}. We evaluate the performance of ASkewSGD [W1/A32], ASkewSGD [W2/A4], and ASkewSGD [W4/A4] on TinyImageNet and ImageNet [5]} datasets with a ResNet-18 network. For ASkewSGD  we project NN weights onto the set of quantization values before evaluating it on the test set. For ImageNet, we keep the first convolution layer in full-precision. We use the same pre-processing (centering and data normalization) for all the methods: we resize the input images to \(256 \times 256\)  and then randomly crop them to \(224 \times 224\)  while centering them to the appropriate sizes during training. Standard multiclass cross entropy loss is used. All models are fine-tuned for 200 epochs using the Adam [6]} optimizer with dynamics of \(0.9\)  and \(0.999\)  and a batch of size 512. All methods are trained with an initial learning rate of \(0.06\)  for TinyImagenet and \(0.1\)  for ImageNet. The same hyperparameters are used as in the previous section for TinyImageNet. For ImageNet, the learning rate at epochs \([30, 60, 90]\)  is divided by 10 for all methods.
We have run the codehttps://openreview.net/forum?id=clwYez4n8e8 from LUQ and adapted it to TinyImageNet dataset. For a fair comparison we compute neural gradients in full precision. The results for the method Ultra-low [2]} are taken from [1]}.
