For completeness, we finally describe the quantization of the activation function when ASkewSGD is used to train a deep NN. During the forward pass, we employ a round-to-nearest approach INT4 quantization methods for the activations, taken from [1]}. We make use of Statistics Aware Weight Binning (SAWB) of [2]}, which finds the optimal scaling factor that minimizes the quantization
error based on the statistical characteristics of activation distribution. As emphasized by [1]}, [2]}, non-linearities of loss and activation functions make unnecessary the use of an unbiased scalar quantizer. After scaling, we use a uniform quantization (e.g., INT4): the set of quantization values \(\mathcal {Q}\)  is defined coordinate wise: \(\lbrace c^i_1,\dots ,c^i_{K^i}\rbrace \) . In our experiments (see sec:experiments) both the weights and the activations are rescaled layerwise to fit the quantization interval (e.g., \([-2^3, 2^3]\)  for INT4). The quantization values \(\lbrace c^i_1,\dots ,c^i_{K^i}\rbrace \)  are the integer from the quantization interval (e.g., \(\lbrace -8, -7,\dots ,8\rbrace \)  for INT4). After quantization, both the weights and the activation are rescaled using the scaling factor calculated layerwise.
