Activation function Quantization: We have so far described the quantization of the network weights. But an efficient implementation also requires the quantization of the activation functions. For BNN, [1]}, [2]}, [3]} proposed to use \(\operatorname{sign(\cdot )}\)  function, but this approach significantly affects the performance.
More complex quantization schemes have been considered in [4]} alleviating performance degradation. Hybrid formats FP8 [5]} or INT8 [6]}, [7]} were successfully employed to achieve a low precision training. Recent works have proposed to jointly optimize the quantization parameters (of weights and activations) and the weights parameters. This task can be done by modifying the learning loss or by minimizing the quantization error [8]}, [9]}, [10]}.
