Pruning a few gradients with small values has little impact on weights update. However, once all of these small gradients are set to 0, the distribution of activation gradients will be affected significantly, which will influence the weights update and cause severe accuracy loss.
Inspired by Stochastic Rounding in [1]}, we adopt stochastic pruning to solve this problem.
