Without loss of generality, we assume that the numerical values of activation gradient satisfy normal distributions, and a threshold \({\tau }\)  can be calculated based on this hypothesis. And then stochastic pruning is applied on the activation gradients with the threshold \({\tau }\)  while the gradients are set to zero or \({\pm \tau }\)  randomly.
Since the ReLU layers usually make the gradients distributed irregularly, we divide common networks into two categories, one is networks using Conv-ReLU as basic blocks such as AlexNet [1]} and VGGNet [2]}, another is those using Conv-BN-ReLU structure such as ResNet [3]}.
Experiments show that our pruning method works for both Conv-ReLU structure and Conv-BN-ReLU structure in modern networks.
A mathematical analysis is provided to demonstrate that stochastic pruning can maintain the convergence properties of CNN training.
Additionally, our proposed training scheme is evaluated both on Intel CPU and ARM CPU platforms, which could achieve \(1.71 \times \sim 3.99 \times \)  and \(1.79 \times \sim 5.92 \times \)  speedups, respectively, when compared with no pruning utilized at back-propagation stage.
