Firstly, we analyze the distribution of activation gradients for two typical structures of modern CNN models, as shown in Fig. REF .
For Conv-ReLU structure, where a Conv layer is followed by a ReLU layer, output activation gradients \(\mathrm {d}\mathbf {O}\)  are sparse, but subject to an irregular distribution.
On the other hand, the input activation gradients \(\mathrm {d}\mathbf {I}\) , which will be propagated to the previous layer, is almost full of non-zero values.
Statistics show that the probability distribution of \(\mathrm {d}\mathbf {I}\)  is symmetrical around zero and its probability density function decreases with the increment of absolute value \(\left| \mathrm {d}\mathbf {I} {\left( { \cdot } \right)} \right|\) .
For Conv-BN-ReLU structure, a BN layer is located between Conv and ReLU layer, and \(\mathrm {d}\mathbf {O}\)  subjects to the similar distribution of \(\mathrm {d}\mathbf {I}\) .
With the same hypothesis [1]}, these gradients are assumed to subject to a normal distribution with mean value 0 and variance \({{\sigma ^2}}\) .
