The essential optimization step of CNN training is to perform Stochastic Gradient Descent (SGD) algorithm in back-propagation procedure. There are several data types involved in training dataflow: weights, weight gradients, activations, activation gradients. Back-propagation starts from computing the weight gradients with the activations and then performs weights update [1]}. Among these steps, activation gradients back-propagation and weight gradients computation require intensive convolution operations thus dominate the total training cost. It is well known that computation cost can be reduced by skipping over zero-values. Since these two convolution steps require the activation gradients as input, improving the sparsity of activation gradients should significantly reduce the computation cost and memory footprint during back-propagation procedure.
