is another approach to reduce training cost but is rarely researched because activation gradients are generated dynamically during back-propagation. Most previous works adopt top-k as the base algorithm for sparsification.
For MLP training, Sun [1]} adopts min-heap algorithm to find and retain the k elements with the largest absolute value in the activation gradients for each layer, and discards the remaining elements to improve sparsity. Wei [2]} further applies this scheme to CNN's training, but only evaluated on LeNet. In the case of larger networks and more complex datasets, directly dropping redundant gradients will cause significant loss of learnt information. To alleviate this problem, Zhang [3]} stores the un-propagated gradients at the last learning step in memory and adds them to the gradients before top-k sparsification in the current iteration.
Our work can be categorized into this scope. We propose two novel algorithms to determine the pruning threshold and preserve the valuable information, respectively.
