Low-rank feature maps [1]} and LASSO regression based channel selection [2]} have also been proposed to perform pruning through activation maps. Similarly, by forcing the outputs of some channels to be constant allow us to prune those constant channels [3]}. Additional losses have also been proposed to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer while rejecting the other channels [4]}. The major drawback of offline pruning strategies is that in offline structured pruning the network convergence is stopped to generate a new pruned network and then this pruned network is converged again and pruned again, this periodic pruning process may result in a pruned network that fails to achieve the same accuracy as compared to the original network. These issues are addressed via online pruning which is discussed next.
