Most of the online and offline pruning strategies are limited to width-wise or filter pruning only. There has been multiple attempts of reducing the depth of the network. This is usually performed by removing several layers and filters from the network in an adhoc manners followed by retraining (for e.g. Yolo vs. Tiny Yolo [1]}[2]}, ResNet-101 vs. ResNet-56 [3]}). This adversely affect the representative power of the network and creates challenges during retraining. In more recent literature the issue of retraining is addressed using knowledge distillation [4]} where the unpruned network called teacher network is used to train a smaller student network via a loss function that minimizes both the data loss as well as the discrepancy between soft logits of teacher and student network. In this work, instead of simplifying a network in an adhoc manner, we extend the idea of online pruning via scaling factors and show how these can be applied to any structured element of any granularity.
