To examine our premise that pruning factors can prune the depth of the network too we created a custom VGG-16 network which had residual connections across all layers except at 4 cases of dimension mismatch. Although dimension mismatch can be handled via pooling and convolution by \(1\times 1 \times d_l\)  or by using mixed connections [1]}, however for simplicity we chose not to handle the dimension mismatch. In addition to this the nature of our original network inherently had just 4 cases of dimension mismatch resulting in significant room in the network to prune. After creating these residual connections we added a pruning factor in front of all the layers with residual connections such that the output of the current layer is multiplied to a pruning factor before being added to the residual connection. We then apply Regularization with weightage (1/total num of prunable) to make these pruning factors sparse. Zeroing out these pruning factors means removing that whole layer from the network.
