Rotate the ReLU to implicitly sparsify deep networks
In the era of Deep Neural Network based solutions for a variety of real-life
tasks, having a compact and energy-efficient deployable model has become fairly
important. Most of the existing deep architectures use Rectifier Linear Unit
(ReLU) activation. In this paper, we propose a novel idea of rotating the ReLU
activation to give one more degree of freedom to the architecture. We show that
this activation wherein the rotation is learned via training results in the
elimination of those parameters/filters in the network which are not important
for the task. In other words, rotated ReLU seems to be doing implicit
sparsification. The slopes of the rotated ReLU activations act as coarse
feature extractors and unnecessary features can be eliminated before
retraining. Our studies indicate that features always choose to pass through a
lesser number of filters in architectures such as ResNet and its variants.
Hence, by rotating the ReLU, the weights or the filters that are not necessary
are automatically identified and can be dropped thus giving rise to significant
savings in memory and computation. Furthermore, in some cases, we also notice
that along with saving in memory and computation we also obtain improvement
over the reported performance of the corresponding baseline work in the popular
datasets such as MNIST, CIFAR-10, CIFAR-100, and SVHN.