Deep Neural Networks pruning via the Structured Perspective Regularization
In Machine Learning, Artificial Neural Networks (ANNs) are a very powerful
tool, broadly used in many applications. Often, the selected (deep)
architectures include many layers, and therefore a large amount of parameters,
which makes training, storage and inference expensive. This motivated a stream
of research about compressing the original networks into smaller ones without
excessively sacrificing performances. Among the many proposed compression
approaches, one of the most popular is \emph{pruning}, whereby entire elements
of the ANN (links, nodes, channels, \ldots) and the corresponding weights are
deleted. Since the nature of the problem is inherently combinatorial (what
elements to prune and what not), we propose a new pruning method based on
Operational Research tools. We start from a natural Mixed-Integer-Programming
model for the problem, and we use the Perspective Reformulation technique to
strengthen its continuous relaxation. Projecting away the indicator variables
from this reformulation yields a new regularization term, which we call the
Structured Perspective Regularization, that leads to structured pruning of the
initial architecture. We test our method on some ResNet architectures applied
to CIFAR-10, CIFAR-100 and ImageNet datasets, obtaining competitive
performances w.r.t.~the state of the art for structured pruning.