Broadly, two types of pruning techniques are used to compress neural networks: unstructured pruning and structured pruning. Seminal studies on unstructured pruning [1]}, [2]} remove unimportant network weights based on the Hessian of the network's error function. Among others, alternative approaches include low rank matrix factorization of the final weight layers [3]} or pruning the unimportant connections below a threshold [4]}. Studies under structured pruning regime remove entire neurons/nodes (used interchangeably henceforth) keeping the networks dense [5]}, [6]}, [7]}. Though dense networks can benefit from modern hardware, sparse models outperform dense ones for a fixed number of parameters across domains [8]}, [9]}, [10]}. [11]} is a recent review which highlights these observations after investigating 81 studies on pruning techniques. Our work is closely relates to [12]}, where a DPP sampling technique is used to select a set of diverse neurons/nodes to be preserved during pruning. The authors also introduce a reweighting procedure to compensate contributions of the pruned neurons in the network. Finally, they compare DIVNET (DPP node pruning with reweighting as in [12]}) with random and importance node pruning [5]} on real datasets.
