With the increasing complexity of problems the size of deep convolutional neural networks (CNNs) is also increasing exponentially to a point where CNNs are so huge that they are already hitting the computational limits of the devices on which they are running. There is a significant amount of redundancy in these huge CNNs that can be removed (i.e: pruned) to accelerate them. It is well known that neural nets are nature inspired, it is also a fact that pruning these neural nets is also nature inspired. According to human brain development studies [1]} “one of the fundamental phenomena in brain development is the reduction of the amount of synapses that occurs between early childhood and puberty” and is commonly referred to as synaptic pruning. One of the earliest work on pruning neural networks was performed by LeCun in the 1990s [2]} in which network connections were pruned based on 2nd derivative of weights. Development of CNNs in 1998 again by LeCun [3]} and their successful application in ImageNet classification challenge in 2012 [4]} resulted in a revival of this concept of pruning in 2015 [5]}. They used L1 norm to prune away lighter weights.
<FIGURE>