The main goal of network pruning is imposing sparsity on the neural network by increasing the number of parameters with zero value in order to reduce the architecture size and the computational speedup. Recent advances in deep neural networks came with ideas to train deep architectures that have led to near-human accuracy for image recognition, object categorization and a wide variety of other applications LeCun et al. (2015) ; Maturana & Scherer (2015) ; Schmidhuber (2015) ; Mnih et al. (2013) ; . One possible issue is that an over-parameterized network may make the architecture overcomplicated for the task at hand and it might be prone to over-fitting as well. In addition to the model complexity, a huge amount of computational power is required to train such deep models due to having billions of weights. Moreover, even if a huge model is trained, it cannot be effectively employed for model evaluation on low-power devices mainly due to having exhaustive matrix multiplications Courbariaux et al. (2015) . So far, a wide variety of approaches have been proposed for creating more compact models. Traditional methods include model compression Ba & Caruana (2014) ; , network pruning Han et al. (2015b) , sparsity-inducing regularizer Collins & Kohli (2014) , and low-rank approximation Jaderberg et al. (2014) ; Denton et al. (2014) ; Ioannou et al. (2015) ; Tai et al. (2015) . The aforementioned methods usually induce random connection pruning which yields to few or no improvement in the computational cost. On the other hand, structured pruning methods proposed to compress the architecture with significant computational efficiency Wen et al. (2016) ; Neklyudov et al. (2017) . One of the critical subjects of interest in sparsity learning is to maintain the accuracy level. In this paper, we discuss the intuitive reasons behind the accuracy drop and propose a method to prevent it. The important step is to determine how the sparsity and accuracy are connected together in order to be able to propose a mechanism for controlling the sparsity to prevent severe accuracy drop. In order to connect the sparsity to accuracy, intuitively, the accuracy drop is caused by imposing too much sparsity on the network in a way that the remaining elements cannot transfer enough information for optimal feature extraction for the desired task. Another intuitive reasoning is to argue that the sparsity is not supervised with any attention towards the model performance during optimization. For effective network pruning and feature selection, different approaches such as employing the group lasso for sparse structure learning Yuan & Lin (2006) , structure scale constraining Liu et al. (2015) , and structured regularizing deep architectures known as Structured Sparsity Learning (SSL) Wen et al. (2016) have previously been proposed. For most of the previous research efforts, there is lack of addressing the direct effect of the proposed method on the combination of the sparsity and accuracy drop. One may claim that successful sparsity imposing with negligible accuracy drop might be due to the initial over-parameterizing the network. Moreover, there is no control mechanism to supervise the sparsity operation connected to the model performance which limits the available methods to intensive hyper-parameter tuning and multiple stages of training. Our contribution. We designed and employed a supervised attention mechanism for sparsity learning which: (1) performs model compression for having less number of parameters (2) prevents the accuracy drop by sparsity supervision by paying an attention towards the network using variance regularization and (3) is a generic mechanism that is not restricted by the sparsity penalty or any other limiting assumption regarding the network architecture. To the best of our knowledge, this is the first research effort which proposes a supervised attention mechanism for sparsity learning. Paper Organization. At first, we provide a review of the related research efforts (Section 2). Then, we introduce the attention mechanism which is aimed at forcing some sections of the network to be active (Section 3). Later in Section 4, we propose an algorithm only for the attention supervision. We complement our proposed method in Section 5, by providing experimental results for which we target the sparsity level, accuracy drop and robustness of the model to hyper-parameter tuning. As will be observed, the proposed mechanism prevents the severe accuracy drop in higher levels of sparsity. We will empirically show the robustness to exhaustive hyper-parameter tuning in addition to performance superiority of the proposed method in higher sparsity levels.
Proposing a novel method based on the guided attention to enforce the sparisty in deep neural networks.
{"src": "scitldr-AIC_train", "paper_id": "S1ljpTEFPB"}