We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ﬁnd the optimal network parameters and sparse network structure in a uniﬁed optimization process with trainable pruning thresholds. These thresholds can have ﬁne-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models. Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efﬁciency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures. Despite the impressive success that deep neural networks have achieved in a wide range of challenging tasks, the inference in deep neural network is highly memory-intensive and computationintensive due to the over-parameterization of deep neural networks. Network pruning (LeCun et al. (1990) ; Han et al. (2015) ; Molchanov et al. (2017) ) has been recognized as an effective approach to improving the inference efficiency in resource-limited scenarios. Traditional pruning methods consist of dense network training followed with pruning and fine-tuning iterations. To avoid the expensive pruning and fine-tuning iterations, many sparse training methods (Mocanu et al., 2018; Bellec et al., 2017; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) have been proposed, where the network pruning is conducted during the training process. However, all these methods suffers from following three problems: Coarse-grained predefined pruning schedule. Most of the existing pruning methods use predefined pruning schedule with many additional hyperparameters like pruning a% parameter each time and then fine-tuning for b epochs with totally c pruning steps. It is non-trivial to determine these hyperparameters for network architectures with various degree of complexity. Therefore, usually a fixed pruning schedule is adopted for all the network architectures, which means that a very simple network architecture like LeNet-300-100 will have the same pruning schedule as a far more complex network like ResNet-152. Besides, almost all the existing pruning methods conduct epoch-wise pruning, which means that the pruning is conducted between two epochs and no pruning operation happens inside each epoch. Failure to properly recover the pruned weights. Almost all the existing pruning methods conduct "hard" pruning that prunes weights by directly setting their values to 0. Many works (Guo et al., 2016; Mocanu et al., 2018; He et al., 2018) have argued that the importance of network weights are not fixed and will change dynamically during the pruning and training process. Previously unimportant weights may tend to be important. So the ability to recover the pruned weights is of high significance. However, directly setting the pruned weights to 0 results in the loss of historical parameter importance, which makes it difficult to determine: 1) whether and when each pruned weight should be recovered, 2) what values should be assigned to the recovered weights. Therefore, existing methods that claim to be able to recover the pruned weights simply choose a predefined portion of pruned weights to recover and these recover weights are randomly initialized or initialized to the same value. Failure to properly determine layer-wise pruning rates. Modern neural network architectures usually contain dozens of layers with various number of parameters. Therefore, the degrees of parameter redundancy are very different among the layers. For simplicity, some methods prune the same percentage of parameter at each layer, which is not optimal. To obtain dynamic layer-wise pruning rates, a single global pruning threshold or layer-wise greedy algorithms are applied. Using a single global pruning threshold is exceedingly difficult to assess the local parameter importance of the individual layer, since each layer has a significantly different amount of parameter and contribution to the model performance. This makes pruning algorithms based on a single global threshold inconsistent and non-robust. The problem of layer-by-layer greedy pruning methods is that the unimportant neurons in an early layer may have a significant influence on the responses in later layers, which may result in propagation and amplification of the reconstruction error (Yu et al., 2018) . We propose a novel end-to-end sparse training algorithm that properly solves the above problems. With only one additional hyperparameter used to set the final model sparsity, our method can achieve dynamic fine-grained pruning and recovery during the whole training process. Meanwhile, the layerwise pruning rates will be adjusted automatically with respect to the change of parameter importance during the training and pruning process. Our method achieves state-of-the-art performance compared with other sparse training algorithms. The proposed algorithm has following promising properties: • Step-wise pruning and recovery. A training epoch usually will have tens of thousands of training steps, which is the feed-forward and back-propagation pass for a single mini-batch. Instead of pruning between two training epochs with predefined pruning schedule, our method prunes and recovers the network parameter at each training step, which is far more fine-grained than existing methods. • Neuron-wise or filter-wise trainable thresholds. All the existing methods adopt a single pruning threshold for each layer or the whole architecture. Our method defines a threshold vector for each layers. Therefore, our method adopts neuron-wise pruning thresholds for fully connected and recurrent layer and filter-wise pruning thresholds for convolutional layer. Additionally, all these pruning thresholds are trainable and will be updated automatically via back-propagation. • Dynamic pruning schedule. The training process of deep neural network consists of many hyperparameters. The learning rate is perhaps the most important hyperparameter. Usually, the learning rate will decay during the training process. Our method can automatic adjust the layer-wise pruning rates under different learning rate to get the optimal sparse network structure. • Consistent sparse pattern. Our algorithm can get consistent layer-wise sparse pattern under different model sparsities, which indicates that our method can automatic determine the optimal layer-wise pruning rates given the target model sparsity.
We present a novel network pruning method that can find the optimal sparse structure during the training process with trainable pruning threshold
{"src": "scitldr-AIC_train", "paper_id": "SJlbGJrtDB"}