Network pruning is widely used for reducing the heavy computational cost of deep models. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. In this work, we make a rather surprising observation: fine-tuning a pruned model only gives comparable or even worse performance than training that model with randomly initialized weights. Our results have several implications: 1) training a large, over-parameterized model is not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are not necessarily useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited weights, is what leads to the efficiency benefit in the final model, which suggests that some pruning algorithms could be seen as performing network architecture search. Network pruning is a commonly used approach for obtaining an efficient neural network. A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2) prune the unimportant weights according to a certain criterion, and 3) fine-tune the pruned model to regain accuracy. Generally, there are two common beliefs behind this pruning procedure. First, it is believed that training a large network first is important [1] and the three-stage pipeline can outperform directly training the small model from scratch. Second, both the architectures and the weights of the pruned model are believed to be essential for the final efficient model, which is why most existing pruning methods choose to fine-tune the pruned model instead of training it from scratch. Also because of this, how to select the set of important weights is a very active research topic [1, 2, 3, 4].Predefined : prune x% channels in each layer Automatic: prune a%, b%, c%, d% channels in each layer A 4-layer model Figure 1 : Difference between predefined and non-predefined (automatically discovered) target architectures. The sparsity x is user-specified, while a, b, c, d are determined by the pruning algorithm.In this work, we show that both of the beliefs mentioned above are not necessarily true. We make a surprising observation that directly training the target pruned model from random initialization can achieve the same or better performance as the model obtained from the three-stage pipeline. This means that, for pruning methods with a predefined target architecture (Figure 1 ), starting with a large model is not necessary and one could instead directly train the target model from scratch. For pruning algorithms with automatically discovered target architectures, what brings the efficiency benefit is the obtained architecture, instead of the inherited weights. Our results advocate a rethinking of existing network pruning algorithms: the preserved "important" weights from the large model are not necessary for obtaining an efficient final model; instead, the value of automatic network pruning methods may lie in identifying efficient architectures and performing implicit architecture search. In practice, for methods with predefined target architectures, training from scratch is more computationally efficient and saves us from implementing the pruning procedure and tuning the additional hyper-parameters. Still, pruning methods are useful when a pretrained large model is already given, in this case fine-tuning is much faster. Also, obtaining multiple models of different sizes can be done quickly by pruning from a large model by different ratios.In summary, our experiments have shown that training the small pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical "training, pruning and fine-tuning" procedure. This changed our previous belief that over-parameterization is necessary for obtaining a final efficient model, and the understanding on effectiveness of inheriting weights that are considered important by the pruning criteria. We further demonstrated the value of automatic pruning algorithms could be regarded as finding efficient architectures and providing architecture design guidelines.
In network pruning, fine-tuning a pruned model only gives comparable or worse performance than training it from scratch. This advocate a rethinking of existing pruning algorithms.
{"src": "scitldr-AIC_train", "paper_id": "r1eLk2mKiX"}