In this paper, we propose an efficient framework to accelerate convolutional neural networks. We utilize two types of acceleration methods: pruning and hints. Pruning can reduce model size by removing channels of layers. Hints can improve the performance of student model by transferring knowledge from teacher model. We demonstrate that pruning and hints are complementary to each other. On one hand, hints can benefit pruning by maintaining similar feature representations. On the other hand, the model pruned from teacher networks is a good initialization for student model, which increases the transferability between two networks. Our approach performs pruning stage and hints stage iteratively to further improve the
 performance. Furthermore, we propose an algorithm to reconstruct the parameters of hints layer and make the pruned model more suitable for hints. Experiments were conducted on various tasks including classification and pose estimation. Results on CIFAR-10, ImageNet and COCO demonstrate the generalization and superiority of our framework. In recent years, convolutional neural networks (CNN) have been applied in many computer vision tasks, e.g. classification BID21 ; BID6 , objects detection BID8 ; BID30 , and pose estimation BID25 . The success of CNN drives the development of computer vision. However, restricted by large model size as well as computation complexity, many CNN models are difficult to be put into practical use directly. To solve the problem, more and more researches have focused on accelerating models without degradation of performance.Pruning and knowledge distillation are two of mainstream methods in model acceleration. The goal of pruning is to remove less important parameters while maintaining similar performance of the original model. Despite pruning methods' superiority, we notice that for many pruning methods with the increase of pruned channel number, the performance of pruned model drops rapidlly. Knowledge distillation describes teacher-student framework: use high-level representations from teacher model to supervise student model. Hints method BID31 shares a similar idea of knowledge distillation, where the feature map of teacher model is used as high-level representations. According to BID36 , the student network can achieve better performance in knowledge transfer if its initialization can produce similar features as the teacher model. Inspired by this work, we propose that pruned model outputs similar features with original model's and provide a good initialization for student model, which does help distillation. And on the other hand, hints can help reconstruct parameters and alleviate degradation of performance caused by pruning operation. FIG0 illustrates the motivation of our framework. Based on this analysis, we propose an algorithm: we do pruning and hints operation iteratively. And for each iteration, we conduct a reconstructing step between pruning and hints operations. And we demonstrate that this reconstructing operation can provide a better initialization for student model and promote hints step (See FIG1 . We name our method as PWH Framework. To our best knowledge, we are the first to combine pruning and hints together as a framework.Our framework can be applied on different vision tasks. Experiments on CIFAR- 10 Krizhevsky & Hinton (2009) , ImageNet Deng et al. (2016) and COCO Lin et al. (2014) Hints can help pruned model reconstruct parameters. And the network pruned from the teacher model can provide a good initialization for student model in hints learning.effectiveness of our framework. Furthermore, our method is a framework where different pruning and hints methods can be included.To summarize, the contributions of this paper are as follows: FORMULA0 We analyze the properties of pruning and hints methods and show that these two model acceleration methods are complementary to each other. (2) To our best knowledge, this is the first work that combines pruning and hints. Our framework is easy to be extended to different pruning and hints methods. (3) Sufficient experiments show the effectiveness of our framework on different datasets for different tasks. In this paper, we propose PWH Framework, an iterative framework for model acceleration. Our framework takes the advantage of both pruning and hints methods. To our best knowledge, this is the first work that combine these two model acceleration methods. Furthermore, we conduct reconstructing operation between hints and pruning steps as a cascader. We analyze the property of these two methods and show they are complementary to each other: pruning provides a better initialization for student model and hints method helps to adjust parameters in pruned model. Experiments on CIFAR-10, ImageNet and COCO datasets for classification and pose estimation tasks demonstrate the superiority of PWH Framework.
This is a work aiming for boosting all the existing pruning and mimic method.
{"src": "scitldr-AIC_train", "paper_id": "Hyffti0ctQ"}