Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices. Their energy is dominated by the number of multiplies needed to perform the convolutions. Winograd’s minimal filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can reduce the operation count, but these two methods cannot be straightforwardly combined — applying the Winograd transform fills in the sparsity in both the weights and the activations. We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity. First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations. Second, we prune the weights in the Winograd domain to exploit static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by 10.4x, 6.8x and 10.8x respectively with loss of accuracy less than 0.1%, outperforming previous baselines by 2.0x-3.0x. We also show that moving ReLU to the Winograd domain allows more aggressive pruning. Deep Convolutional Neural Networks (CNNs) have shown significant improvement in many machine learning applications. However, CNNs are compute-limited. Their performance is dominated by the number of multiplies needed to perform the convolutions. Moreover, the computational workload of CNNs continues to grow over time. BID16 proposed a CNN model with less than 2.3 × 10 7 multiplies for handwritten digit classification. Later, BID13 developed AlexNet, an ImageNet-winning CNN with more than 1.1 × 10 9 multiplies. In 2014, ImageNetwinning and runner up CNNs increased the number of multiplies to 1.4 × 10 9 BID24 ) and 1.6 × 10 10 BID22 respectively. Despite the powerful representational ability of large scale CNNs, their computational workload prohibits deployment on mobile devices. Two research directions have been explored to address the problem. BID14 proposed using Winograd's minimal filtering algorithm BID25 to reduce the number of multiplies needed to perform 3 × 3 kernel convolutions. On the other end, pruning the model BID5 and exploiting the dynamic sparsity of activations due to ReLU also reduces the required multiplies. Unfortunately, the above two directions are not compatible: the Winograd transformation fills in the zeros in both the weights and the activations FIG0 ) -eliminating the gain from exploiting sparsity. Thus, for a pruned network, Winograd's algorithm actually increases the number of multiplies; the loss of sparsity more than offsets the reduced operation count.In this paper, we introduce two modifications to the original Winograd-based convolution algorithm to eliminate this problem. First, we move the ReLU operation to be after the Winograd transform to also make the activations sparse at the point where the multiplies are performed. Second, we prune the weights after (rather than before) they are transformed. Thus, the weights are sparse when the elementwise multiply is performed -reducing the operation count. Together, these two modifications enable the gains of Winograd's algorithm and of exploiting sparsity to be combined. We open-source our code and models at https://github.com/xingyul/Sparse-Winograd-CNN. In this section, we summarize the experiment results and compare the three models in terms of a) weight and activation dimensions and b) the dynamic density of activations. We then visualize the kernels to illustrate the pattern of the proposed Winograd-ReLU model kernel. DISPLAYFORM0 We have shown that we can combine the computational savings of sparse weights and activations with the savings of the Winograd transform by making two modifcations to conventional CNNs. To make the weights sparse at the point of multiplication, we train and prune the weights in the transform domain. This simple approach does not reduce the workload with respect to spatial pruning, though, so we move the ReLU non-linear operation after the Winograd transform to make the activations sparse at the point of multiplication. Moving ReLU to the Winograd domain also allows the weights to be more aggressively pruned without losing accuracy. With a 2 × 2 output patch (p = 4), the net result is a reduction of 10.4×, 6.8× and 10.8× in computation on three datasets: CIFAR-10, CIFAR-100 and ImageNet.We plan to extend this work in the following directions. First, we expect that even greater savings on computation can be realized by using larger patch sizes (e.g., p = 6), and there may be benefit in exploring different Winograd transformation matrices (B,G and A). Second, we expect that using different pruning rates r i for each network layer will help maintain accuracy and improve overall workload reduction. Finally, we expect that combining our Winograd-ReLU network with other network simplification techniques, e.g. quantization of weights and/or activations BID4 BID18 BID20 , will reduce the energy of computation even further.
Prune and ReLU in Winograd domain for efficient convolutional neural network
{"src": "scitldr-AIC_train", "paper_id": "HJzgZ3JCW"}