To reduce memory footprint and run-time latency, techniques such as neural net-work pruning and binarization have been explored separately.  However, it is un-clear how to combine the best of the two worlds to get extremely small and efficient models .  In this paper, we, for the first time, define the filter-level pruning problem for binary neural networks, which cannot be solved by simply migrating existing structural pruning methods for full-precision models .  A novel learning-based approach is proposed to prune filters in our main/subsidiary network frame-work, where the main network is responsible for learning representative features to optimize the prediction performance, and the subsidiary component works as a filter selector on the main network . To avoid gradient mismatch when training the subsidiary component, we propose a layer-wise and bottom-up scheme .  We also provide the theoretical and experimental comparison between our learning-based and greedy rule-based methods .  Finally, we empirically demonstrate the effectiveness of our approach applied on several binary models,  including binarizedNIN, VGG-11, and ResNet-18, on various image classification datasets .  For bi-nary ResNet-18 on ImageNet, we use 78.6% filters but can achieve slightly better test error 49.87% (50.02%-0.15%) than the original model Deep neural networks (DNN), especially deep convolution neural networks (DCNN), have made remarkable strides during the last decade. From the first ImageNet Challenge winner network, AlexNet, to the more recent state-of-the-art, ResNet, we observe that DNNs are growing substantially deeper and more complex. These modern deep neural networks have millions of weights, rendering them both memory intensive and computationally expensive. To reduce computational cost, the research into network acceleration and compression emerges as an active field.A family of popular compression methods are the DNN pruning algorithms, which are not only efficient in both memory and speed, but also enjoy relatively simple procedure and intuition. This line of research is motivated by the theoretical analysis and empirical discovery that redundancy does exist in both human brains and several deep models BID7 BID8 . According to the objects to prune, we can categorize existing research according to the level of the object, such as connection (weights)-level pruning, unit/channel/filter-level pruning, and layer-level pruning BID28 . Connection-level pruning is the most widely studied approach, which produces sparse networks whose weights are stored as sparse tensors. Although both the footprint memory and the I/O consumption are reduced BID12 , Such methods are often not helpful towards the goal of computation acceleration unless specifically-designed hardware is leveraged. This is because the dimensions of the weight tensor remain unchanged, though many entries are zeroed-out. As a wellknown fact, the MAC operations on random structured sparse matrices are generally not too much faster than the dense ones of the same dimension. In contrast, structural pruning techniques BID28 , such as unit/channel/filter-level pruning, are more hardware friendly, since they aim to produce tensors of reduced dimensions or having specific structures. Using these techniques, it is possible to achieve both computation acceleration and memory compression on general hardware and is common for deep learning frameworks.We consider the structural network pruning problem for a specific family of neural networks -binary neural networks. A binary neural network is a compressed network of a general deep neural network through the quantization strategy. Convolution operations in DCNN 1 inherently involve matrix multiplication and accumulation (MAC). MAC operations become much more energy efficient if we use low-precision (1 bit or more) fixed-point number to approximate weights and activation functions (i.e., to quantify neurons) BID3 . To the extreme extent, the MAC operation can even be degenerated to Boolean operations, if both weights and activation are binarized. Such binary networks have been reported to achieve ∼58x computation saving and ∼32x memory saving in practice. However, the binarization operation often introduces noises into DNNs , thus the representation capacity of DNNs will be impacted significantly, especially if we also binarize the activation function. Consequently, binary neural networks inevitably require larger model size (more parameters) to compensate for the loss of representation capacity.Although Boolean operation in binary neural networks is already quite cheap, even smaller models are still highly desired for low-power embedded systems, like smart-phones and wearable devices in virtual reality applications. Even though quantization (e.g., binarization) has significantly reduced the redundancy of each weight/neuron representation, our experiment shows that there is still heavy redundancy in binary neural networks, in terms of network topology. In fact, quantization and pruning are orthogonal strategies to compress neural networks: Quantization reduces the precision of parameters such as weights and activations, while pruning trims the connections in neural networks so as to attain the tightest network topology. However, previous studies on network pruning are all designed for full-precision models and cannot be directly applied for binary neural networks whose both weights and activations are 1-bit numbers. For example, it no longer makes any sense to prune filters by comparing the magnitude or L 1 norm of binary weights, and it is nonsensical to minimize the distance between two binary output tensors.We, for the first time, define the problem of simplifying binary neural networks and try to learn extremely efficient deep learning models by combining pruning and quantization strategies. Our experimental results demonstrate that filters in binary neural networks are redundant and learning-based pruning filter selection is constantly better than those existing rule-based greedy pruning criteria (like by weight magnitude or L 1 norm).We propose a learning-based method to simplify binary neural network with a main-subsidiary framework, where the main network is responsible for learning representative features to optimize the prediction performance, whereas the subsidiary component works as a filter selector on the main network to optimize the efficiency. The contributions of this paper are summarized as follows:• We propose a learning-based structural pruning method for binary neural networks to significantly reduce the number of filters/channels but still preserve the prediction performance on large-scale problems like the ImageNet Challenge.• We show that our non-greedy learning-based method is superior to the classical rule-based methods in selecting which objects to prune. We design a main-subsidiary framework to iteratively learn and prune feature maps. Limitations of the rule-based methods and advantages of the learning-based methods are demonstrated by theoretical and experimental results. In addition , we also provide a mathematical analysis for L 1 -norm based methods.• To avoid gradient mismatch of the subsidiary component, we train this network in a layerwise and bottom-up scheme. Experimentally , the iterative training scheme helps the main network to adopt the pruning of previous layers and find a better local optimal point.2 RELATED WORK 2.1 PRUNING Deep Neural Network pruning has been explored in many different ways for a long time. BID13 proposed Optimal Brain Surgeon (OBS) to measure the weight importance using the second-order derivative information of loss function by Taylor expansion. BID9 further adapts OBS for deep neural networks and has reduced the retraining time. Deep Compression BID12 prunes connections based on weight magnitude and achieved great compression ratio. The idea of dynamic masks BID10 is also used for pruning. Other approaches used Bayesian methods and exploited the diversity of neurons to remove weights BID23 BID22 . However, these methods focus on pruning independent connection without considering group information. Even though they harvest sparse connections, it is still hard to attain the desired speedup on hardware.To address the issues in connection-level pruning, researchers proposed to increase the groupsparsity by applying sparse constraints to the channels, filters, and even layers BID28 BID0 BID25 BID1 . used LASSO constraints and reconstruction loss to guide network channel selection. introduced L 1 -Norm rank to prune filters , which reduces redundancy and preserves the relatively important filters using a greedy policy. BID21 leverages a scaling factor from batch normalization to prune channels. To encourage the scaling factor to be sparse , a regularization term is added to the loss function. On one hand, methods mentioned above are all designed for full-precision models and cannot be trivially transferred to binary networks. For example, to avoid introducing any non-Boolean operations, batch normalization in binary neural networks (like XNOR-Net) typically doesn't have scaling (γ) and shifting (β) parameters BID3 . Since all weights and activation only have two possible values {1, −1}, it is also invalid to apply classical tricks such as ranking filters by their L 1 -Norms, adding a LASSO constraint, or minimizing the reconstruction error between two binary vectors. On the other hand, greedy policies that ignore the correlations between filters cannot preserve all important filters.
we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.
{"src": "scitldr-AIC_train", "paper_id": "ryxfHnCctX"}