This paper aims to simultaneously accelerate and compress off-the-shelf CNN models via filter pruning strategy. The importance of each filter is evaluated by the proposed entropy-based method first. Then several unimportant filters are discarded to get a smaller CNN model. Finally, fine-tuning is adopted to recover its generalization ability which is damaged during filter pruning. Our method can reduce the size of intermediate activations, which would dominate most memory footprint during model training stage but is less concerned in previous compression methods. Experiments on the ILSVRC-12 benchmark demonstrate the effectiveness of our method. Compared with previous filter importance evaluation criteria, our entropy-based method obtains better performance. We achieve 3.3x speed-up and 16.64x compression on VGG-16, 1.54x acceleration and 1.47x compression on ResNet-50, both with about 1 top-5 accuracy decrease.
Parameter quantization is another classic compression method that has been well studied. One of the widely used methods is product quantization @cite_35 , which decomposes the space into a Cartesian product of low-dimensional subspaces and quantizes each subspace separately. Gong al @cite_27 compared product quantization with several different quantization methods, and found that even with a simple k-means based method, their approach could achieve impressive results. Chen al @cite_38 introduced the hashing trick into model compression, and proposed HashedNets. All connections mapped into the same hash bucket shared the same parameter value. Similar strategy is also adopted in the Deep Compression method @cite_34 . However, @cite_27 @cite_16 required the parameter matrices to be reconstructed during test phase, which limited its further practical application. In order to reduce the run-time memory consumption, Wu al @cite_33 proposed a novel quantization method, which achieved simultaneous acceleration and compression for both convolutional and fully-connected layers.
@cite_35 @cite_38 @cite_33 @cite_16 @cite_27 @cite_34
2582794520 2767123230 2152981240 2785362247
Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively. Application of the hashing method to large-scale image retrieval has drawn much attention because of the high efficiency and favorable accuracy of the method. Its related research generally involves two basic problems: similarity-preserving projection and information-preserving quantization. Most previous works focused on learning projection approaches, while the importance of quantization strategies was ignored. Although several hashing quantization models have been recently proposed to improve retrieval performance by assigning multiple bits to projected directions, these models still suffer from suboptimal results, as the critical information loss that occurs in the quantization procedure is not considered. In this paper, to construct an effective quantization model, we utilize rate-distortion theory in the hashing quantization procedure and minimize the distortion to reduce the information loss. Furthermore, combining principal component analysis with our quantization strategy, we present a quantization-based hashing method named distortion minimization hashing. Extensive experiments involving one synthetic data set and three image data sets demonstrate the superior performance of our proposed methods over several quantization techniques and state-of-the-art hashing methods. Quantization of compressed sensing measurements is typically justified by the robust recovery results of Candes, Romberg and Tao, and of Donoho. These results guarantee that if a uniform quantizer of step size ź is used to quantize m measurements y=źx of a k-sparse signal xźźN, where ź satisfies the restricted isometry property, then the approximate recovery x# via l1-minimization is within O(ź) of x. The simplest and commonly assumed approach is to quantize each measurement independently. In this paper, we show that if instead an rth-order ΣΔ (Sigma---Delta) quantization scheme with the same output alphabet is used to quantize y, then there is an alternative recovery method via Sobolev dual frames which guarantees a reduced approximation error that is of the order ź(k m)(rź1 2)ź for any 0<ź<1, if mźr,źk(logN)1 (1źź). The result holds with high probability on the initial draw of the measurement matrix ź from the Gaussian distribution, and uniformly for all k-sparse signals x whose magnitudes are suitably bounded away from zero on their support. Compression is a key step to deploy large neural networks on resource-constrained platforms. As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight. In this paper, we study the representation power of quantized neural networks. First, we prove the universal approximability of quantized ReLU networks. Then we provide upper bounds of storage size given the approximation error bound and the bit-width of weights for function-independent and function-dependent structures. To the best of the authors' knowledge, this is the first work on the universal approximability as well as the associated storage size bound of quantized neural networks.