Weight pruning has been introduced as an efficient model compression technique. Even though pruning removes significant amount of weights in a network, memory requirement reduction was limited since conventional sparse matrix formats require significant amount of memory to store index-related information. Moreover, computations associated with such sparse matrix formats are slow because sequential sparse matrix decoding process does not utilize highly parallel computing systems efficiently. As an attempt to compress index information while keeping the decoding process parallelizable, Viterbi-based pruning was suggested. Decoding non-zero weights, however, is still sequential in Viterbi-based pruning. In this paper, we propose a new sparse matrix format in order to enable a highly parallel decoding process of the entire sparse matrix. The proposed sparse matrix is constructed by combining pruning and weight quantization. For the latest RNN models on PTB and WikiText-2 corpus, LSTM parameter storage requirement is compressed 19x using the proposed sparse matrix format compared to the baseline model. Compressed weight and indices can be reconstructed into a dense matrix fast using Viterbi encoders. Simulation results show that the proposed scheme can feed parameters to processing elements 20 % to 106 % faster than the case where the dense matrix values directly come from DRAM. Deep neural networks (DNNs) require significant amounts of memory and computation as the number of training data and the complexity of task increases BID0 . To reduce the memory burden, pruning and quantization have been actively studied. Pruning removes redundant connections of DNNs without accuracy degradation BID6 . The pruned results are usually stored in a sparse matrix format such as compressed sparse row (CSR) format or compressed sparse column (CSC) format, which consists of non-zero values and indices that represent the location of non-zeros. In the sparse matrix formats, the memory requirement for the indices is not negligible.Viterbi-based pruning BID14 significantly reduces the memory footprint of sparse matrix format by compressing the indices of sparse matrices using the Viterbi algorithm BID3 . Although Viterbi-based pruning compresses the index component considerably, weight compression can be further improved in two directions. First, the non-zero values in the sparse matrix can be compressed with quantization. Second, sparse-to-dense matrix conversion in Viterbi-based pruning is relatively slow because assigning non-zero values to the corresponding indices requires sequential processes while indices can be reconstructed in parallel using a Viterbi Decompressor (VD).Various quantization techniques can be applied to compress the non-zero values, but they still cannot reconstruct the dense weight matrix quickly because it takes time to locate non-zero values to the corresponding locations in the dense matrix. These open questions motivate us to find a non-zero value compression method, which also allows parallel sparse-to-dense matrix construction. The contribution of this paper is as follows.(a) To reduce the memory footprint of neural networks further, we propose to combine the Viterbibased pruning BID14 ) with a novel weight-encoding scheme, which also uses the Viterbi-based approach to encode the quantized non-zero values. (b) We suggest two main properties of the weight matrix that increase the probability of finding "good" Viterbi encoded weights. First, the weight matrix with equal composition ratio of '0' and '1' for each bit is desired. Second, using the pruned parameters as "Don't Care" terms increases the probability of finding desired Viterbi weight encoding. (c) We demonstrate that the proposed method can be applied to Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) with various sizes and depths. (d) We show that using the same Viterbi-based approach to compress both indices and non-zero values allows us to build a highly parallel sparse-to-dense reconstruction architecture. Using a custom cycle-simulator , we demonstrate that the reconstruction can be done fast. We proposed a DNN model compression technique with high compression rate and fast dense matrix reconstruction process. We adopted the Viterbi-based pruning and alternating multi-bit quantization technique to reduce the memory requirement for both non-zeros and indices of sparse matrices. Then, we encoded the quantized binary weight codes using Viterbi algorithm once more. As the non-zero values and the corresponding indices are generated in parallel by multiple Viterbi encoders, the sparse-to-dense matrix conversion can be done very fast. We also demonstrated that the proposed scheme significantly reduces the memory requirements of the parameters for both RNN and CNN. A APPENDIX A.1 PRUNING USING THE VITERBI ALGORITHM In Viterbi-based pruning scheme, the binary outputs generated by a Viterbi Decompressor (VD) are used as the index matrix that indicates whether a weight element is pruned ('0') or not ('1'). Suppose the number of elements in a target weight matrix is q, and the number of outputs generated by a VD at each time step is N ind , then only 2 q/N ind binary matrices can be generated by the VD among all 2 q binary matrices. The index matrix which minimizes the accuracy loss should be selected among binary matrix candidates which VD can generate in this pruning scheme, and the Viterbi algorithm is used for this purpose.The overall pruning process is similar to the binary weight encoding process using the Viterbi algorithm in Section 3.3. First, Trellis diagram ( FIG3 ) of the VD which is used for pruning is constructed, and then the cost function is computed by using the path metric and the branch metric. The same path metric shown in Equation 1 in Section 3.3 is used to select the branch which maximizes the path metric between two connected branches from the previous states. On the other hand, a different branch metric λ i,j t is used for pruning, which is expressed as: DISPLAYFORM0 where W i,j,m t is the magnitude of a parameter at the m th VD output and time index t, normalized by the maximum absolute value of all elements in target weight matrix, and TH p is the pruning threshold value determined heuristically. As β i,j,m t gives additional points (penalties) to the parameters with large magnitude to survive (be pruned), the possibility to prune small-magnitude parameters is maximized. S 1 and S 2 are the scaling factors which is empirically determined. BID14 uses 5.0 and 10 4 each). After computing the cost function through the whole time steps, the state with the maximum path metric is chosen, and we trace the previous state by selecting the surviving branch and corresponding indices backward until the first state is reached.The ideal pruning rate of the Viterbi-based pruning is 50 %, because the VD structures act like random number generator and the probability to generate '0' or '1' is 50 % each. For various pruning rates, comparators and comparator threshold value, TH c , are used. A N C -bit comparator receives N c VD outputs and generates 1-bit result whether the value made by the combination of the received VD outputs (e.g. {out 1 , out 2 , · · · , out N ind } where out i indicates the i th VD output) is greater than TH c or not. For example, suppose a 4-bit comparator is used to the VD in Figure 1 and TH c = 3, then the probability for the comparator to generate '1' is 25%(= (3 + 1)/2 4 ) and this percentage is the target pruning rate. Comparators and TH c control the value of pruning rates and the index compression ratio decreases by 1/N c times.It is reported that a low N ind is desired to prune weights of convolutional layers while high N ind can be used to prune the weights of fully-connected layers because of the trade-off between the index compression ratio and the accuracy BID14 . Thus, in our paper, we use N ind = 50 and N c = 5 to prune weights of LSTMs and fully-connected layers in VGG-6 on CIFAR-10. On the other hand, we use N ind = 10 and N c = 5 to prune weights of convolutional layers in VGG-6 on CIFAR-10.
We present a new weight encoding scheme which enables high compression ratio and fast sparse-to-dense matrix conversion.
{"src": "scitldr-AIC_train", "paper_id": "HkfYOoCcYX"}