Modern neural networks often require deep compositions of high-dimensional nonlinear functions (wide architecture) to achieve high test accuracy, and thus can have overwhelming number of parameters. Repeated high cost in prediction at test-time makes neural networks ill-suited for devices with constrained memory or computational power. We introduce an efficient mechanism, reshaped tensor decomposition, to compress neural networks by exploiting three types of invariant structures: periodicity, modulation and low rank. Our reshaped tensor decomposition method exploits such invariance structures using a technique called tensorization (reshaping the layers into higher-order tensors) combined with higher order tensor decompositions on top of the tensorized layers. Our compression method improves low rank approximation methods and can be incorporated to (is complementary to) most of the existing compression methods for neural networks to achieve better compression. Experiments on LeNet-5 (MNIST), ResNet-32 (CI- FAR10) and ResNet-50 (ImageNet) demonstrate that our reshaped tensor decomposition outperforms (5% test accuracy improvement universally on CIFAR10) the state-of-the-art low-rank approximation techniques under same compression rate, besides achieving orders of magnitude faster convergence rates. Modern neural networks achieve unprecedented accuracy over many difficult learning problems at the cost of deeper and wider architectures with overwhelming number of model parameters. The large number of model parameters causes repeated high cost in test-time as predictions require loading the network into the memory and repeatedly passing the unseen examples through the large network. Therefore, the model size becomes a practical bottleneck when neural networks are deployed on constrained devices, such as smartphones and IoT cameras.Compressing a successful large network (i.e., reducing the number of parameters), while maintaining its performance, is non-trivial. Many approaches have been employed, including pruning, quantization, encoding and knowledge distillation (see appendix A for a detailed survey). A complementary compression technique, on top of which the aforementioned approaches can be used, is low rank approximation. For instance, singular value decomposition (SVD) can be performed on fully connected layers (weights matrices) and tensor decomposition on convolutional layers (convolutional kernels). Low rank approximation methods can work well and reduce the number of parameters by a factor polynomial in the dimension only when the weight matrices or convolutional kernels have low rank structures, which might not always hold in practice.We propose to exploit additional invariant structures in the neural network for compression. A set of experiments on several benchmark datasets justified our conjecture (Section 4): large neural networks have some invariant structures, namely periodicity, modulation and low rank, which make part of the parameters redundant. Consider this toy example of a vector with periodic structure [1, 2, 3, 1, 2, 3, 1, 2, 3] or modulated structure [1, 1, 1, 2, 2, 2, 3, 3, 3] in FIG23 . The number of parameters needed to represent this vector, naively, is 9. However if we map or reshape the vector into a higher order object, for instance, a matrix [1,1,1;2,2,2;3,3,3] where the columns of the matrix are repeated, then apparently this reshaped matrix can be decomposed into rank one without losing information. Therefore only 6 parameters are needed to represent the original length-9 vector. [1, 2, 3, 1, 2, 3, 1, 2, 3] Periodic structure [1, 1, 1, 2, 2, 2, 3, 3, 3] [ ] FIG23 : A toy example of invariant structures. The periodic and modulated structures are picked out by exploiting the low rank structure in the reshaped matrix.Although the invariant structures in large neural networks allow compression of redundant parameters, designing a sophisticated way of storing a minimal representation of the parameters (while maintaining the expressive power of the network) is nontrivial. To solve this problem, we proposed a new framework called reshaped tensor decomposition (RTD) which has three phases:1. Tensorization. We reshape the neural network layers into higher-order tensors.• For instance, consider a special square tensor convolutional kernel T ∈ R D×D×D×D , we reshape T into a higher m-order tensor 2. Higher-order tensor decomposition. We deploy tensor decomposition (a low rank approximation technique detailed in section 3) on the tensorized layers to exploit the periodic, modulated as well as low rank structures in the original layers.• A rank-R tensor decomposition of the above 4-order tensor T will result in R number of components (each contains 4D parameters), and thus 4DR number of parameters in totalsmaller than the original D 4 number of parameters if R is small.• A rank-R tensor decomposition of the above reshaped m-order kernel tensor T ′ maps the layer into m + 1 narrower layers. The decomposition will result in R number of components with mD 4 m parameters and thus mD 4 m R in total -better than the 4DR number of parameters required by doing tensor decomposition on the original tensor T (D is usually large). Now the weights of the tensorized neural networks are the components of the tensor, i.e., result of the tensor decomposition. However, decomposing higher order tensors is challenging and known methods are not guaranteed to converge to the minimum error decomposition (Hillar & Lim, 2013) . Therefore fine tuning is needed to achieve high performance.3. Data reconstruction-based sequential tuning. We fine-tune the parameters using a data reconstruction-based sequential tuning (Seq) method which minimizes the difference between training output of the uncompressed and compressed, layer by layer. Our Seq tuning is a novel approach inspired by a sequential training method proved to converge faster and achieve guaranteed accuracy using a boosting framework (Huang et al., 2017) . Unlike traditional end-to-end (E2E ) backpropagation through the entire network, Seq tunes individual compressed "blocks" one at a time, reducing the memory and complexity required during compression. We describe an efficient mechanism for compressing neural networks by tensorizing network layers. We implement tensorized decompositions to find approximations of the tensorized kernel, potentially preserving invariance structures missed by implementing decompositions on the original kernels. We extend vector/matrix operations to their higher order tensor counterparts, providing systematic notations and libraries for tensorization of neural networks and higher order tensor decompositions.As a future step, we will explore optimizing the parallel implementations of the tensor algebra. Recognition, pp. 1984 Recognition, pp. -1992 Recognition, pp. , 2015 .
Compression of neural networks which improves the state-of-the-art low rank approximation techniques and is complementary to most of other compression techniques. 
{"src": "scitldr-AIC_train", "paper_id": "rkl85oRqYX"}