With the increasing demand to deploy convolutional neural networks (CNNs) on mobile platforms, the sparse kernel approach was proposed, which could save more parameters than the standard convolution while maintaining accuracy. However, despite the great potential, no prior research has pointed out how to craft an sparse kernel design with such potential (i.e., effective design), and all prior works just adopt simple combinations of existing sparse kernels such as group convolution. Meanwhile due to the large design space it is also impossible to try all combinations of existing sparse kernels. In this paper, we are the first in the field to consider how to craft an effective sparse kernel design by eliminating the large design space. Specifically, we present a sparse kernel scheme to illustrate how to reduce the space from three aspects. First, in terms of composition we remove designs composed of repeated layers. Second, to remove designs with large accuracy degradation, we find an unified property named~\emph{information field} behind various sparse kernel designs, which could directly indicate the final accuracy. Last, we remove designs in two cases where a better parameter efficiency could be achieved. Additionally, we provide detailed efficiency analysis on the final 4 designs in our scheme. Experimental results validate the idea of our scheme by showing that our scheme is able to find designs which are more efficient in using parameters and computation with similar or higher accuracy. CNNs have achieved unprecedented success in visual recognition tasks. The development of mobile devices drives the increasing demand to deploy these deep networks on mobile platforms such as cell phones and self-driving cars. However, CNNs are usually resource-intensive, making them difficult to deploy on these memory-constrained and energy-limited platforms.To enable the deployment, one intuitive idea is to reduce the model size. Model compression is the major research trend for it. Previously several techniques have been proposed, including pruning BID18 , quantization BID28 and low rank approximation BID6 . Though these approaches can can offer a reasonable parameter reduction with minor accuracy degradation, they suffer from the three drawbacks: 1) the irregular network structure after compression, which limits performance and throughput on GPU; 2) the increased training complexity due to the additional compression or re-training process; and 3) the heuristic compression ratios depending on networks, which cannot be precisely controlled.Recently the sparse kernel approach was proposed to mitigate these problems by directly training networks using structural (large granularity) sparse convolutional kernels with fixed compression ratios. The idea of sparse kernel was originally proposed as different types of convolutional approach. Later researchers explore their usages in the context of CNNs by combining some of these sparse kernels to save parameters/computation against the standard convolution. For example, MobileNets BID12 realize 7x parameter savings with only 1% accuracy loss by adopting the combination of two sparse kernels, depthwise convolution BID26 and pointwise convoluiton BID20 , to replace the standard convolution in their networks.However, despite the great potential with sparse kernel approach to save parameters/computation while maintaining accuracy, it is still mysterious in the field regarding how to craft an sparse kernel design with such potential (i.e., effective sparse kernel design). Prior works like MobileNet BID12 and Xception BID1 just adopt simple combinations of existing sparse kernels, and no one really points out the reasons why they choose such kind of design. Meanwhile, it has been a long-existing question in the field whether there is any other sparse kernel design that is more efficient than all state-of-the-art ones while also maintaining a similar accuracy with the standard convolution.To answer this question, a native idea is to try all possible combinations and get the final accuracy for each of them. Unfortunately, the number of combination will grow exponentially with the number of kernels in a design, and thus it is infeasible to train each of them. Specifically, even if we limit the design space to four common types of sparse kernels -group convolution BID16 , depthwise convolution BID26 , pointwise convolution BID20 and pointwise group convolution ) -the total number of possible combinations would be 4 k , given that k is the number of sparse kernels we allow to use in a design (note that each sparse kernel can appear more than once in a design).In this paper, we craft the effective sparse kernel design by efficiently eliminating poor candidates from the large design space. Specifically , we reduce the design space from three aspects: composition, performance and efficiency. First, observing that in normal CNNs it is quite common to have multiple blocks which contain repeated patterns such as layers or structures, we eliminate the design space by ignoring the combinations including repeated patterns. Second, realizing that removing designs with large accuracy degradation would significantly reduce the design space, we identify a easily measurable quantity named information field behind various sparse kernel designs, which is closely related to the model accuracy. We get rid of designs that lead to a smaller information field compared to the standard convolution model. Last, in order to achieve a better parameter efficiency, we remove redundant sparse kernels in a design if the same size of information field is already retained by other sparse kernels in the design. With all aforementioned knowledge , we present a sparse kernel scheme that incorporates the final four different designs manually reduced from the original design space.Additionally, in practice, researchers would also like to select the most parameter/computation efficient sparse kernel designs based on their needs, which drives the demand to study the efficiency for different sparse kernel designs. Previously no research has investigated on the efficiency for any sparse kernel design. In this paper, three aspects of efficiency are addressed for each of the sparse kernel designs in our scheme: 1) what are the factors which could affect the efficiency for each design? 2) how does each factor affect the efficiency alone? 3) when is the best efficiency achieved combining all these factors in different real situations?Besides, we show that the accuracy of models composed of new designs in our scheme are better than that of all state-of-the-art methods under the same constraint of parameters, which implies that more efficient designs are constructed by our scheme and again validates the effectiveness of our idea.The contributions of our paper can be summarized as follows:• We are the first in the field to point out that the information field is the key for the sparse kernel designs. Meanwhile we observe the model accuracy is positively correlated to the size of the information field.• We present a sparse kernel scheme to illustrate how to eliminate the original design space from three aspects and incorporate the final 4 types of designs along with rigorous mathematical foundation on the efficiency.• We provide some potential network designs which are in the scope of our scheme and have not been explored yet and show that they could have superior performances. In this paper, we present a scheme to craft the effective sparse kernel design by eliminating the large design space from three aspects: composition, performance and efficiency. During the process to reduce the design space, we find an unified property named information field behind various designs, which could directly indicate the final accuracy. Meanwhile we show the final 4 designs in our scheme along with detailed efficiency analysis. Experimental results also validate the idea of our scheme.
We are the first in the field to show how to craft an effective sparse kernel design from three aspects: composition, performance and efficiency.
{"src": "scitldr-AIC_train", "paper_id": "rJlg1n05YX"}