	We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via \emph{ad hoc} processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces {parameter sharing} throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to \textbf{180$\times$} smaller and theoretically up to \textbf{16$\times$} faster than the well-established baselines, without noticeable performance drop. Despite remarkable successes in various applications, including e.g. audio classification, speech recognition and natural language processing, deep neural networks (DNNs) usually suffer following two problems that stem from their inherent huge parameter space. First, most of state-of-the-art deep architectures are prone to over-fitting even when trained on large datasets BID42 . Secondly, DNNs usually consume large amount of storage memory and energy BID17 . Therefore these networks are difficult to embed into devices with limited memory and power (such as portable devices or chips). Most existing networks aim to reduce computational budget through network pruning BID16 BID1 BID11 , filter factorization BID23 BID28 , low bit representation BID36 for weights and knowledge transfering BID20 . In contrast to the above works that ignore the strong dependencies among weights and learn filters independently based on existing network architectures, this paper proposes to explicitly enforce the parameter sharing among filters to more effectively learn compact and efficient deep networks.In this paper, we propose a Weight Sampling deep neural network (i.e. WSNet) to significantly reduce both the model size and computation cost of deep networks, achieving more than 100× smaller size and up to 16× speedup at negligible performance drop or even achieving better performance than the baseline (i.e. conventional networks that learn filters independently). Specifically, WSNet is parameterized by layer-wise condensed filters from which each filter participating in actual convolutions can be directly sampled, in both spatial and channel dimensions. Since condensed filters have significantly fewer parameters than independently trained filters as in conventional CNNs, learning by sampling from them makes WSNet a more compact model compared to conventional CNNs. In addition, to reduce the ubiquitous computational redundancy in convolving the overlapped filters and input patches, we propose an integral image based method to dramatically reduce the computation cost of WSNet in both training and inference. The integral image method is also advantageous because it enables weight sampling with different filter size and minimizes computational overhead to enhance the learning capability of WSNet.In order to demonstrate the efficacy of WSNet, we conduct extensive experiments on the challenging acoustic scene classification and music detection tasks. On each test dataset, including MusicDet200K (a self-collected dataset, as detailed in Section 4), ESC-50 (Piczak, 2015a) , UrbanSound8K BID40 and DCASE BID45 , WSNet significantly reduces the model size of the baseline by 100× with comparable or even higher classification accuracy. When compressing more than 180×, WSNet is only subject to negligible accuracy drop. At the same time, WSNet significantly reduces the computation cost (up to 16×). Such results strongly establish the capability of WSNet to learn compact and efficient networks. Although we detailed experiments mostly limited to 1D CNNs in this paper, we will explore how the same approach can be naturally generalized to 2D CNNs in future work. In this paper, we present a class of Weight Sampling networks (WSNet) which are highly compact and efficient. A novel weight sampling method is proposed to sample filters from condensed filters which are much smaller than the independently trained filters in conventional networks. The weight sampling in conducted in two dimensions of the condensed filters, i.e. by spatial sampling and channel sampling. TAB2
We present a novel network architecture for learning compact and efficient deep neural networks
{"src": "scitldr-AIC_train", "paper_id": "H1I3M7Z0b"}