Deep CNNs have achieved state-of-the-art performance for numerous machine learning and computer vision tasks in recent years, but as they have become increasingly deep, the number of parameters they use has also increased, making them hard to deploy in memory-constrained environments and difficult to interpret. Machine learning theory implies that such networks are highly over-parameterised and that it should be possible to reduce their size without sacrificing accuracy, and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this. In this paper, we take a further step in this direction by proposing a filter-sharing approach to compressing deep CNNs that reduces their memory footprint by repeatedly applying a single convolutional mapping of learned filters to simulate a CNN pipeline. We show, via experiments on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet that this allows us to reduce the parameter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional to their depth, whilst leaving their accuracy largely unaffected. At a broader level, our approach also indicates how the scale-space regularities found in visual signals can be leveraged to build neural architectures that are more parsimonious and interpretable. Deep CNNs have achieved state-of-the-art results on a wide range of tasks, from image understanding (Redmon & Farhadi, 2017; Jetley et al., 2017; Kim et al., 2018; Oktay et al., 2018) to natural language processing (Oord et al., 2016; Massiceti et al., 2018) . However, these network architectures are often highly overparameterised (Zhang et al., 2016) , and thus require the supervision of a large number of input-output mappings and significant training time to adapt their parameters to any given task. Recent studies have discovered several different redundancies in these network architectures (Garipov et al., 2016; Hubara* et al., 2018; Wu et al., 2018; Frankle & Carbin, 2019; Yang et al., 2019a; b) and certain simplicities (Pérez et al., 2018; Jetley et al., 2018) in the functions that they implement. For instance, Frankle & Carbin (2019) showed that a large classification network can be distilled down to a small sub-network that, owing to its lucky initialisation, is trainable in isolation without compromising the original classification accuracy. Jetley et al. (2018) observed that deep classification networks learn simplistic non-linearities for class identification, a fact that might well underlie their adversarial vulnerability, whilst challenging the need for complex architectures. Attempts at knowledge distillation have regularly demonstrated that it is possible to train small student architectures to mimic larger teacher networks by using ancillary information extracted from the latter, such as their attention patterns (Zagoruyko & Komodakis, 2017) , predicted soft-target distributions (Hinton et al., 2014) or other kinds of meta-data (Lopes et al., 2017) . These works and others continue to expose the high level of parameter redundancy in deep CNNs, and comprise a foundational body of work towards studying and simplifying networks for safe and practical use. Our paper experiments with yet another scheme for simplifying CNNs, in the hope that it will not only shrink the effective footprint of these networks, but also open up new pathways for network understanding and redesign. In particular, we propose the use of a common set of convolutional filters at different levels of a convolutional hierarchy to achieve class disentanglement. Mathematically, we formulate a classification CNN as an iterative function in which a small set of learned convolutional mappings are applied repeatedly as different layers of a CNN pipeline (see Figure 1) . In doing so, we are able to reduce the parameter count of the network by a factor proportional to its depth, whilst leaving its accuracy largely unaffected. We also investigate the introduction of non-shared linear widths n of the shared convolutional layer, compared to the baseline VGGNet (Simonyan & Zisserman, 2015) , for CIFAR-10 (a) and . The compression factor is plotted on a logarithmic scale. layers before certain shared convolutional layers to enhance the flexibility of the model by allowing it to linearly combine shared filter maps for the disentanglement task. Earlier, Fig. 2 showed the accuracy vs. compression trade-off for S-VGGNet, relative to the original VGGNet (Simonyan & Zisserman, 2015) , for different widths n of the shared convolutional layer. Here, Fig. 3 illustrates the improvements in accuracy due to the learned linear layers (i.e. the blend- The compression factor C is plotted on a log scale. Simonyan & Zisserman, 2015) and (for CIFAR-10) variants of LegoNet (Yang et al., 2019b) , another state-of-the-art compression method. Baseline models marked with a * were retrained for this study. ing layers) on CIFAR-10, CIFAR-100 and Tiny ImageNet. Observably, the use of the linear layers provides greater benefit for datasets that involve discriminating between a larger number of classes, such as CIFAR-100 and Tiny ImageNet. For CIFAR-10, CIFAR-100 and Tiny ImageNet we compare the accuracies of the best-performing 'SL' variants of VGGNet with those of the baseline architecture (and competing compression methods for these datasets, where available) in Table 1 . For CIFAR-10 (see Table 1b ), we are able to achieve comparable classification accuracy to the VGGNet baseline using only n = 256 channels for our shared convolutional layer, which yields a compression factor of ≈ 17×. For CIFAR-100 (Table 1c) , which has 10× more classes, we had to use n = 512 channels to achieve comparable accuracy, but this still yields a significant compression factor of 4.3. Higher compression factors can be achieved by reducing the number of channels, in exchange for some loss in accuracy. Evaluating our shared architecture on Tiny ImageNet (in Table 1d ) evidences a similar trend in the results, with SL2-VGGNet (n = 512 channels) achieving an accuracy comparable to the non-shared baseline, whilst using only 23% of its parameters. Detailed accuracy and memory usage numbers for E-VGGNet, S-VGGNet and SL-VGGNet, for CIFAR-10, are in Table 1a , while the results for CIFAR-100 and Tiny Imagenet can be found in the appendix (see Table 6 in §A.5) We also evaluate our shared ResNet architecture (SL-ResNet) on Tiny ImageNet and ImageNet, with the results shown in Table 2 (the corresponding results for CIFAR-10 and CIFAR-100 can be found in the appendix, see Table 7 in §A.5). For Tiny ImageNet, our SL-ResNet34 (n = 512) variant is able to achieve a compression rate of 8.4 with only a negligible loss in accuracy. For ImageNet, the same variant similarly achieves a compression rate of 8. (Boulch, 2018) , LegoNet (Yang et al., 2019b) , FSNet (Yang et al., 2019a) and Shared Wide ResNet (SWRN) (Savarese & Maire, 2019) . Baseline models marked with a * were retrained for this study. Figure 4: A visual depiction of the linear layers used to blend the input channels in our approach. We show the layers for the two variants in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red. an accuracy trade-off, we achieve a greater compression rate than competing methods that achieve similar accuracies. Note that SWRN is able to achieve state-of-the-art levels of accuracy, but does not provide savings in the number of parameters. In this paper, we leverage the regularities in visual signals across different scale levels to successfully extend the filter-sharing paradigm to an entire convolutional pipeline for feature extraction. In particular, we instantiate a single convolutional layer and apply it iteratively to simulate conventional VGGNet-like and ResNet-like architectures. We evaluate our shared architectures on four standard benchmarks -CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet -and achieve compression rates that are higher than existing sharing-based methods that have equivalent performance. We further show that even higher compression rates, with little additional loss in performance, can be achieved by combining our method with the magnitude-based weight pruning approach of Han et al. (2015) . Study of our complementarity to more structured pruning techniques targeting complete filters and channels is reserved for future work. We conclude with two final observations. Firstly, our use of blending layers and a parameter to tune the width of the shared convolutional layer n makes it easy to adjust the architecture so as to achieve a desired trade-off between compression rate C and accuracy. Secondly, there are interesting connections between our work and the idea of energy-based pruning explored in (Yang et al., 2017) , where the authors note that a significant fraction of the energy demands of deep network processing come from transferring weights to and from the file system. Our approach bypasses this bottleneck by using the same compact set of weights in an iterative manner. We aim to further investigate this aspect of our method in subsequent work. A APPENDIX
We compress deep CNNs by reusing a single convolutional layer in an iterative manner, thereby reducing their parameter counts by a factor proportional to their depth, whilst leaving their accuracies largely unaffected
{"src": "scitldr-AIC_train", "paper_id": "S1xRxgSFvH"}