MINT: Deep Network Compression via Mutual Information-based Neuron Trimming

Most approaches to deep neural network compression via pruning either evaluate a filter's importance using its weights or optimize an alternative objective function with sparsity constraints. While these methods offer a useful way to approximate contributions from similar filters, they often either ignore the dependency between layers or solve a more difficult optimization objective than standard cross-entropy. Our method, Mutual Information-based Neuron Trimming (MINT), approaches deep compression via pruning by enforcing sparsity based on the strength of the relationship between filters of adjacent layers, across every pair of layers. The relationship is calculated using conditional geometric mutual information which evaluates the amount of similar information exchanged between the filters using a graph-based criterion. When pruning a network, we ensure that retained filters contribute the majority of the information towards succeeding layers which ensures high performance. Our novel approach outperforms existing state-of-the-art compression-via-pruning methods on the standard benchmarks for this task: MNIST, CIFAR-10, and ILSVRC2012, across a variety of network architectures. In addition, we discuss our observations of a common denominator between our pruning methodology's response to adversarial attacks and calibration statistics when compared to the original network.


I. INTRODUCTION
Balancing the trade-off between the size of a deep network and achieving high performance is the most important constraint when designing deep neural networks (DNN) that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems like medical diagnosis [1]- [3], autonomous vehicles [4]- [6], and others, they consume a large amount of memory and computational resources which limit their large-scale deployment. With current state-of-the-art deep networks spanning hundreds of millions if not billions of parameters [7], [8], compressing them while maintaining high performance is challenging.
In this work, we approach DNN compression using network pruning [9]. There are two broad approaches to network pruning, (a) unstructured pruning, where a filter's importance is evaluated using weights [9] or constraints like the l 1 norm [10] on them, without considering the overall structure of sparsity induced, and (b) structured pruning, where the objective function is modified to include structured sparsity constraints [11]. Most structured pruning approaches ignore the dependency between layers and the impact of pruning on downstream layers while unstructured pruning methods force the network to optimize a harder and more sensitive optimization objective. The underlying common theme between both approaches is the use of filter weights as a proxy for their importance.
Evaluating a filter's importance purely from its weights is insufficient since it does not take into account the dependencies between filters or account for any form of uncertainty. These factors are critical since higher weight values do not always represent its true importance and a filter's contribution can be compensated elsewhere in the network. Consider the example shown in Fig. 1, where a simple weight-based criterion suggests the removal of small valued weights. However, the mutual information (MI) score, which we use to measure the dependency between pairs of filters and emphasize their importance, values the smaller weights over the large ones. Pruning based on the MI scores would ensure a network where the retained filters pass on as much information as possible to the next layer.
To overcome these issues, we propose Mutual Informationbased Neuron Trimming (MINT) as a novel approach to compression-via-pruning in deep networks which stochastically accounts for the dependency between layers. Fig. 2 outlines our approach. In MINT, we use an estimator for conditional geometric mutual information (GMI), inspired by [12], to measure the dependency between filters of successive layers. Specifically, we use a graph-based criterion (Friedman-Rafsky Statistic [13]) to measure the conditional GMI between the activations of filters at layer l and l + 1, denoted by F  Fig. 2: Illustration of the experimental setup highlighting the components of MINT. Between every pair of filters in consecutive layers (l, l + 1) we compute the conditional geometric mutual information (GMI), using the activations from each filter, as the importance score. The total number of filters in each layer is defined by N (l) and N (l+1) . The conditional GMI score indicates the importance of a filter in layer l's contribution towards a filter in layer l + 1. We then threshold filters based on the importance scores to ensure that we retain only filters that pass the majority of the information to successive layers. Finally, we retrain the network once to maintain a desired level of performance evaluating all such dependencies, we sort the importance scores between filters of every pair of layers in the network. Finally, we threshold a desired percentage of these values to retain filters that contribute the majority of the information to successive layers. Hence, MINT maintains high performance with compressed and retrained networks. Through MINT, we contribute a network pruning method that addresses the need to use dependency between layers as an important factor to prune networks. By maintaining filters that contribute the majority of the information passed between layers, we ensure that the impact of pruning on downstream layers is minimized. In doing so, we achieve state-of-theart performance across multiple Dataset-CNN architecture combinations, highlighting the general applicability of our method.
Further, we empirically analyze our approach using adversarial attacks, expected calibration error, and visualizations that illustrate the focus of learned representations, to provide a better understanding of our approach. We highlight the common denominator between its security vulnerability and decrease in calibration error while illustrating the intended effects of retaining filters that contribute the majority of information between layers.

II. RELATED WORKS
Deep network compression offers a number of strategies to help reduce network size while maintaining high performance, such as low-rank approximations [14]- [16], quantization [17]- [20], knowledge distillation [21], [22], and network pruning [9], [23], [24]. In this work, we focus on network pruning since it offers a controlled set up to study and compare changes in the dynamics of a network when filters are removed. We broadly classify network pruning methods into two categories, unstructured, and structured, which we describe below. Also, we highlight common strategies to calculate multivariate dependencies and how they vary from our method.
A. Network Pruning 1) Unstructured pruning: Some of the earliest in this line of work used the second-order relationship between the objective function and weights of a network to determine which weights to remove or retain [25], [26]. Although these methods provide deep insights into the relationships within networks, their dense computational requirements and large runtimes made them less practical. They were surpassed by an alternative approach that thresholded the weight values themselves to obtain a desired level of sparsity before retraining [9]. Apart from the simplicity of this approach, it also highlighted the importance of re-training from known minima as opposed to from scratch. Instead of pruning weights in one shot, [27] offered a continuous and recursive strategy of using miniiterations to evaluate the importance of connections using their weights, remove unimportant ones and train the network. Similarly, [10] proposed pruning filters using the l 1 -norm of their weight values as a measure of importance. By melding standard network pruning using weights with network quantization and Huffman coding, [28] showed superior compression performance compared to any individual pipeline. However, the direct use of weight values across all these methods does not capture the relationships between different layers or the impact of removing weights on downstream layers. In MINT, we address this issue by explicitly computing the dependency between filters of successive layers and only retaining filters that contribute a majority of the information. This ensures that there isn't a severe impact on downstream layers.
A subset of methods uses data to derive the importance of filter weights. Among them, ThiNet [23] posed the reconstruction of outcomes with the removal of weights as an optimization objective to decide on weights. More recently, NISP [29] used the contribution of neurons towards the reconstruction of outcomes in the second to last layer as a metric to retain/remove filters. These works represent a shift to data-driven logic to consider the downstream impact of pruning, with the use of external computations (e.g., feature ranking in [29]) combined with the quality of features approximated using weights from a single trial. Compared to the deterministic relationship between the weights and feature ranking methods, our method uses a probabilistic approach to measure the dependency between filters, thereby accounting for some form of uncertainty. Further, our method uses one single prune-retrain step compared to multiple iterations of fine-tuning performed in these methods.
2) Structured Pruning: The shift to structured pruning was based on the idea of seamlessly interfacing with hardware systems as opposed to relying on software accelerators. One of the first methods to do this extended the original brain damage [25] formulation to include fixed pattern masks for specified groups while using a group-sparsity regularizer [30]. This idea was further extended in works that used the grouplasso formulation [11], individual or multiple l n norm constraints [31] on the channel parameters and in works that balanced individual vs. group relationships [32], [33] to induce sparsity across desired structures. These methods explicitly affect the training phase of a chosen network by optimizing a harder and more sensitive objective function. Further, sideeffects like-stability to adversarial attacks, calibration, and difference in learned representations have not been fully quantified. In our work, we optimize the standard crossentropy objective function and characterize the behaviour of the original network and their compressed counterparts.
Hybrid methods combine the notion of a modified objective function with the measurement of downstream impact of pruning by enforcing sparsity constraints on the outcomes of groups [34], using a custom group-lasso formulation with a squared dependency on weight values as the importance measure [35] or an adversarial pruned network generator to compete with the features derived from the original network [24]. The disadvantages of these methods include their multi-pass scheme and large training times as well as those inherited from modifying the objective function.

B. Multivariate Dependency Measures
The accurate estimation of multivariate dependency in highdimensional settings is a hard task. The first in this line of work involved Shannon Mutual Information which was succeeded by a number of plug-in estimators including Kernel Density Estimators [36] and KNN estimators [37]. However, their dependence on density estimates and large runtime complexity means they are not suitable for large scale applications including neural networks. A faster plug-in method based on graph theory and nearest neighbour ratios [38] was proposed as an alternative. More solutions that use statistics like KL divergence like [39] or Renyi-α [40] were proposed to help bypass density estimation fully. Instead, in this work, we focus on a conditional GMI estimator, similar to [12], which bypasses the difficult task of density estimation and is nonparametric and scalable.

III. MINT
MINT is a data-driven approach to pruning networks by learning the dependency between filters/nodes of successive layers. For every pair of layers in a deep network, MINT uses the conditional GMI (Section III-A) between the activations of every filter from a chosen layer l and a filter from layer l + 1, given the existence of every other possible filter in layer l to compute an importance score. Here, data flows from layer l to l + 1. Once all such importance scores are evaluated, we remove a fixed portion of filters with the lowest scores. This induces the desired level of sparsity in the network before retraining the network to maintain high accuracy. The core algorithm is outlined and explained in the sections below.

A. Conditional Geometric Mutual Information
In this section, we review conditional GMI estimation [12] and use a close approximation to their method to calculate multivariate dependencies in our proposed algorithm.
Definition We first define a general form of GMI denoted by I p : For parameters p ∈ (0, 1) and q = 1 − p consider two random variables X ∈ R dx and Y ∈ R dy with joint and marginal distributions f (x, y), f (x), and f (y) respectively. The GMI between X and Y is given by Considering the special case of p = q = 1/2 in Eqn. 1 we obtain, The conditional form of this measure is proposed in [12] as, and each filter in the previous layer l. In each ρ() computation, the highlighted filters in layer l represent the conditioned variables while the arrows indicate the filters whose actual dependence we compute. These steps are repeated for every possible combination of filters across every pair of consecutive layers in the network

B. Approach
Setup In a deep network containing a total of L layers, we compute the dependency (ρ) between filters in every consecutive pair of layers. Here, the layer l is closer to the input while layer l+1 is closer to the output among the chosen pair of consecutive layers. The activations for a given node in layer l + 1, are computed as, where x ∈ R m×d , m is the total number of samples, d is the feature dimension and x is the input to a given layer used to compute the activations. σ() is an activation function, w ∈ R N (l) , and b are the weight vector and bias. • For a given pair consecutive of layers (l, l + 1), we compute the dependency between every filter in layer l + 1 in relation to filters in layer l. The main intent of framing the algorithm in this perspective is that the activations from layers closer to the input have a direct effect on downstream layers while the reverse is not true for a forward pass of the network. • Using the activations F () for the selected filters, j ) we compute the conditional GMI between Algorithm 1: MINT pruning between filters of layers (l, l + 1) for Every pair of layers (l, l + 1), l ∈ 1, 2, . . . , them given all the remaining filters in layer l, as shown in Fig. 3. This dependency captures the relationship between filters in the context of all the contributions from the preceding layer. Since the activations of layer l + 1 are the result of contributions from all filters in the preceding layer, we need to account for this when considering the dependence of activations between two selected filters.
j ), the contribution of filters from the previous layer is either retained/removed. We define a threshold δ for this purpose, a key hyper-parameter.
i stores the indices of all filters from layer l that are retained for a selected filter F (l+1) i . The weights for retained filters are left the same while the weights for the entire kernel in the other filters are zeroed out. In the context of fully connected layers, we retain or zero out specific weight values in the weight matrix.
Group Extension While evaluating dependencies between every pair of filters allows us to take a close look at their relationships, it does not scale well to deeper or wider architectures. To address this issue, we evaluate filters in groups rather than one-by-one. We define G as the total number of groups in a layer, where each group contains an equal number of filters. We explore in detail the impact of varying the number of groups in Section IV-D. Although there are multiple approaches to grouping filters, in this work we restrict ourselves to sequential grouping, where groups are constructed from consecutive filters. There is no explicit requirement for a pre-grouping step before our algorithm so long as a balanced grouping of filters is used.
Finer Details MINT is constructed on the assumption that the majority of information from the preceding layer is available and the filter in consideration can selectively retain contributions for a subset of previous filters. This allows us to work on isolated pairs of layers with minimal interference on downstream layers since retaining filters with high MI will ensure the retention of filters that contribute the most information to the next layer. By maintaining as much information as possible between layers, the amount of critical information passed to layers further on is maintained.

IV. EXPERIMENTAL RESULTS
We breakdown the experimental results into three major sections. Section IV-C focuses on the comparison of our method to state-of-the-art deep network compression algorithms, Section IV-D highlights the significance of various hyper-parameters used in MINT, and finally, Section IV-E characterizes MINT-compressed networks w.r.t. their response to adversarial attacks, calibration statistics and learned representations in comparison to their original counterparts. As a prelude to all these three parts, we describe the datasets, models, and metrics used across our experiments. We restrict the implementation details to the appendices.

A. Datasets and Models
Our experiments are divided into the following Dataset-Architecture combinations in order of increasing complexity in dataset and architectures, MNIST [ Fig. 4: (a) An increase in the number of groups per layer allows for finer grouping of filters which in turn leads to more accurate GMI estimates and thresholding. Thus, there is a steady increase in the number of parameters that can be removed to achieve > 98.50 performance. (b) Keeping G = 20, we observe that increasing the number of samples per class improves the GMI estimate accuracy which in turn allows for better thresholding and an increase in the parameters pruned. The values on top of the bar plots are the recognition accuracies this value by using SciPy [50] to convert weight matrices to CSR format and storing them in "npz" files.

C. Comparison against existing methods
As a first step to showcasing MINT's abilities, we compare it against state-of-the-art baselines in network pruning. The baselines in Table I are arranged in ascending order of the percentage of parameters pruned. Our algorithm clearly outperforms most of the SOTA pruning baselines across the number of pruned parameters while maintaining high accuracy and reducing the memory footprint of the network. We note that while some of the pruning baselines listed use multiple prune-retrain steps to achieve their result, we use only a single step to match and outperform them.
In Fig. 5 we take a deeper look at how the overall compression % is spread throughout the network. Comparing Figs. 5a, 5b and 5c, we can establish the strong influence of datasets and network architecture on where redundancies are stored. In the cases of VGG16 and ResNet56, training with CIFAR-10 leads to the storage of possibly redundant information in the latter portion of the networks. The early portions of the network are extremely sensitive to pruning. ResNet50 when trained on ILSVRC2012 forces compression to be more spread out across the network, possibly indicating the spread of redundant features at different levels of the network.

D. Hyper-parameter Empirical Analysis
We take a closer look at two important hyper-parameters that help MINT scale well to deep networks, (a) number of groups in a layer G, and (b) the number of samples per class, m, used to compute the conditional GMI. Below, we look into how each of them impacts the maximum number of parameters pruned to achieve > 98.50% accuracy for MNIST on MLP.  Group size G directly corresponds to the number of filters that are grouped together when computing conditional GMI and thresholding. More groups lead to lesser filters per group, which allows for more fine-grained computation of multivariate dependency and thereby, more precise pruning. In this experiment, m = 250. Results in Fig. 4a match our expectation by illustrating the increase in the upper limit for parameters pruned to achieve the desired performance..

E. Characterization
The standard metric used to compare deep network compression methods is the percentage of parameters pruned while maintaining recognition performance close to the baseline. However, the original intent of compressing networks is to deploy them in real-world scenarios which necessitate other characterizations like robustness to adversarial attacks and an ability to reflect true confidence in predictions.
To understand the impact of pruning networks in the context of adversarial attacks we use two common adversarial attacks, Iterative FGSM [52], which doesn't exclusively target a desired class, and Iterative-LL [53], which targets the selection of the least likely class. Fig. 6 shows the response of other the original and MINT-compressed networks to both attacks. We clearly observe that MINT-compressed networks are more vulnerable to targeted and non-targeted attacks.
While the core idea behind MINT is to retain filters that contribute the majority of the information passed to the next layer, in using a subset of the available filters we remove a certain portion of the information passed down. Fig. 7 compares the portions of the image that contribute towards a desired target class, between the original (top row) and MINT-compressed networks (bottom row). We observe that the use of a subset of filters in the compressed network has Calibration statistics [54], [55] measure the agreement between the confidence provided by the network and the actual probability. These measures provide an orthogonal perspective to adversarial attacks since they measure statistics only for indomain images while adversarial attacks alter the input. Fig. 8 highlights the decrease in Expected Calibration Error (ECE) for the MINT-compressed networks when compared to their original counterparts. The plot illustrates that the histogram trend is closer to matching the ideal trend indicated by the linear red curve. After pruning, the sparse networks seem to behave similarly to a regularizer by focusing on a smaller subset of features and decreasing the ECE. On the other hand, the original networks contain many levels of redundancies which translates to overfitting and having higher ECE.

V. CONCLUSION
In this work, we propose MINT as a novel approach to network pruning in which the dependency between filters of successive layers is used as a measure of importance. We use conditional GMI to evaluate importance and incorporate stochasticity in our algorithm in order to retain filters that pass the majority of the information through layers. In doing so, MINT achieves better pruning performance than SOTA baselines, using a single prune-retrain step. When characterizing the behaviour of MINT-pruned networks, we observe that it behaves like a regularizer and improves the calibration of the network. However, a reduction in the number of filters used and redundancies makes pruned networks susceptible to adversarial attacks. Our future direction of work includes improving the robustness of compressed networks to adversarial attacks as well as detailing the sensitivity of layers in various architectures to network pruning.
ACKNOWLEDGMENT This work has been partially supported (Madan Ravi Ganesh and Jason J. Corso) by NSF IIS 1522904 and NIST 60NANB17D191 and (Salimeh Yasaei Sekeh) by NSF 1920908; the findings are those of the authors only and do not represent any position of these funding bodies. The authors would also like to thank Stephan Lemmer for his invaluable input on the calibration of deep networks.

A. Practical Implementation Hyper-parameter
In Section 3.2 of the main paper, we describe MINT in detail. In the description, we use δ as a threshold on ρ() values to help retain only those dependencies that contribute a majority of the information to the next layer. However, in practice, since the ρ() values are not normalized w.r.t. each other, there are cases when certain layers can be fully removed when the δ parameter is large. In order to protect the network from such behaviour, we use an additional threshold called γ to cap the maximum percentage of filters that can be removed from a selected layer. If the pruning ratio within a layer exceeds the value of γ, then γ is used to re-evaluate δ such that it falls within the desired pruning ratio. The new temporary value of δ is computed using the ρ() values within the pair of layers considered, and is only used prune filters in the selected pair of layers. Here, we observe a relatively steady increase in the maximum number of parameters pruned as the number of groups (G) increases, when no γ is provided or when a linear/quadratic dependence on G is used to compute γ. However, when γ is fixed at 0.8, we observe irregular behaviour, where the maximum percentage of parameters removed decreases with an increase in G. We posit that this behaviour arises from two factors, (a) the varied sensitivity of different layers is not compensated for by a fixed γ value, and (b) the difference in G values leads to different GMI estimates which account for more uncertainty in measurement.

B. Experimental Setup
In the following section we outline the hyper-parameters used during each stage of our experimental pipeline, training, pruning and retraining. The values are provided here in an effort to help users replicate our experiments. We plan to release our code soon. Training Setup: Table B-1 outlines the setups used to obtain trained models from which mutual information (MI) estimates are computed. A standard pretrained model from PyTorch zoo is used for the ResNet50-ImageNet experiments. Pruning Setup: For the conditional geometric mutual information (GMI) estimate we provide a set of parameters describing the number of groups per layer, G as well as the number of samples per class, m. We use the average activation across a filter's dimensions as the samples for GMI computations. MLP -MNIST: Between FC 1 and FC2, G is set to 250 and 100 while between FC 2 and FC 3 it is set to 300 and 10. 650 samples per class are used to compute the conditional GMI estimates. VGG16 -CIFAR10: All the layers between Convolution 4 and Linear 1 use G = 64 while m = 650. γ = 0.45 is used for Convolution layers 5, 6, 7, and 8.