Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be ``compressed to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size that, combined with off-the-shelf compression algorithms, leads to state-of-the-art generalization guarantees. In particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. Additionally, we show that compressibility of models that tend to overfit is limited. Empirical results show that an increase in overfitting increases the number of bits required to describe a trained network. A pivotal question in machine learning is why deep networks perform well despite overparameterization. These models often have many more parameters than the number of examples they are trained on, which enables them to drastically overfit to training data BID39 . In common practice, however, such networks perform well on previously unseen data.Explaining the generalization performance of neural networks is an active area of current research. Attempts have been made at adapting classical measures such as VC-dimension BID14 or margin/norm bounds BID4 , but such approaches have yielded bounds that are vacuous by orders of magnitudes. Other authors have explored modifications of the training procedure to obtain networks with provable generalization guarantees BID10 . Such procedures often differ substantially from standard procedures used by practitioners, and empirical evidence suggests that they fail to improve performance in practice BID38 .We begin with an empirical observation: it is often possible to "compress" trained neural networks by finding essentially equivalent models that can be described in a much smaller number of bits; see BID9 for a survey. Inspired by classical results relating small model size and generalization performance (often known as Occam's razor), we establish a new generalization bound based on the effective compressed size of a trained neural network. Combining this bound with off-the-shelf compression schemes yields the first non-vacuous generalization bounds in practical problems. The main contribution of the present paper is the demonstration that, unlike many other measures, this measure is effective in the deep-learning regime.Generalization bound arguments typically identify some notion of complexity of a learning problem, and bound generalization error in terms of that complexity. Conceptually , the notion of complexity we identify is: complexity = compressed size − remaining structure.(1) The first term on the right-hand side represents the link between generalization and explicit compression. The second term corrects for superfluous structure that remains in the compressed representation. For instance, the predictions of trained neural networks are often robust to perturbations of the network weights. Thus, a representation of a neural network by its weights carries some irrelevant information. We show that accounting for this robustness can substantially reduce effective complexity.Our results allow us to derive explicit generalization guarantees using off-the-shelf neural network compression schemes. In particular:• The generalization bound can be evaluated by compressing a trained network, measuring the effective compressed size, and substituting this value into the bound.• Using off-the-shelf neural network compression schemes with this recipe yields bounds that are state-of-the-art, including the first non-vacuous bounds for modern convolutional neural nets.The above result takes a compression algorithm and outputs a generalization bound on nets compressed by that algorithm. We provide a complementary result by showing that if a model tends to overfit then there is an absolute limit on how much it can be compressed. We consider a classifier as a (measurable ) function of a random training set, so the classifier is viewed as a random variable. We show that the entropy of this random variable is lower bounded by a function of the expected degree of overfitting. Additionally, we use the randomization tests of BID39 to show empirically that increased overfitting implies worse compressibility, for a fixed compression scheme.The relationship between small model size and generalization is hardly new: the idea is a variant of Occam's razor, and has been used explicitly in classical generalization theory BID34 BID5 BID27 BID16 BID33 ). However, the use of highly overparameterized models in deep learning seems to directly contradict the Occam principle. Indeed, the study of generalization and the study of compression in deep learning has been largely disjoint; the later has been primarily motivated by computational and storage limitations, such as those arising from applications on mobile devices BID9 . Our results show that Occam type arguments remain powerful in the deep learning regime. The link between compression and generalization is also used in work by BID0 , who study compressibility arising from a form of noise stability. Our results are substantially different, and closer in spirit to the work of BID10 ; see Section 3 for a detailed discussion. BID39 study the problem of generalization in deep learning empirically. They observe that standard deep net architectures-which generalize well on real-world data-are able to achieve perfect training accuracy on randomly labelled data. Of course, in this case the test error is no better than random guessing . Accordingly, any approach to controlling generalization error of deep nets must selectively and preferentially bound the generalization error of models that are actually plausible outputs of the training procedure applied to real-world data. Following Langford & Caruana (2002) ; BID10 , we make use of the PAC-Bayesian framework BID29 BID7 BID30 . This framework allows us to encode prior beliefs about which learned models are plausible as a (prior) distribution π over possible parameter settings. The main challenge in developing a bound in the PAC-Bayes framework bound is to articulate a distribution π that encodes the relative plausibilities of possible outputs of the training procedure. The key insight is that, implicitly, any compression scheme is a statement about model plausibilities: good compression is achieved by assigning short codes to the most probable models, and so the probable models are those with short codes. It has been a long standing observation by practitioners that despite the large capacity of models used in deep learning practice, empirical results demonstrate good generalization performance. We show that with no modifications, a standard engineering pipeline of training and compressing a network leads to demonstrable and non-vacuous generalization guarantees. These are the first such results on networks and problems at a practical scale, and mirror the experience of practitioners that best results are often achieved without heavy regularization or modifications to the optimizer BID38 .The connection between compression and generalization raises a number of important questions. Foremost , what are its limitations? The fact that our bounds are non-vacuous implies the link between compression and generalization is non-trivial. However, the bounds are far from tight. If significantly better compression rates were achievable, the resulting bounds would even be of practical value. For example, if a network trained on ImageNet to 90% training and 70% testing accuracy could be compressed to an effective size of 30 KiB-about one order of magnitude smaller than our current compression-that would yield a sharp bound on the generalization error.A PROOF OF THEOREM 4.3 In this section we describe the construction of the prior π and prove the bound on the KL-divergence claimed in Theorem 4.3. Intuitively, we would like to express our prior as a mixture over all possible decoded points of the compression algorithm. More precisely, define the mixture component π S,Q,C associated with a triplet (S, Q, C) as: DISPLAYFORM0 We then define our prior π as a weighted mixture over all triplets, weighted by the code length of the triplet: DISPLAYFORM1 where the sum is taken over all S and C which are representable by our code, and all Q = (q 1 , . . . , q k ) ∈ {1, . . . , r} k . In practice, S takes values in all possible subsets of {1, . . . , p}, and C takes values in F r , where F ⊆ R is a chosen finite subset of representable real numbers (such as those that may be represented by IEEE-754 single precision numbers), and r is a chosen quantization level. We now give the proof of Theorem 4.3.Proof. We have that: DISPLAYFORM2 where we must have Z ≤ 1 by the same argument as in the proof of Theorem 4.1Suppose that the output of our compression algorithm is a triplet (Ŝ,Q,Ĉ). We recall that our posterior ρ is given by a normal centered at w(Ŝ,Q,Ĉ) with variance σ 2 , and we may thus compute the KL-divergence: DISPLAYFORM3 We are now left with the mixture term, which is a mixture of r k many terms in dimension k, and thus computationally untractable. However, we note that we are in a special case where the mixture itself is independent across coordinates. Indeed, let φ τ denote the density of the univariate normal distribution with mean 0 and variance τ 2 , we note that we may write the mixture as: DISPLAYFORM4 Additionally, as our chosen stochastic estimator ρ is independent over the coordinates, the KLdivergence decomposes over the coordinates, to obtain: DISPLAYFORM5 Plugging the above computation into (13) gives the desired result.
We obtain non-vacuous generalization bounds on ImageNet-scale deep neural networks by combining an original PAC-Bayes bound and an off-the-shelf neural network compression method.
{"src": "scitldr-AIC_train", "paper_id": "BJgqqsAct7"}