While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly. Deep models have proven to work well in applications such as computer vision BID18 BID8 BID14 , speech recognition , and natural language processing BID35 BID6 BID25 . Many deep models have millions of parameters, which is more than the number of training samples, but the models still generalize well BID11 .On the other hand, classical learning theory suggests the model generalization capability is closely related to the "complexity" of the hypothesis space, usually measured in terms of number of parameters, Rademacher complexity or VC-dimension. This seems to be a contradiction to the empirical observations that over-parameterized models generalize well on the test data 1 . Indeed , even if the hypothesis space is complex, the final solution learned from a given training set may still be simple. This suggests the generalization capability of the model is also related to the property of the solution. BID15 and BID1 empirically observe that the generalization ability of a model is related to the spectrum of the Hessian matrix ∇ 2 L(w * ) evaluated at the solution, and large eigenvalues of the ∇ 2 L(w * ) often leads to poor model generalization. Also, BID15 , BID1 and BID31 introduce several different metrics to measure the "sharpness" of the solution, and demonstrate the connection between the sharpness metric and the generalization empirically. BID2 later points out that most of the Hessian-based sharpness measures are problematic and cannot be applied directly to explain generalization. In particular , they show that the geometry of the parameters in RELU-MLP can be modified drastically by re-parameterization.Another line of work originates from Bayesian analysis. Mackay (1995 ) first introduced Taylor expansion to approximate the (log) posterior, and considered the second-order term, characterized by the Hessian of the loss function, as a way of evaluating the model simplicity, or "Occam factor". Recently BID34 use this factor to penalize sharp minima, and determine the optimal batch size. BID4 connect the PAC-Bayes bound and the Bayesian marginal likelihood when the loss is (bounded) negative log-likelihood, which leads to an alternative perspective on Occam's razor. BID19 , and more recently, BID7 BID28 BID29 use PAC-Bayes bound to analyze the generalization behavior of the deep models.Since the PAC-Bayes bound holds uniformly for all "posteriors", it also holds for some particular "posterior", for example, the solution parameter perturbed with noise. This provides a natural The sharp minimum, even though it approximates the true label better, has some complex structures in its predicted labels, while the flat minimum seems to produce a simpler classification boundary. We connect the smoothness of the solution with the model generalization in the PAC-Bayes framework. We prove that the generalization power of a model is related to the Hessian and the smoothness of the solution, the scales of the parameters, as well as the number of training samples. In particular, we prove that the best perturbation level scales roughly as the inverse of the square root of the Hessian, which mostly cancels out scaling effect in the re-parameterization suggested by BID2 . To the best of our knowledge, this is the first work that integrate Hessian in the model generalization bound rigorously. It also roughly explains the effect of re-parameterization over the generalization. Based on our generalization bound, we propose a new metric to test the model generalization and a new perturbation algorithm that adjusts the perturbation levels according to the Hessian. Finally, we empirically demonstrate the effect of our algorithm is similar to a regularizer in its ability to attain better performance on unseen data.
a theory connecting Hessian of the solution and the generalization power of the model
{"src": "scitldr-AIC_train", "paper_id": "BJxOHs0cKm"}