It is well-known that deeper neural networks are harder to train than shallower ones. In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network gets deeper, and as residual connections are added to the architecture. Computing a series of quantitative measures on the Hessian spectrum, we show that the Hessian eigenvalue distribution in deeper networks has substantially heavier tails (equivalently, more outlier eigenvalues), which makes the network harder to optimize with first-order methods. We show that adding residual connections mitigates this effect substantially, suggesting a mechanism by which residual connections improve training. Practical experience in deep learning suggests that the increased capacity that comes with deeper models can significantly improve their predictive performance. It has also been observed that as the network becomes deeper, training becomes harder. In convolutional neural networks (CNNs), residual connections BID5 are used to alleviate this problem. Various explanations are provided for this phenomenon: BID6 suggests that residual connections reduce the flatness of the landscape, whereas BID3 questions this premise, noting that the extremal eigenvalues of the loss Hessian are much larger when residual connections are present: large Hessian eigenvalues indicate that the curvature of the loss is much sharper, and less flat. In a different line of work, BID0 observes that the gradients with respect to inputs in deeper networks decorrelate with depth, and suggest that residual connections reduce the 'shattering' of the gradients.In this paper, we explore the interaction between depth and the loss geometry. We first establish that gradient explosion or vanishing is not responsible for the slowing down of training, as is commonly believed. Searching for an alternative explanation, we study the Hessian eigenvalue density (using the tools introduced in BID3 to obtain estimates of the eigenvalue histogram or density). The classical theory of strongly convex optimization tells us that optimization is slow when the spectrum simultaneously contains very small and very large eigenvalues (i.e., optimization rate is dependent on κ = λ max /λ min ). Following this intuition, we focus on examining the relative spread of the Hessian eigenvalues. In particular, we quantify the extent of the large outliers by computing some scale-invariant classical statistics of the Hessian eigenvalues, namely the skewness and kurtosis. Finally, we observe that in comparable models with residual connections, these magnitude of these outliers is substantially mitigated. In BID3 , it is hypothesised that batch normalization suppresses large outlier eigenvalues, thereby speeding up training; in this paper, we present evidence that residual connections speed up training through essentially the same channel.Throughout, the dataset of interest is CIFAR-10; we describe the specific model architectures used in Appendix A. In this paper, we have presented qualitative and quantitative evidence that depth increases outlier eigenvalues in the Hessian, and that residual connections mitigate this. We believe that this touches upon some of the fundamental dynamics of optimizing neural networks, and that any theoretical explanation of residual connections needs to explain this.
Network depth increases outlier eigenvalues in the Hessian. Residual connections mitigate this.
{"src": "scitldr-AIC_train", "paper_id": "SyxJ2y2qaE"}