Modern neural network architectures take advantage of increasingly deeper layers, and various advances in their structure to achieve better performance. While traditional explicit regularization techniques like dropout, weight decay, and data augmentation are still being used in these new models, little about the regularization and generalization effects of these new structures have been studied. 
 Besides being deeper than their predecessors, could newer architectures like ResNet and DenseNet also benefit from their structures' implicit regularization properties? 
 In this work, we investigate the skip connection's effect on network's generalization features. Through experiments, we show that certain neural network architectures contribute to their generalization abilities. Specifically, we study the effect that low-level features have on generalization performance when they are introduced to deeper layers in DenseNet, ResNet as well as networks with 'skip connections'. We show that these low-level representations do help with generalization in multiple settings when both the quality and quantity of training data is decreased. Deep models have achieved significant success in many applications. However, deep models are hard to train and require longer times to converge. A solution by construction is copying the learned layers from the shallower model and setting additional layers to identity mapping. Skip connection proposed in the Residual Network BID0 , shows the new insight of innovation in network structure for computer vision.In the following years, more new and multi-layer-skipping structures have been proposed and proved to have better performance, among which one typical example is DenseNet BID1 . ResNet BID0 , HighwayNet (Rupesh Kumar BID3 and FractalNets BID2 have all succeeded by passing the deep information directly to the shallow layers via shortcut connection. Densenet further maximize the benefit of shortcut connections to the extreme. In DenseNet (more accurately in one dense block) every two layers has been linked, making each layer be able to use the information from all its previous layers. In doing this, DenseNet is able to effectively mitigate the problem of gradient vanishing or degradation, making the input features of each layer various and diverse and the calculation more efficient.Concatenation in Dense Block: the output of each layer will concatenate with its own input and then being passed forward to the next layer together. This makes the input characteristics of the next layer diversified and effectively improves the computation and helps the network to integrate shallow layer features to learn discriminative feature. Meanwhile, the neurons in the same Dense block are interconnected to achieve the effect of feature reused. This is why DenseNet does not need to be very wide and can achieve very good results.Therefore, shortcut connections form the multi-channel model, making the flow of information from input to output unimpeded. Gradient information can also be fed backward directly from the loss function to the the various nodes.In this paper we make the following contributions:• We design experiments to illustrate that on many occasions it is worth adding some skip connections while sacrificing some of the network width. Every single skip connection replacing some of width is able to benefit the whole network's learning ability. Our 'connection-by-connection' adding experiment results can indicate this well.• We perform experiments to show that networks that reuse low-level features in subsequent layers perform better than a simple feed-forward model. We degrade both the quantity and the quality of the training data in different settings and compare the validation performances of these models. Our results suggest that while all models are able to achieve perfect training accuracy, both DenseNet and ResNet are able to exhibit better generalization performance given similar model complexities.• We investigate solutions learned by the three types of networks in both a regression and classification involving task in low dimensions and compare the effects of both the dense connections and the skip connections. We show that the contribution of the feature maps reintroduced to deeper layers via the connections allow for more representational power. By introducing skip connections, modern neural network has proved better performance in computer vision area. This paper investigates how skip connections works in vision task and how they effect the learning power of networks. For this reason, we have design some experiments and verify that networks with skip connections can do the regression best among tested network architectures. It indicates that we can get the insights of this interesting architecture and its tremendous learning power.
Our paper analyses the tremendous representational power of networks especially with 'skip connections', which may be used as a method  for better generalization.
{"src": "scitldr-AIC_train", "paper_id": "rJbs5gbRW"}