Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual architectures naturally encourage features to move along the negative gradient of loss during the feedforward phase. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and hurts generalization performance, and show that simple existing strategies can help alleviating this problem. Traditionally, deep neural network architectures (e.g. VGG Simonyan & Zisserman (2014) , AlexNet Krizhevsky et al. (2012) , etc. ) have been compositional in nature, meaning a hidden layer applies an affine transformation followed by non-linearity, with a different transformation at each layer. However, a major problem with deep architectures has been that of vanishing and exploding gradients. To address this problem, solutions like better activations (ReLU Nair & Hinton (2010) ), weight initialization methods Glorot & Bengio (2010) ; He et al. (2015) and normalization methods Ioffe & Szegedy (2015) ; BID0 have been proposed. Nonetheless, training compositional networks deeper than 15 − 20 layers remains a challenging task.Recently, residual networks (Resnets He et al. (2016a) ) were introduced to tackle these issues and are considered a breakthrough in deep learning because of their ability to learn very deep networks and achieve state-of-the-art performance. Besides this, performance of Resnets are generally found to remain largely unaffected by removing individual residual blocks or shuffling adjacent blocks Veit et al. (2016) . These attributes of Resnets stem from the fact that residual blocks transform representations additively instead of compositionally (like traditional deep networks). This additive framework along with the aforementioned attributes has given rise to two school of thoughts about Resnets-the ensemble view where they are thought to learn an exponential ensemble of shallower models Veit et al. (2016) , and the unrolled iterative estimation view Liao & Poggio (2016) ; Greff et al. (2016) , where Resnet layers are thought to iteratively refine representations instead of learning new ones. While the success of Resnets may be attributed partly to both these views, our work takes steps towards achieving a deeper understanding of Resnets in terms of its iterative feature refinement perspective. Our contributions are as follows:1. We study Resnets analytically and provide a formal view of iterative feature refinement using Taylor's expansion, showing that for any loss function, a residual block naturally encourages representations to move along the negative gradient of the loss with respect to hidden representations. Each residual block is therefore encouraged to take a gradient step in order to minimize the loss in the hidden representation space. We empirically confirm this by measuring the cosine between the output of a residual block and the gradient of loss with respect to the hidden representations prior to the application of the residual block.2. We empirically observe that Resnet blocks can perform both hierarchical representation learning (where each block discovers a different representation) and iterative feature refinement (where each block improves slightly but keeps the semantics of the representation of the previous layer). Specifically in Resnets, lower residual blocks learn to perform representation learning, meaning that they change representations significantly and removing these blocks can sometimes drastically hurt prediction performance. The higher blocks on the other hand essentially learn to perform iterative inference-minimizing the loss function by moving the hidden representation along the negative gradient direction. In the presence of shortcut connections 1 , representation learning is dominantly performed by the shortcut connection layer and most of residual blocks tend to perform iterative feature refinement.3. The iterative refinement view suggests that deep networks can potentially leverage intensive parameter sharing for the layer performing iterative inference. But sharing large number of residual blocks without loss of performance has not been successfully achieved yet. Towards this end we study two ways of reusing residual blocks: 1. Sharing residual blocks during training; 2. Unrolling a residual block for more steps that it was trained to unroll. We find that training Resnet with naively shared blocks leads to bad performance. We expose reasons for this failure and investigate a preliminary fix for this problem. Our main contribution is formalizing the view of iterative refinement in Resnets and showing analytically that residual blocks naturally encourage representations to move in the half space of negative loss gradient, thus implementing a gradient descent in the activation space (each block reduces loss and improves accuracy). We validate theory experimentally on a wide range of Resnet architectures.We further explored two forms of sharing blocks in Resnet. We show that Resnet can be unrolled to more steps than it was trained on. Next, we found that counterintuitively training residual blocks with shared blocks leads to overfitting. While we propose a variant of batch normalization to mitigate it, we leave further investigation of this phenomena for future work. We hope that our developed formal view, and practical results, will aid analysis of other models employing iterative inference and residual connections. ∂ho , then it is equivalent to updating the parameters of the convolution layer using a gradient update step. To see this, consider the change in h o from updating parameters using gradient descent with step size η. This is given by, DISPLAYFORM0 Thus, moving h o in the half space of − ∂L ∂ho has the same effect as that achieved by updating the parameters W, b using gradient descent. Although we found this insight interesting, we don't build upon it in this paper. We leave this as a future work.
Residual connections really perform iterative inference
{"src": "scitldr-AIC_train", "paper_id": "SJa9iHgAZ"}