We notice that when residual connections exist, the subnetwork achieves higher accuracy at a relatively small learning rate, while a larger learning rate is more preferable in training of a subnetwork without residual connections.
In Figure REF ,
as the residual connection makes the landscape become much smoother [1]}, we can see a subnetwork trained with a small initial learning rate 0.01 achieves a larger contour and a larger basin in the middle, while the contour and basin area with large learning rate 0.1 are relatively small. We conjecture that the optimization is much easier for a smaller initial learning rate on a smooth loss surface, leading to a better network performance.
Without residual connections (as Figure REF ), the above observations are exactly the opposite. Note that the no-residual ResNet-32 creates a more rugged landscape, thus a small initial learning rate 0.01 is more likely to stuck in a sub-optimal local minima, while a large initial learning rate is unlikely to, therefore the SGD process is more likely to find a desired path to high quality solutions.
