ResNet: Residual Neural Networks (ResNet) [1]}, introduced the idea of skip connections on CNNs. The “identity shortcut connection" present in ResNets, usually skipping two or three layers in the model, allowing for efficient training of deep CNNs, which are known to suffer from the vanishing gradient problem. This problem is present in the back-propagation of the calculated gradients to earlier layers. As the gradient is propagated backwards, repeated multiplications might produce very small gradients, resulting in performance degradation. By having residual connections, residual nets learn residual functions concerning the layer inputs. Furthermore, instead of learning a direct mapping using stacked-layers, residual connections let these layers learn residual mappings. This means that instead of having a layer learning a desired mapping \(H(x)\) , regarding to the input \(x\) , residual connections allow to reframe this mapping as \(F(x) := H(x) - x\) , which can then be reframed to \(H(x) := F(x)+x\) . More, if the identity mapping is optimal, residuals can be pushed to zero, meaning that residual networks will, at least, have the same performance of networks using stacked-layers without residual connections.
