The geometric picture easily confirms that the best ResNet structure is to replicate \(U_{ResNet}(l\delta )=I+H_{ResNet}(l\delta )\) . Another related observation is that the highway network[1]} usually performs worse than the ResNet since it introduces unnecessary complexity to the system by adding extra nonlinear transformations to the system so the structure of the highway network does not match the natural structure given by the first-order approximation.
