Comparing the results regarding network structures, we find that when residual connections exist in the network, IMP is more preferable than OMP, and when there are no residual connections the IMP has no advantages over OMP. To further investigate it with “apple-to-apple” comparison, we construct a “ResNet-32-like” network, by removing all residual connections from ResNet-32 while leaving all else intact. We then evaluate the accuracy of IMP and OMP on ResNet-32, versus the newly constructed ResNet-32-like network. We also visualize both optimization trajectories along the contours of the loss surface, using the classical method in [1]}, [2]}.
