When there are no residual connection as Figure REF  shows, however, we do not see much difference between IMP and OMP. Compare to the IMP method in Figure REF , the advantages of the IMP to OMP is diminished.
Note that the landscape will become much more rugged if residual connections are removed from a network [1]}. We conjecture that in our constructed no-residual ResNet-32, the optimization becomes too difficult and neither IMP nor OMP is effective enough to explore a smooth route towards local minima: hence no much difference observed between them.
<FIGURE><FIGURE>