Experimental setups:
In this paper, we conduct our experiments using different learning rates. We empirically set the (initial) learning rate from extremely small to normal, then to very large based on the network and dataset. At each learning rate, we conduct a series of experiments described in Section REF , and each experiment is run three times. For IMP(\(\cdot \) ), we follow the settings in [1]}, [2]} that 20% of the weights are pruned in each iteration. For OMP(\(\cdot \) ), we directly prune the network to the same sparsity ratio as IMP(\(\cdot \) ).
On CIFAR-10/100, We train the network for 160 epochs and the learning rates decrease by a factor of 10 after 80 and 120 epochs.
On ImageNet-1K, We train the network for 90 epochs and cosine annealing learning rate schedule is used.
We conduct our experiments on NVIDIA A100 with 8 GPUs.
Detailed experiment settings are listed in Appendix .
<FIGURE>