Takeaway: The above two observations indicate that
the winning tickets are more likely to exist at a small learning rate or at an insufficient training epochs,
but may not exist at a relatively large learning rate or sufficient training epochs (also observed in [1]}).
However, we would like to point out that using a relatively large learning rate (e.g., 0.1) and sufficient training epochs (e.g., 160, which is the standard settings on CIFAR-10) result in a notably higher accuracy for the pretrained DNN (92.3% vs. 88.0%).
This point is largely missing in the previous discussions, and questions whether the previously identified “winning tickets” are meaningful enough.
