In the standard LTH setup [1]}, key training hyperparameters such as learning rate and training epochs were not much scrutinized nor exhaustively tuned. The winning ticket can be identified in the case of small learning rate, but can fail to emerge at higher initial learning rates especially in deeper networks. For instance, in [1]}, the winning tickets can be identified only in the case of small learning rate 0.01 on networks such as ResNet-20 and VGG-19 on CIFAR-10.
At larger learning rates, however, [3]} reveals that the “winning ticket” has no accuracy advantage over the random reinitialization , which contradicts with the original definition.
On the other hand, the settings in [1]} train 78 epochs for ResNet-20 on CIFAR-10. Such insufficient training causes a relatively low pretraining accuracy. When pruned iteratively, the subnetwork accuracy can easily match that pretraining accuracy of the original network. Under such experimental conditions, the existence of the winning ticket is questionable.
