
Lottery Ticket Hypothesis. The lottery ticket hypothesis and the definition of the “winning ticket ” are firstly proposed in [1]}.
Concurrent work [2]} finds that the identical initialized weights will not provide any advantage over training with randomly initialized weights at relatively large learning rates. [2]}, [4]}, [5]} also confirm that the matching subnetworks at nontrivial sparsity are hard to find in more challenging tasks.
The following work [6]} extends the subnetwork training from initial weights to the weights at early stage of pretraining (rewinding), and improve the accuracy in more challenging tasks at nontrivial sparsity.
