Find Winning Ticket at Early Stage of Training. The potential of training a sparse network from initialization suggested by the lottery ticket hypothesis has motivated the study of deriving the “winning tickets” at an early stage of training, thereby accelerating training process. There is a number of work in this direction.
[1]} conducts a retraining process after searching sub-network topology for a few epochs.
[2]} examines the network state during early iterations of training, and analyzes the weight distribution and its reliance on the dataset.
SNIP [3]} finds the sparse mask based on the saliency score of each weight that is obtained after training the dense model for only a few iterations.
GraSP [4]} prunes weights based on preserving the gradient flow in the network.
