Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Recent works on sparse neural network training (sparse training) have shown
that a compelling trade-off between performance and efficiency can be achieved
by training intrinsically sparse neural networks from scratch. Existing sparse
training methods usually strive to find the best sparse subnetwork possible in
one single run, without involving any expensive dense or pre-training steps.
For instance, dynamic sparse training (DST), as one of the most prominent
directions, is capable of reaching a competitive performance of dense training
by iteratively evolving the sparse topology during the course of training. In
this paper, we argue that it is better to allocate the limited resources to
create multiple low-loss sparse subnetworks and superpose them into a stronger
one, instead of allocating all resources entirely to find an individual
subnetwork. To achieve this, two desiderata are required: (1) efficiently
producing many low-loss subnetworks, the so-called cheap tickets, within one
training process limited to the standard training time used in dense training;
(2) effectively superposing these cheap tickets into one stronger subnetwork
without going over the constrained parameter budget. To corroborate our
conjecture, we present a novel sparse training approach, termed
\textbf{Sup-tickets}, which can satisfy the above two desiderata concurrently
in a single sparse-to-sparse training process. Across various modern
architectures on CIFAR-10/100 and ImageNet, we show that Sup-tickets integrates
seamlessly with the existing sparse training methods and demonstrates
consistent performance improvement.