Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates
Benchmarking the tradeoff between neural network accuracy and training time
is computationally expensive. Here we show how a multiplicative cyclic learning
rate schedule can be used to construct a tradeoff curve in a single training
run. We generate cyclic tradeoff curves for combinations of training methods
such as Blurpool, Channels Last, Label Smoothing and MixUp, and highlight how
these cyclic tradeoff curves can be used to evaluate the effects of algorithmic
choices on network training efficiency.