LeRaC: Learning Rate Curriculum
Most curriculum learning methods require an approach to sort the data samples
by difficulty, which is often cumbersome to perform. In this work, we propose a
novel curriculum learning approach termed Learning Rate Curriculum (LeRaC),
which leverages the use of a different learning rate for each layer of a neural
network to create a data-free curriculum during the initial training epochs.
More specifically, LeRaC assigns higher learning rates to neural layers closer
to the input, gradually decreasing the learning rates as the layers are placed
farther away from the input. The learning rates increase at various paces
during the first training iterations, until they all reach the same value. From
this point on, the neural model is trained as usual. This creates a model-level
curriculum learning strategy that does not require sorting the examples by
difficulty and is compatible with any neural network, generating higher
performance levels regardless of the architecture. We conduct comprehensive
experiments on eight datasets from the computer vision (CIFAR-10, CIFAR-100,
Tiny ImageNet), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D)
domains, considering various convolutional (ResNet-18, Wide-ResNet-50,
DenseNet-121), recurrent (LSTM) and transformer (CvT, BERT, SepTr)
architectures, comparing our approach with the conventional training regime.
Moreover, we also compare with Curriculum by Smoothing (CBS), a
state-of-the-art data-free curriculum learning approach. Unlike CBS, our
performance improvements over the standard training regime are consistent
across all datasets and models. Furthermore, we significantly surpass CBS in
terms of training time (there is no additional cost over the standard training
regime for LeRaC).