The computation necessary for training Transformer-based language models has skyrocketed in recent years.
This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training.
In this work, we revisit three categories of such algorithms:dynamic architectures(layer stacking,layer dropping) ,batch selection(selective backprop,RHO loss) , andefficient optimizers(Lion,Sophia) .
When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to abaselinewith a fully-decayed learning rate.
We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine, which we callreference system time.
We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures:https://github.com/JeanKaddour/NoTrainNoGain.