Scaling Laws vs Model Architectures :
How does Inductive Bias Inﬂuence Scaling?
Yi TayMostafa DehghaniSamira Abnar Hyung Won C
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer
models. However, not much has been done on the front of investigating the
effect of scaling properties of different inductive biases and model
architectures. Do model architectures scale differently? If so, how does
inductive bias affect scaling behaviour? How does this influence upstream
(pretraining) and downstream (transfer)? This paper conducts a systematic study
of scaling behaviour of ten diverse model architectures such as Transformers,
Switch Transformers, Universal Transformers, Dynamic convolutions, Performers,
and recently proposed MLP-Mixers. Via extensive experiments, we show that (1)
architecture is an indeed an important consideration when performing scaling
and (2) the best performing model can fluctuate at different scales. We believe
that the findings outlined in this work has significant implications to how
model architectures are currently evaluated in the community.
Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung William Fedus Jinfeng Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler