After selecting the Transformer model architecture, the challenge becomes dealing with the large model size. It may seem that smaller models would train faster than larger ones; however, this does not always hold true. For example, Li et al. [1]} demonstrated that, in some instances, training large models and then compressing results leads to better results. The so called lottery ticket hypothesis [2]}, pruning can be applied to reduce the model size. Also, quantization to a lower precision allows for a smaller model. However, there are inevitable trade offs when using a larger model after pruning/quantization versus a smaller one. The most relevant is ensuring that the training dataset is large to avoid overfitting. Using a small dataset can lead to poor generalization. In general, when trying to train a Transformer model, we recommend using large models instead of the more conventional approach of starting with smaller models and then adding layers.
