Deeper vs Wider: A Revisit of Transformer Configuration
Transformer-based models have delivered impressive results on many tasks,
particularly vision and language tasks. In many model training situations,
conventional configurations are typically adopted. For example, we often set
the base model with hidden dimensions (i.e. model width) to be 768 and the
number of transformer layers (i.e. model depth) to be 12. In this paper, we
revisit these conventional configurations. Through theoretical analysis and
experimental evaluation, we show that the masked autoencoder is effective in
alleviating the over-smoothing issue in deep transformer training. Based on
this finding, we propose Bamboo, an idea of using deeper and narrower
transformer configurations, for masked autoencoder training. On ImageNet, with
such a simple change in configuration, re-designed model achieves 87.1% top-1
accuracy and outperforms SoTA models like MAE and BEiT. On language tasks,
re-designed model outperforms BERT with default setting by 1.1 points on
average, on GLUE datasets.