@inproceedings{
barbero2024transformers,
title={Transformers need glasses! Information over-squashing in language tasks},
author={Federico Barbero and Andrea Banino and Steven Kapturowski and Dharshan Kumaran and Jo{\~a}o Guilherme Madeira Ara{\'u}jo and Alex Vitvitskyi and Razvan Pascanu and Petar Veli{\v{c}}kovi{\'c}},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=93HCE8vTye}
}
We study how information propagates in decoder-only Transformers, which are the architectural foundation of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis---specifically, we analyse the representations of the last token in the final layer of the Transformer, as this is the representation used for next-token prediction. Our analysis reveals a representational collapse phenomenon: we prove that certain distinct pairs of inputs to the Transformer can yield arbitrarily close representations in the final token. This effect is exacerbated by the low-precision floating-point formats frequently used in modern LLMs. As a result, the model is provably unable to respond to these sequences in different ways---leading to errors in, e.g., tasks involving counting or copying. Further, we show that decoder-only Transformer language models can lose sensitivity to specific tokens in the input, which relates to the well-known phenomenon of over-squashing in graph neural networks. We provide empirical evidence supporting our claims on contemporary LLMs. Our theory points to simple solutions towards ameliorating these issues.
Transformers need glasses! Information over-squashing in language tasks
Federico Barbero Andrea Banino Steven Kapturowski Dharshan Kumaran João Guilherme Madeira Araújo Alex Vitvitskyi Razvan Pascanu Petar Veličković