The Impact of Positional Encoding on Length Generalization in Transformers
Paper
•
2305.19466
•
Published
•
2
NoPE
-
-
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Paper • 2305.13571 • Published • 2 -
Position Prediction as an Effective Pretraining Strategy
Paper • 2207.07611 • Published • 1 -
Transformer Language Models without Positional Encodings Still Learn Positional Information
Paper • 2203.16634 • Published • 5 -
Language Modeling with Deep Transformers
Paper • 1905.04226 • Published • 1 -
2-D SSM: A General Spatial Layer for Visual Transformers
Paper • 2306.06635 • Published • 1 -
Transformers Can Do Arithmetic with the Right Embeddings
Paper • 2405.17399 • Published • 51