We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens.
With a 2 trillion token database, our Retrieval-Enhanced Transformer (Retro)  obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25fewer parameters.
After fine-tuning,Retroperformance translates to downstream knowledge-intensive tasks such as question answering.Retrocombines a frozenBertretriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training.
We typically trainRetrofrom scratch, yet can also rapidlyRetrofit pre-trained transformers with retrieval and still achieve good performance.
Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.