Large decoder-only language models (LMs)  can be largely improved in terms of perplexity by retrieval (e.g.,Retro) , but its impact on text generation quality and downstream task accuracy is unclear.
Thus, it is still an open question:shall we pretrain large autoregressive LMs with retrieval?To answer it, we perform a comprehensive study on ascalable pretrainedretrieval-augmented LM (i.e.,Retro)  compared with standard GPT and retrieval-augmented GPT incorporated at fine-tuning or inference stages.
We first provide the recipe to reproduceRetroup to 9.5B parameters while retrieving a text corpus with 330B tokens.
Based on that, we have the following novel findings:i) Retrooutperforms GPT on text generation with much less degeneration (i.e., repetition) , moderately higher factual accuracy, and slightly lower toxicity with a nontoxic retrieval database.ii) On the LM Evaluation Harness benchmark,Retrolargely outperforms GPT on knowledge-intensive tasks, but is on par with GPT on other tasks.
Furthermore, we introduce a simple variant of the model,Retro++, which largely improves open-domain QA results of originalRetro(e.g., EM scoreon Natural Question)  and significantly outperforms retrieval-augmented GPT in both finetuning and zero-shot evaluation settings.
Our findings highlight the promising direction of pretraining autoregressive LMs with retrieval as future foundation models.
We release our code and model at:https://github.com/NVIDIA/Megatron-LM/blob/main/tools/retro/README.md.