It has been shown that properly leveraging information from retrieval memory could improve the performance of large pre-trained language model. To build a more accurate language model, [1]} propose to incorporate a soft memory module into the system. Specifically, an index is built by caching the hidden states of the training corpus. Then, the language model accesses the index via k-NN search and displays a greatly improved performance.
As another example, [2]} propose a new paradigm that applies retrieval-augmented technique into the pre-training of generative language model. During learning, they train a neural selector that dynamically samples a relevant text to guide the reconstruction of a corrupted input sequence. In this way, the pre-trained model delivers better results by explicitly grounding on the retrieval memory. [3]} combine language model pre-training with a paraphrasing approach. During learning, an input sequence to the model is first corrupted. In the meantime, a set of multi-lingual texts are retrieved based on which the model learns to reconstruct the original input sequence.
Recently, [4]} propose RETRO, a large pre-trained language model enhanced with retrieved documents, and obtained comparable performances with GPT-3 using 25\(\times \)  fewer parameters.
