Long-context language models (LCLMs)  have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
Leveraging LCLMs’ ability to natively ingest and process entire corpora of information offers numerous advantages.
It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system.
To assess this paradigm shift, we introduceLOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs’ performance on in-context retrieval and reasoning.
Our findings reveal LCLMs’ surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks.
Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow.
Overall,LOFTprovides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.111TheLOFTbenchmark is available athttps://github.com/google-deepmind/loft.Figure 1:An overview of theLOFTbenchmark, made of six tasks which measure LCLMs’ ability to do in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens.
We compare the performance of LCLMs against specialized models (e.g.,CLIP for visual retrieval) , which often rely on complex task-specific fine-tuning or pipelining.
Unlike specialized models, we show how LCLMs can simplify various tasks through Corpus-in-Context Prompting (Section3) .