Are-gram language models still relevant in this era of neural large language models (LLMs) ?
Our answer isyes, and we show their values in both text analysis and improving neural LLMs.
Yet this necessitates modernizing-gram models in two aspects.
First, we train them at the same data scale as neural LLMs –1.4 trillion tokens.
This is the largest-gram model ever built.
Second, existing-gram models use smallwhich hinders their performance; we instead allowto be arbitrarily large, by introducing a new-gram LMwith backoff.
Instead of pre-computing-gram count tables (which would be very expensive) , we develop an engine namedinfini-gram– powered by suffix arrays – that can compute-gram (as well as-gram with arbitrary)  probabilities withmillisecond-level latency.
The-gram framework and infini-gram engine enable us to conduct many novel and interesting analyses of human-written and machine-generated text:
we find that the-gram LM has fairly high accuracy for next-token prediction (47%) , and can complement neural LLMs to greatly reduce their language modeling perplexities.
When analyzing machine-generated text, we also observe irregularities in the machine–-gram agreement level with respect to the suffix length, which indicates deficiencies in neural LLM pretraining and the positional embeddings of Transformers.
We open-source our infini-gram engine in the hopes of enabling more study on how to best use verbatim information retrieved from large text corpora.