@inproceedings{
alabdulmohsin2024fractal,
title={Fractal Patterns May Illuminate the Success of Next-Token Prediction},
author={Ibrahim Alabdulmohsin and Vinh Q. Tran and Mostafa Dehghani},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=clAFYReaYE}
}
We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not  formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately 0.7.
Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how  next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust.
Finally, we demonstrate that the tiny variations  in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.
Fractal Patterns May Illuminate the Success of Next-Token Prediction
Ibrahim Alabdulmohsin Vinh Q. Tran Mostafa Dehghani