While many different tokenization approaches have been proposed, ranging from character-based to word-based methods, the potential impact of different tokenizers is underexplored w.r.t. LLMs, especially in the context of multilingual LLMs. Recent work proposed by Petrov et al. (2023) demonstrates that carelessly designed tokenizers applied to the training of multilingual LLMs result in severe inequalities and limitations across languages. Text passages translated into different languages resulted in tokenized sequences that differ in length up to a factor of 15, affecting inference costs and latency during inference. Furthermore, it is known that the learning of long-range dependencies Vaswani et al. (2017), is an essential property for effectively learning transformer-based LLMs. Given a fixed sequence length, learning to relate words far apart in the input text is impossible for languages whose text is excessively fragmented by the tokenizer.