Character Tokenization.Tokenization can also be performed on a character level or based on UTF-8 bytes. However, this results in an increased sequence length, which becomes computationally expensive in the transformer architecture, the current predominated architecture for LLMs due to the quadratic complexity of the self-attention layer in the sequence length (Vaswani et al., 2017). Though, several approaches have been proposed to address this limitation (Gao et al., 2020; Tay et al., 2021; Xue et al., 2022; Clark et al., 2022; Yu et al., 2023).