Tokenizers in Encoder ModelsMost research on tokenization has been conducted on encoder models. Rust et al. (2021) investigated whether the tokenizer choice impacts the downstream performance of multi- and monolingual BERT (Devlin et al., 2018) models. Zhang et al. (2022) showed that better machine translation performance is often obtained when languages are equally sampled during the tokenizer training. Toraman et al. (2023) trained several medium-sized language models for Turkish and suggested that different subword tokenizers perform roughly equivalent, whereas word- and character-level tokenizers perform drastically worse on downstream tasks. Finally, (Chirkova and Troshin, 2022) analyzed the effect of employing different tokenizations on code-related tasks and demonstrated that carefully configured tokenizers could reduce average sequence length up to 40% or allow for small downstream performance improvements by up to 2% at a lower compression rate.