@inproceedings{cognetta-etal-2024-two,
    title = "Two Counterexamples to Tokenization and the Noiseless Channel",
    author = "Cognetta, Marco  and
      Zouhar, Vil{\'e}m  and
      Moon, Sangwhan  and
      Okazaki, Naoaki",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1469",
    pages = "16897--16906",
    abstract = "In Tokenization and the Noiseless Channel (Zouhar et al., 2023), R{\'e}nyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R{\'e}nyi efficiency of the unigram distribution should be chosen. The R{\'e}nyi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that R{\'e}nyi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase R{\'e}nyi efficiency while decreasing the downstream model performance. These counterexamples expose cases where R{\'e}nyi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.",
}

In Tokenization and the Noiseless Channel (Zouhar et al., 2023), R{\'e}nyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R{\'e}nyi efficiency of the unigram distribution should be chosen. The R{\'e}nyi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that R{\'e}nyi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase R{\'e}nyi efficiency while decreasing the downstream model performance. These counterexamples expose cases where R{\'e}nyi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.
Cognetta, Marco Zouhar, Vil{\'e}m Moon, Sangwhan Okazaki, Naoaki
Two Counterexamples to Tokenization and the Noiseless Channel