Tokenized BLEU considered harmful - Discussion on community-based process
@srush @thomwolf 

ModelFront could provide (automated, "QE-based") evaluation for all the pretrained translation models you host.  Not bottom-up and not valid for claiming SoTA, but independent, practical for builders and not top-down.

For that I would also suggest some diverse benchmarks (so split it out into datasets with only user-generated data, or only constants, or only UI strings, or only READMEs) which tease out known trade-offs.  Even hypothetical magic eval is limited if we always reduce it to a single number.

Realistically people want to know how a model compares to an API like Google Translate, Microsoft Translator, DeepL or Yandex (especially for a language pair like EN:RU, or for the many languages that only Yandex supports), and that could be done too.

https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.  
Tokenized BLEU considered harmful - Discussion on community-based process 
 https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.   
 @srush @thomwolf 

ModelFront could provide (automated, "QE-based") evaluation for all the pretrained translation models you host.  Not bottom-up and not valid for claiming SoTA, but independent, practical for builders and not top-down.

For that I would also suggest some diverse benchmarks (so split it out into datasets with only user-generated data, or only constants, or only UI strings, or only READMEs) which tease out known trade-offs.  Even hypothetical magic eval is limited if we always reduce it to a single number.

Realistically people want to know how a model compares to an API like Google Translate, Microsoft Translator, DeepL or Yandex (especially for a language pair like EN:RU, or for the many languages that only Yandex supports), and that could be done too.
