Tokenized BLEU considered harmful - Discussion on community-based process
Thanks for sharing your thoughts. This is a very important discussion.

Also one of the first items on our mid-term roadmap (we will try to clean it and share it soon) is to introduce mechanisms to get high-quality traceability and reproducibility for all the processes related to the library.

So having the signature for `sacrebleu` is really important!

Regarding BLEU, I guess we can just remove it from the canonical metrics included in the repo itself (it won't prevent people to add it as "user-metrics" but at least we won't be promoting it).

On a more general note (definitely too large for the scope of this issue) we are wondering, with @srush in particular, how we could handle the selection of metrics/datasets with the most community-based and bottom-up approach possible. If you have opinions on this, please share!
https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.  
Tokenized BLEU considered harmful - Discussion on community-based process 
 https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.   
 Thanks for sharing your thoughts. This is a very important discussion.

Also one of the first items on our mid-term roadmap (we will try to clean it and share it soon) is to introduce mechanisms to get high-quality traceability and reproducibility for all the processes related to the library.

So having the signature for `sacrebleu` is really important!

Regarding BLEU, I guess we can just remove it from the canonical metrics included in the repo itself (it won't prevent people to add it as "user-metrics" but at least we won't be promoting it).

On a more general note (definitely too large for the scope of this issue) we are wondering, with @srush in particular, how we could handle the selection of metrics/datasets with the most community-based and bottom-up approach possible. If you have opinions on this, please share!