Tokenized BLEU considered harmful - Discussion on community-based process
I do not consider as a sufficient solution switching this library's default metric from BLEU to the wrapper around SacreBLEU. 

As currently implemented, the wrapper allows end users to toggle SacreBLEU options, but doesn't pass along the SacreBLEU signature. As @mjpost showed in [Post18](https://www.aclweb.org/anthology/W18-6319.pdf), it's simply not credible to assume that people will stick to the defaults, therefore, the signature is necessary to be explicit about what options were used. 

In addition to the `v13a` or `intl` options for the SacreBLEU `tokenize` argument, which was pointed out earlier, papers frequently differ on whether they lowercase text before scoring (`lowercase`) and the smoothing method used (`smooth_method`). BLEU scores can differ substantially (over 1 BLEU) just by changing these options. 

Losing the SacreBLEU signature is a regression in reproducibility and clarity.

(Perhaps this should belong in a separate issue?)
https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.  
Tokenized BLEU considered harmful - Discussion on community-based process 
 https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.  

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .  

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.  

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.   
 I do not consider as a sufficient solution switching this library's default metric from BLEU to the wrapper around SacreBLEU. 

As currently implemented, the wrapper allows end users to toggle SacreBLEU options, but doesn't pass along the SacreBLEU signature. As @mjpost showed in [Post18](https://www.aclweb.org/anthology/W18-6319.pdf), it's simply not credible to assume that people will stick to the defaults, therefore, the signature is necessary to be explicit about what options were used. 

In addition to the `v13a` or `intl` options for the SacreBLEU `tokenize` argument, which was pointed out earlier, papers frequently differ on whether they lowercase text before scoring (`lowercase`) and the smoothing method used (`smooth_method`). BLEU scores can differ substantially (over 1 BLEU) just by changing these options. 

Losing the SacreBLEU signature is a regression in reproducibility and clarity.

(Perhaps this should belong in a separate issue?)