**Stop Using BLEU** BLEU, a metric extensively utilized for decades, often diverges from neural-based and reference-free metrics, a phenomenon also observed in previous studies (Xu et al., 2023; Freitag et al., 2023). For instance, WMT competition winners often exhibit superior performance according to BLEU (or COMET-22), yet this is not corroborated by reference-free models. A case in point is the WMT winners scoring an exceptionally high 64.14 BLEU in cs\(\rightarrow\)en translations, significantly outperforming other models by 20 BLEU points. However, reference-free evaluations suggest these translations are inferior to those generated by our models and GPT-4. We hypothesize that this discrepancy may arise from WMT models being trained on domain-specific data closely related to the WMT test set, leading to high lexical matches but lacking semantic depth as evaluated by neural-based metrics. While BLEU scores are effective for assessing basic functionality in weaker models, their utility diminishes with advanced translation models capable of generating diverse translations. In such contexts, relying solely on BLEU for evaluation appears increasingly outdated.