<jupyter_start><jupyter_text>BLEU: BiLingual Evaluation Understudy*NLP evaluation metric used in Machine Translation tasks**Suitable for measuring corpus level similarity**$n$-gram comparison between words in candidate sentence and reference sentences**Range: 0 (no match) to 1 (exact match)*  1. Libraries*Install and import necessary libraries*<jupyter_code>import nltk
import nltk.translate.bleu_score as bleu

import math
import numpy
import os

try:
  nltk.data.find('tokenizers/punkt')
except LookupError:
  nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.<jupyter_text>2. Dataset*Array of words: candidate and reference sentences split into words*<jupyter_code>hyp = str('she read the book because she was interested in world history').split()
ref_a = str('she read the book because she was interested in world history').split()
ref_b = str('she was interested in world history because she read the book').split()<jupyter_output><empty_output><jupyter_text>3. *Sentence* score calculation*Compares 1 hypothesis (candidate or source sentence) with 1+ reference sentences, returning the highest score when compared to multiple reference sentences.*<jupyter_code>score_ref_a = bleu.sentence_bleu([ref_a], hyp)
print("Hyp and ref_a are the same: {}".format(score_ref_a))
score_ref_b = bleu.sentence_bleu([ref_b], hyp)
print("Hyp and ref_b are different: {}".format(score_ref_b))
score_ref_ab = bleu.sentence_bleu([ref_a, ref_b], hyp)
print("Hyp vs multiple refs: {}".format(score_ref_ab))<jupyter_output>Hyp and ref_a are the same: 1.0
Hyp and ref_b are different: 0.7400828044922853
Hyp vs multiple refs: 1.0<jupyter_text>4. *Corpus* score calculation*Compares 1 candidate document with multiple sentence and 1+ reference documents also with multiple sentences.** Different than averaging BLEU scores of each sentence, it calculates the score by *"summing the numerators and denominators for each hypothesis-reference(s) pairs before the division"*<jupyter_code>score_ref_a = bleu.corpus_bleu([[ref_a]], [hyp])
print("1 document with 1 reference sentence: {}".format(score_ref_a))
score_ref_a = bleu.corpus_bleu([[ref_a, ref_b]], [hyp])
print("1 document with 2 reference sentences: {}".format(score_ref_a))
score_ref_a = bleu.corpus_bleu([[ref_a], [ref_b]], [hyp, hyp])
print("2 documents with 1 reference sentence each: {}".format(score_ref_a))<jupyter_output>1 document with 1 reference sentence: 1.0
1 document with 2 reference sentences: 1.0
2 documents with 1 reference sentence each: 0.8778107713916036<jupyter_text>5. BLEU-$n$*In BLEU-$n$, $n$-gram scores can be obtained in both **sentence** and **corpus** calculations and they're indicated by the **weights** parameter.** *weights*: length 4, where each index contains a weight corresponding to its respective $n$-gram.* $n$-gram with $n \in \{1, 2, 3, 4\}$* $\textit{weights}=(W_{N=1}, W_{N=2}, W_{N=3}, W_{N=4})$<jupyter_code>score_1gram = bleu.sentence_bleu([ref_b], hyp, weights=(1,0,0,0))
score_2gram = bleu.sentence_bleu([ref_b], hyp, weights=(0,1,0,0))
score_3gram = bleu.sentence_bleu([ref_b], hyp, weights=(0,0,1,0))
score_4gram = bleu.sentence_bleu([ref_b], hyp, weights=(0,0,0,1))
print("N-grams: 1-{}, 2-{}, 3-{}, 4-{}".format(score_1gram, score_2gram, score_3gram, score_4gram))<jupyter_output>N-grams: 1-1.0, 2-0.9, 3-0.6666666666666666, 4-0.5<jupyter_text>* Cumulative N-grams: *by default, the score is calculatedby considering all $N$-grams equally in a geometric mean*<jupyter_code>score_ngram1 = bleu.sentence_bleu([ref_b], hyp)
score_ngram = bleu.sentence_bleu([ref_b], hyp, weights=(0.25,0.25,0.25,0.25))
score_ngram_geo = (11/11*9/10*6/9*4/8)**0.25
print("N-grams: {} = {} = ".format(score_ngram1, score_ngram, score_ngram_geo))<jupyter_output>N-grams: 0.7400828044922853 = 0.7400828044922853<jupyter_text>Further testing<jupyter_code>hyp = str('she read the book because she was interested in world history').split()
ref_a = str('she was interested in world history because she read the book').split()
hyp_b = str('the book she read was about modern civilizations.').split()
ref_b = str('the book she read was about modern civilizations.').split()

score_a = bleu.sentence_bleu([ref_a], hyp)
score_b = bleu.sentence_bleu([ref_b], hyp_b)
score_ab = bleu.sentence_bleu([ref_a], hyp_b)
score_ba = bleu.sentence_bleu([ref_b], hyp)
score_ref_a = bleu.corpus_bleu([[ref_a], [ref_b]], [hyp, hyp_b])
average = (score_a+score_b)/2
corpus = math.pow((11+8)/19 * (9+7)/(17) * (6+6)/(9+6) * (4+5)/(8+5), 1/4)
print("Sent: {}, {}, {}, {} - Corpus {}, {}, {}".format(score_a, score_b, score_ab, score_ba, score_ref_a, average, corpus))<jupyter_output>Sent: 0.7400828044922853, 1.0, 0.44677880536150705, 0.5491004867761125 - Corpus 0.8496988908521796, 0.8700414022461427, 0.8496988908521795
{'fraction_non_alphanumeric': 0.14038699053108275, 'fraction_numerical': 0.06504734458624949, 'mean_word_length': 6.712698412698413, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '393906', 'license': 'MIT', 'n_tokens_mistral': 1932, 'n_tokens_neox': 1723, 'n_words': 457, 'path': 'notebooks/bleu.ipynb', 'repo_name': 'fighting41love/NLPMetrics', 'size': '4858'}