<jupyter_start><jupyter_text>Evaluating Generated Text with the BLEU Score* BLEU(Bilingual Evaluation Understudy) is a score for computing a candidate translation of text to one or more reference translations.  1. Sentence BLEU Score<jupyter_code>from nltk.translate.bleu_score import sentence_bleu
reference = [['this','is','a','test'],['this','is','test']]
candidate = ['this','is','a','test']
score = sentence_bleu(reference,candidate)
print(score)<jupyter_output>1.0<jupyter_text>2. Corpus BLEU Score* calculating blew score for multiple sentences such as a paragraph or a document.<jupyter_code># two references for one document
from nltk.translate.bleu_score import corpus_bleu
references = [[['this','is','a','test'],['this','is','test']]]
candidates = [['this','is','a','test']]
score = corpus_bleu(references,candidates)
print(score)<jupyter_output>1.0<jupyter_text>3. Individual n-gram Scores<jupyter_code># 1-gram individual BLEU
from nltk.translate.bleu_score import sentence_bleu
reference = [['this','is','small','test']]
candidate = ['this','is','a','test']
score = sentence_bleu(reference,candidate,weights=(1,0,0,0))
print(score)
# n-gram individual BLEU
from nltk.translate.bleu_score import sentence_bleu
reference = [['this','is','a','test']]
candidate = ['this','is','a','test']
print('Individual 1-gram: %f' %sentence_bleu(reference,candidate,weights=(1,0,0,0)))
print('Individual 2-gram: %f' %sentence_bleu(reference,candidate,weights=(0,1,0,0)))
print('Individual 3-gram: %f' %sentence_bleu(reference,candidate,weights=(0,0,1,0)))
print('Individual 4-gram: %f' %sentence_bleu(reference,candidate,weights=(0,0,0,1)))<jupyter_output>Individual 1-gram: 1.000000
Individual 2-gram: 1.000000
Individual 3-gram: 1.000000
Individual 4-gram: 1.000000<jupyter_text>4. Cumulative n-gram Scores<jupyter_code># 4-gram cumulative BLEU
from nltk.translate.bleu_score import sentence_bleu
reference = [['this','is','small','test']]
candidate = ['this','is','a','test']
score = sentence_bleu(reference,candidate,weights=(0.25,0.25,0.25,0.25))
print(score)
# cumulative BLEU scores
from nltk.translate.bleu_score import sentence_bleu
reference = [['this','is','small','test']]
candidate = ['this','is','a','test']
print('Cumulative 1-gram: %f' %sentence_bleu(reference,candidate,weights=(1,0,0,0)))
print('Cumulative 2-gram: %f' %sentence_bleu(reference,candidate,weights=(0.5,0.5,0,0)))
print('Cumulative 3-gram: %f' %sentence_bleu(reference,candidate,weights=(0.33,0.33,0.33,0)))
print('Cumulative 4-gram: %f' %sentence_bleu(reference,candidate,weights=(0.25,0.25,0.25,0.25)))<jupyter_output>Cumulative 1-gram: 0.750000
Cumulative 2-gram: 0.500000
Cumulative 3-gram: 0.000000
Cumulative 4-gram: 0.000000<jupyter_text>5. Worked Example<jupyter_code># perfect match
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['the','quick','brown','fox','jumped','over','the','lazy','dog']
score = sentence_bleu(reference,candidate)
print(score)
# one word different
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['the','fast','brown','fox','jumped','over','the','lazy','dog']
score = sentence_bleu(reference,candidate)
print(score)
# two words different
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['the','fast','brown','fox','jumped','over','the','sleepy','dog']
score = sentence_bleu(reference,candidate)
print(score)
# all words different
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['a','b','c','d','e','f','g','h','i']
score = sentence_bleu(reference,candidate)
print(score)
# shorter candidate
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['the','quick','brown','fox','jumped','over','the']
score = sentence_bleu(reference,candidate)
print(score)
# longer candidate
from nltk.translate.bleu_score import sentence_bleu
reference = [['the','quick','brown','fox','jumped','over','the','lazy','dog']]
candidate = ['the','quick','brown','fox','jumped','over','the','lazy','dog','from','space']
score = sentence_bleu(reference,candidate)
print(score)<jupyter_output>0.7860753021519787
{'fraction_non_alphanumeric': 0.20295368091295593, 'fraction_numerical': 0.03669724770642202, 'mean_word_length': 10.671018276762402, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '47711', 'license': 'MIT', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1550, 'n_words': 265, 'path': 'Evaluating Generated Text with the BLEU Score.ipynb', 'repo_name': 'kuluruvineeth/Deep-Learning-for-NLP', 'size': '4469'}