<jupyter_start><jupyter_text>GLEU: Google-BLEU*NLP evaluation metric used in Machine Translation tasks**Suitable for measuring sentence level similarity**Range: 0 (no match) to 1 (exact match)*  1. Libraries*Install and import necessary libraries*<jupyter_code>import nltk
import nltk.translate.gleu_score as gleu

import numpy
import os

try:
  nltk.data.find('tokenizers/punkt')
except LookupError:
  nltk.download('punkt')<jupyter_output><empty_output><jupyter_text>2. Dataset*Array of words: candidate and reference sentences split into words*<jupyter_code>hyp = str('she read the book because she was interested in world history').split()
ref_a = str('she read the book because she was interested in world history').split()
ref_b = str('she was interested in world history because she read the book').split()<jupyter_output><empty_output><jupyter_text>3. *Sentence* score calculation*Compares 1 hypothesis (candidate or source sentence) with 1+ reference sentences, returning the highest score when compared to multiple reference sentences.*<jupyter_code>score_ref_a = gleu.sentence_gleu([ref_a], hyp)
print("Hyp and ref_a are the same: {}".format(score_ref_a))
score_ref_b = gleu.sentence_gleu([ref_b], hyp)
print("Hyp and ref_b are different: {}".format(score_ref_b))
score_ref_ab = gleu.sentence_gleu([ref_a, ref_b], hyp)
print("Hyp vs multiple refs: {}".format(score_ref_ab))<jupyter_output>Hyp and ref_a are the same: 1.0
Hyp and ref_b are different: 0.7894736842105263
Hyp vs multiple refs: 1.0<jupyter_text>4. *Corpus* score calculation*Compares 1 candidate document with multiple sentence and 1+ reference documents also with multiple sentences.*<jupyter_code>score_ref_a = gleu.corpus_gleu([[ref_a]], [hyp])
print("1 document with 1 reference sentence: {}".format(score_ref_a))
score_ref_a = gleu.corpus_gleu([[ref_a, ref_b]], [hyp])
print("1 document with 2 reference sentences: {}".format(score_ref_a))
score_ref_a = gleu.corpus_gleu([[ref_a], [ref_b]], [hyp, hyp])
print("2 documents with 1 reference sentence each: {}".format(score_ref_a))<jupyter_output>1 document with 1 reference sentence: 1.0
1 document with 2 reference sentences: 1.0
2 documents with 1 reference sentence each: 0.8947368421052632<jupyter_text>5. GLEU-$n$*In GLEU-$n$, $n$-gram scores can be obtained in both **sentence** and **corpus** calculations and they're indicated by **min_len** and **max_len** parameters.** *min_len*: minimum order of $n$-grams it should extract* *max_len*: maximum order of $n$-grams it should extract<jupyter_code>score_1to4grams = gleu.sentence_gleu([ref_b], hyp, min_len=1, max_len=4)
score_1to2grams = gleu.sentence_gleu([ref_b], hyp, min_len=1, max_len=2)
print("1 to 4 grams: {}".format(score_1to4grams))
print("1 to 2 grams: {}".format(score_1to2grams))<jupyter_output>1 to 4 grams: 0.7894736842105263
1 to 2 grams: 0.9523809523809523
{'fraction_non_alphanumeric': 0.11752360965372508, 'fraction_numerical': 0.0416229450856943, 'mean_word_length': 7.0790960451977405, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '326645', 'license': 'MIT', 'n_tokens_mistral': 1052, 'n_tokens_neox': 957, 'n_words': 289, 'path': 'notebooks/gleu.ipynb', 'repo_name': 'fighting41love/NLPMetrics', 'size': '2859'}