natural language toolkit gleu score c 20012023 nltk project s contributors mike schuster michael wayne goodman liling tan url https www nltk org for license information see license txt gleu score implementation from collections import counter from nltk util import everygrams ngrams def sentencegleureferences hypothesis minlen1 maxlen4 return corpusgleureferences hypothesis minlenminlen maxlenmaxlen def corpusgleulistofreferences hypotheses minlen1 maxlen4 sanity check assert lenlistofreferences len hypotheses the number of hypotheses and their references should be the same sum matches and maxtokenlengths over all sentences corpusnmatch 0 corpusnall 0 for references hypothesis in ziplistofreferences hypotheses hypngrams countereverygramshypothesis minlen maxlen tpfp sumhypngrams values true positives false positives hypcounts for reference in references refngrams countereverygramsreference minlen maxlen tpfn sumrefngrams values true positives false negatives overlapngrams refngrams hypngrams tp sumoverlapngrams values true positives while gleu is defined as the minimum of precision and recall we can reduce the number of division operations by one by instead finding the maximum of the denominators for the precision and recall formulae since the numerators are the same precision tp tpfp recall tp tpfn gleuscore minprecision recall tp maxtpfp tpfn nall maxtpfp tpfn if nall 0 hypcounts appendtp nall use the reference yielding the highest score if hypcounts nmatch nall maxhypcounts keylambda hc hc0 hc1 corpusnmatch nmatch corpusnall nall corner case empty corpus or empty referencesdon t divide by zero if corpusnall 0 gleuscore 0 0 else gleuscore corpusnmatch corpusnall return gleuscore natural language toolkit gleu score c 2001 2023 nltk project s contributors mike schuster michael wayne goodman liling tan url https www nltk org for license information see license txt gleu score implementation calculates the sentence level gleu google bleu score described in yonghui wu mike schuster zhifeng chen quoc v le mohammad norouzi wolfgang macherey maxim krikun yuan cao qin gao klaus macherey jeff klingner apurva shah melvin johnson xiaobing liu lukasz kaiser stephan gouws yoshikiyo kato taku kudo hideto kazawa keith stevens george kurian nishant patil wei wang cliff young jason smith jason riesa alex rudnick oriol vinyals greg corrado macduff hughes jeffrey dean 2016 google s neural machine translation system bridging the gap between human and machine translation eprint arxiv 1609 08144 https arxiv org pdf 1609 08144v2 pdf retrieved on 27 oct 2016 from wu et al 2016 the bleu score has some undesirable properties when used for single sentences as it was designed to be a corpus measure we therefore use a slightly different score for our rl experiments which we call the gleu score for the gleu score we record all sub sequences of 1 2 3 or 4 tokens in output and target sequence n grams we then compute a recall which is the ratio of the number of matching n grams to the number of total n grams in the target ground truth sequence and a precision which is the ratio of the number of matching n grams to the number of total n grams in the generated output sequence then gleu score is simply the minimum of recall and precision this gleu score s range is always between 0 no matches and 1 all match and it is symmetrical when switching output and target according to our experiments gleu score correlates quite well with the bleu metric on a corpus level but does not have its drawbacks for our per sentence reward objective note the initial implementation only allowed a single reference but now a list of references is required which is consistent with bleu_score sentence_bleu the infamous the the the example ref the cat is on the mat split hyp the the the the the the the split sentence_gleu ref hyp doctest ellipsis 0 0909 an example to evaluate normal machine translation outputs ref1 str it is a guide to action that ensures that the military will forever heed party commands split hyp1 str it is a guide to action which ensures that the military always obeys the commands of the party split hyp2 str it is to insure the troops forever hearing the activity guidebook that party direct split sentence_gleu ref1 hyp1 doctest ellipsis 0 4393 sentence_gleu ref1 hyp2 doctest ellipsis 0 1206 param references a list of reference sentences type references list list str param hypothesis a hypothesis sentence type hypothesis list str param min_len the minimum order of n gram this function should extract type min_len int param max_len the maximum order of n gram this function should extract type max_len int return the sentence level gleu score rtype float calculate a single corpus level gleu score aka system level gleu for all the hypotheses and their respective references instead of averaging the sentence level gleu scores i e macro average precision wu et al 2016 sum up the matching tokens and the max of hypothesis and reference tokens for each sentence then compute using the aggregate values from mike schuster via email for the corpus we just add up the two statistics n_match and n_all max n_all_output n_all_target for all sentences then calculate gleu_score n_match n_all so it is not just a mean of the sentence gleu scores in our case longer sentences count more which i think makes sense as they are more difficult to translate hyp1 it is a guide to action which ensures that the military always obeys the commands of the party ref1a it is a guide to action that ensures that the military will forever heed party commands ref1b it is the guiding principle which guarantees the military forces always being under the command of the party ref1c it is the practical guide for the army always to heed the directions of the party hyp2 he read the book because he was interested in world history ref2a he was interested in world history because he read the book list_of_references ref1a ref1b ref1c ref2a hypotheses hyp1 hyp2 corpus_gleu list_of_references hypotheses doctest ellipsis 0 5673 the example below show that corpus_gleu is different from averaging sentence_gleu for hypotheses score1 sentence_gleu ref1a hyp1 score2 sentence_gleu ref2a hyp2 score1 score2 2 doctest ellipsis 0 6144 param list_of_references a list of reference sentences w r t hypotheses type list_of_references list list list str param hypotheses a list of hypothesis sentences type hypotheses list list str param min_len the minimum order of n gram this function should extract type min_len int param max_len the maximum order of n gram this function should extract type max_len int return the corpus level gleu score rtype float sanity check sum matches and max token lengths over all sentences true positives false positives true positives false negatives true positives while gleu is defined as the minimum of precision and recall we can reduce the number of division operations by one by instead finding the maximum of the denominators for the precision and recall formulae since the numerators are the same precision tp tpfp recall tp tpfn gleu_score min precision recall tp max tpfp tpfn use the reference yielding the highest score corner case empty corpus or empty references don t divide by zero
from collections import Counter

from nltk.util import everygrams, ngrams


def sentence_gleu(references, hypothesis, min_len=1, max_len=4):
    
    return corpus_gleu([references], [hypothesis], min_len=min_len, max_len=max_len)


def corpus_gleu(list_of_references, hypotheses, min_len=1, max_len=4):
    
    
    assert len(list_of_references) == len(
        hypotheses
    ), "The number of hypotheses and their reference(s) should be the same"

    
    corpus_n_match = 0
    corpus_n_all = 0

    for references, hypothesis in zip(list_of_references, hypotheses):
        hyp_ngrams = Counter(everygrams(hypothesis, min_len, max_len))
        tpfp = sum(hyp_ngrams.values())  

        hyp_counts = []
        for reference in references:
            ref_ngrams = Counter(everygrams(reference, min_len, max_len))
            tpfn = sum(ref_ngrams.values())  

            overlap_ngrams = ref_ngrams & hyp_ngrams
            tp = sum(overlap_ngrams.values())  

            
            
            
            
            
            
            
            n_all = max(tpfp, tpfn)

            if n_all > 0:
                hyp_counts.append((tp, n_all))

        
        if hyp_counts:
            n_match, n_all = max(hyp_counts, key=lambda hc: hc[0] / hc[1])
            corpus_n_match += n_match
            corpus_n_all += n_all

    
    if corpus_n_all == 0:
        gleu_score = 0.0
    else:
        gleu_score = corpus_n_match / corpus_n_all

    return gleu_score