natural language toolkit nist score c 20012023 nltk project s contributors url https www nltk org for license information see license txt nist score implementation import fractions import math from collections import counter from nltk util import ngrams def sentencenistreferences hypothesis n5 return corpusnistreferences hypothesis n def corpusnistlistofreferences hypotheses n5 before proceeding to compute nist perform sanity checks assert lenlistofreferences len hypotheses the number of hypotheses and their references should be the same collect the ngram coounts from the reference sentences ngramfreq counter totalreferencewords 0 for references in listofreferences for each source sent there s a list of reference sents for reference in references for each order of ngram count the ngram occurrences for i in range1 n 1 ngramfreq updatengramsreference i totalreferencewords lenreference compute the information weights based on the reference sentences eqn 2 in doddington 2002 infow1 wn log2 of occurrences of w1 wn1 of occurrences of w1 wn informationweights for ngram in ngramfreq w1 wn mgram ngram 1 w1 wn1 from https github commosessmtmosesdecoderblobmasterscriptsgenericmtevalv13a pll546 it s computed as such denominator ngramfreqmgram if mgram and mgram in ngramfreq else denominator totalreferencewords informationweightsngram 1 math logngramfreqngramdenominator math log2 mathematically it s equivalent to the our implementation if mgram and mgram in ngramfreq numerator ngramfreqmgram else numerator totalreferencewords informationweightsngram math lognumerator ngramfreqngram 2 microaverage nistprecisionnumeratorperngram counter nistprecisiondenominatorperngram counter lref lsys 0 0 for each order of ngram for i in range1 n 1 iterate through each hypothesis and their corresponding references for references hypothesis in ziplistofreferences hypotheses hyplen lenhypothesis find reference with the best nist score nistscoreperref for reference in references reflen lenreference counter of ngrams in hypothesis hypngrams counterngramshypothesis i if lenhypothesis i else counter refngrams counterngramsreference i if lenreference i else counter ngramoverlaps hypngrams refngrams precision part of the score in eqn 3 numerator sum informationweightsngram count for ngram count in ngramoverlaps items denominator sumhypngrams values precision 0 if denominator 0 else numerator denominator nistscoreperref append precision numerator denominator reflen best reference precision numerator denominator reflen maxnistscoreperref nistprecisionnumeratorperngrami numerator nistprecisiondenominatorperngrami denominator lref reflen lsys hyplen final nist microaverage mean aggregation nistprecision 0 for i in nistprecisionnumeratorperngram precision nistprecisionnumeratorperngrami nistprecisiondenominatorperngrami nistprecision precision eqn 3 in doddington2002 return nistprecision nistlengthpenaltylref lsys def nistlengthpenaltyreflen hyplen ratio hyplen reflen if 0 ratio 1 ratiox scorex 1 5 0 5 beta math logscorex math logratiox 2 return math expbeta math logratio 2 else ratio 0 or ratio 1 return maxminratio 1 0 0 0 natural language toolkit nist score c 2001 2023 nltk project s contributors url https www nltk org for license information see license txt nist score implementation calculate nist score from george doddington 2002 automatic evaluation of machine translation quality using n gram co occurrence statistics proceedings of hlt morgan kaufmann publishers inc https dl acm org citation cfm id 1289189 1289273 darpa commissioned nist to develop an mt evaluation facility based on the bleu score the official script used by nist to compute bleu and nist score is mteval 14 pl the main differences are bleu uses geometric mean of the ngram overlaps nist uses arithmetic mean nist has a different brevity penalty nist score from mteval 14 pl has a self contained tokenizer note the mteval 14 pl includes a smoothing function for bleu score that is not used in the nist score computation hypothesis1 it is a guide to action which ensures that the military always obeys the commands of the party hypothesis2 it is to insure the troops forever hearing the activity guidebook that party direct reference1 it is a guide to action that ensures that the military will forever heed party commands reference2 it is the guiding principle which guarantees the military forces always being under the command of the party reference3 it is the practical guide for the army always to heed the directions of the party sentence_nist reference1 reference2 reference3 hypothesis1 doctest ellipsis 3 3709 sentence_nist reference1 reference2 reference3 hypothesis2 doctest ellipsis 1 4619 param references reference sentences type references list list str param hypothesis a hypothesis sentence type hypothesis list str param n highest n gram order type n int calculate a single corpus level nist score aka system level bleu for all the hypotheses and their respective references param references a corpus of lists of reference sentences w r t hypotheses type references list list list str param hypotheses a list of hypothesis sentences type hypotheses list list str param n highest n gram order type n int before proceeding to compute nist perform sanity checks collect the ngram coounts from the reference sentences for each source sent there s a list of reference sents for each order of ngram count the ngram occurrences compute the information weights based on the reference sentences eqn 2 in doddington 2002 info w_1 w_n log_2 of occurrences of w_1 w_n 1 of occurrences of w_1 w_n w_1 w_n w_1 w_n 1 from https github com moses smt mosesdecoder blob master scripts generic mteval v13a pl l546 it s computed as such denominator ngram_freq _mgram if _mgram and _mgram in ngram_freq else denominator total_reference_words information_weights _ngram 1 math log ngram_freq _ngram denominator math log 2 mathematically it s equivalent to the our implementation micro average for each order of ngram iterate through each hypothesis and their corresponding references find reference with the best nist score counter of ngrams in hypothesis precision part of the score in eqn 3 best reference final nist micro average mean aggregation eqn 3 in doddington 2002 calculates the nist length penalty from eq 3 in doddington 2002 penalty exp beta log min len hyp len ref 1 0 where beta is chosen to make the brevity penalty factor 0 5 when the no of words in the system output hyp is 2 3 of the average no of words in the reference translation ref the nist penalty is different from bleu s such that it minimize the impact of the score of small variations in the length of a translation see fig 4 in doddington 2002 ratio 0 or ratio 1
import fractions
import math
from collections import Counter

from nltk.util import ngrams


def sentence_nist(references, hypothesis, n=5):
    
    return corpus_nist([references], [hypothesis], n)


def corpus_nist(list_of_references, hypotheses, n=5):
    
    
    assert len(list_of_references) == len(
        hypotheses
    ), "The number of hypotheses and their reference(s) should be the same"

    
    ngram_freq = Counter()
    total_reference_words = 0
    for (
        references
    ) in list_of_references:  
        for reference in references:
            
            for i in range(1, n + 1):
                ngram_freq.update(ngrams(reference, i))
            total_reference_words += len(reference)

    
    
    
    information_weights = {}
    for _ngram in ngram_freq:  
        _mgram = _ngram[:-1]  
        
        
        
        
        
        
        if _mgram and _mgram in ngram_freq:
            numerator = ngram_freq[_mgram]
        else:
            numerator = total_reference_words
        information_weights[_ngram] = math.log(numerator / ngram_freq[_ngram], 2)

    
    nist_precision_numerator_per_ngram = Counter()
    nist_precision_denominator_per_ngram = Counter()
    l_ref, l_sys = 0, 0
    
    for i in range(1, n + 1):
        
        for references, hypothesis in zip(list_of_references, hypotheses):
            hyp_len = len(hypothesis)

            
            nist_score_per_ref = []
            for reference in references:
                _ref_len = len(reference)
                
                hyp_ngrams = (
                    Counter(ngrams(hypothesis, i))
                    if len(hypothesis) >= i
                    else Counter()
                )
                ref_ngrams = (
                    Counter(ngrams(reference, i)) if len(reference) >= i else Counter()
                )
                ngram_overlaps = hyp_ngrams & ref_ngrams
                
                _numerator = sum(
                    information_weights[_ngram] * count
                    for _ngram, count in ngram_overlaps.items()
                )
                _denominator = sum(hyp_ngrams.values())
                _precision = 0 if _denominator == 0 else _numerator / _denominator
                nist_score_per_ref.append(
                    (_precision, _numerator, _denominator, _ref_len)
                )
            
            precision, numerator, denominator, ref_len = max(nist_score_per_ref)
            nist_precision_numerator_per_ngram[i] += numerator
            nist_precision_denominator_per_ngram[i] += denominator
            l_ref += ref_len
            l_sys += hyp_len

    
    nist_precision = 0
    for i in nist_precision_numerator_per_ngram:
        precision = (
            nist_precision_numerator_per_ngram[i]
            / nist_precision_denominator_per_ngram[i]
        )
        nist_precision += precision
    
    return nist_precision * nist_length_penalty(l_ref, l_sys)


def nist_length_penalty(ref_len, hyp_len):
    
    ratio = hyp_len / ref_len
    if 0 < ratio < 1:
        ratio_x, score_x = 1.5, 0.5
        beta = math.log(score_x) / math.log(ratio_x) ** 2
        return math.exp(beta * math.log(ratio) ** 2)
    else:  
        return max(min(ratio, 1.0), 0.0)