natural language toolkit c 20012023 nltk project ilia kurenkov ilia kurenkovgmail com url https www nltk org for license information see license txt language model counter class for counting ngrams will count any ngram sequence you give it first we need to make sure we are feeding the counter sentences of ngrams text a b c d a c d c from nltk util import ngrams textbigrams ngramssent 2 for sent in text textunigrams ngramssent 1 for sent in text the counting itself is very simple from nltk lm import ngramcounter ngramcounts ngramcountertextbigrams textunigrams you can conveniently access ngram counts using standard python dictionary notation string keys will give you unigram counts ngramcounts a 2 ngramcounts aliens 0 if you want to access counts for higher order ngrams use a list or a tuple these are treated as context keys so what you get is a frequency distribution over all continuations after the given context sortedngramcounts a items b 1 c 1 sortedngramcounts a items b 1 c 1 this is equivalent to specifying explicitly the order of the ngram in this case 2 for bigram and indexing on the context ngramcounts2 a is ngramcounts a true note that the keys in conditionalfreqdist cannot be lists only tuples it is generally advisable to use the less verbose and more flexible square bracket notation to get the count of the full ngram a b do this ngramcounts a b 1 specifying the ngram order as a number can be useful for accessing all ngrams in that order ngramcounts2 conditionalfreqdist with 4 conditions the keys of this conditionalfreqdist are the contexts we discussed earlier unigrams can also be accessed with a humanfriendly alias ngramcounts unigrams is ngramcounts1 true similarly to collections counter you can update counts after initialization ngramcounts e 0 ngramcounts updatengramsd e f 1 ngramcounts e 1 creates a new ngramcounter if ngramtext is specified counts ngrams from it otherwise waits for update method to be called explicitly param ngramtext optional text containing sentences of ngrams as for update method type ngramtext iterableiterabletuplestr or none updates ngram counts from ngramtext expects ngramtext to be a sequence of sentences sequences each sentence consists of ngrams as tuples of strings param iterableiterabletuplestr ngramtext text containing sentences of ngrams raises typeerror if the ngrams are not tuples returns grand total number of ngrams stored this includes ngrams from all orders so some duplication is expected rtype int from nltk lm import ngramcounter counts ngramcountera b c d e counts n 3 userfriendly access to ngram counts if isinstanceitem int return self countsitem elif isinstanceitem str return self counts getitem1item elif isinstanceitem sequence return self counts getitemlenitem 1tupleitem def strself return with ngram orders and ngrams format self class name lenself counts self n def lenself return self counts len def containsself item return item in self counts natural language toolkit c 2001 2023 nltk project ilia kurenkov ilia kurenkov gmail com url https www nltk org for license information see license txt language model counter class for counting ngrams will count any ngram sequence you give it first we need to make sure we are feeding the counter sentences of ngrams text a b c d a c d c from nltk util import ngrams text_bigrams ngrams sent 2 for sent in text text_unigrams ngrams sent 1 for sent in text the counting itself is very simple from nltk lm import ngramcounter ngram_counts ngramcounter text_bigrams text_unigrams you can conveniently access ngram counts using standard python dictionary notation string keys will give you unigram counts ngram_counts a 2 ngram_counts aliens 0 if you want to access counts for higher order ngrams use a list or a tuple these are treated as context keys so what you get is a frequency distribution over all continuations after the given context sorted ngram_counts a items b 1 c 1 sorted ngram_counts a items b 1 c 1 this is equivalent to specifying explicitly the order of the ngram in this case 2 for bigram and indexing on the context ngram_counts 2 a is ngram_counts a true note that the keys in conditionalfreqdist cannot be lists only tuples it is generally advisable to use the less verbose and more flexible square bracket notation to get the count of the full ngram a b do this ngram_counts a b 1 specifying the ngram order as a number can be useful for accessing all ngrams in that order ngram_counts 2 conditionalfreqdist with 4 conditions the keys of this conditionalfreqdist are the contexts we discussed earlier unigrams can also be accessed with a human friendly alias ngram_counts unigrams is ngram_counts 1 true similarly to collections counter you can update counts after initialization ngram_counts e 0 ngram_counts update ngrams d e f 1 ngram_counts e 1 creates a new ngramcounter if ngram_text is specified counts ngrams from it otherwise waits for update method to be called explicitly param ngram_text optional text containing sentences of ngrams as for update method type ngram_text iterable iterable tuple str or none updates ngram counts from ngram_text expects ngram_text to be a sequence of sentences sequences each sentence consists of ngrams as tuples of strings param iterable iterable tuple str ngram_text text containing sentences of ngrams raises typeerror if the ngrams are not tuples returns grand total number of ngrams stored this includes ngrams from all orders so some duplication is expected rtype int from nltk lm import ngramcounter counts ngramcounter a b c d e counts n 3 user friendly access to ngram counts
from collections import defaultdict
from collections.abc import Sequence

from nltk.probability import ConditionalFreqDist, FreqDist


class NgramCounter:
    

    def __init__(self, ngram_text=None):
        
        self._counts = defaultdict(ConditionalFreqDist)
        self._counts[1] = self.unigrams = FreqDist()

        if ngram_text:
            self.update(ngram_text)

    def update(self, ngram_text):
        

        for sent in ngram_text:
            for ngram in sent:
                if not isinstance(ngram, tuple):
                    raise TypeError(
                        "Ngram <{}> isn't a tuple, " "but {}".format(ngram, type(ngram))
                    )

                ngram_order = len(ngram)
                if ngram_order == 1:
                    self.unigrams[ngram[0]] += 1
                    continue

                context, word = ngram[:-1], ngram[-1]
                self[ngram_order][context][word] += 1

    def N(self):
        
        return sum(val.N() for val in self._counts.values())

    def __getitem__(self, item):
        
        if isinstance(item, int):
            return self._counts[item]
        elif isinstance(item, str):
            return self._counts.__getitem__(1)[item]
        elif isinstance(item, Sequence):
            return self._counts.__getitem__(len(item) + 1)[tuple(item)]

    def __str__(self):
        return "<{} with {} ngram orders and {} ngrams>".format(
            self.__class__.__name__, len(self._counts), self.N()
        )

    def __len__(self):
        return self._counts.__len__()

    def __contains__(self, item):
        return item in self._counts