this module is a port of the textblob averaged perceptron tagger matthew honnibal honnibalghgmail com long duong longdt219gmail com nltk port url https github comsloriatextblobaptagger https www nltk org 2013 matthew honnibal nltk modifications 2015 the nltk project this module is provided under the terms of the mit license an averaged perceptron as implemented by matthew honnibal see more implementation details here https explosion aiblogpartofspeechpostaggerinpython each feature gets its own weight vector so weights is a dictofdicts the accumulated values for the averaging these will be keyed by featureclas tuples the last time the feature was changed for the averaging also keyed by featureclas tuples tstamps is short for timestamps number of instances seen dotproduct the features and current weights and return the best label scores defaultdictfloat for feat value in features items if feat not in self weights or value 0 continue weights self weightsfeat for label weight in weights items scoreslabel value weight do a secondary alphabetic sort for stability bestlabel maxself classes keylambda label scoreslabel label compute the confidence conf maxself softmaxscores if returnconf true else none return bestlabel conf def updateself truth guess features average weights from all iterations for feat weights in self weights items newfeatweights for clas weight in weights items param feat clas total self totalsparam total self i self tstampsparam weight averaged roundtotal self i 3 if averaged newfeatweightsclas averaged self weightsfeat newfeatweights def saveself path load the pickled model weights self weights loadpath def encodejsonobjself return self weights classmethod def decodejsonobjcls obj return clsobj jsontags registertag class perceptrontaggertaggeri jsontag nltk tag sequential perceptrontagger start start start2 end end end2 def initself loadtrue self model averagedperceptron self tagdict self classes set if load apmodelloc file str findtaggersaveragedperceptrontagger pickle self loadapmodelloc def tagself tokens returnconffalse usetagdicttrue prev prev2 self start output context self start self normalizew for w in tokens self end for i word in enumeratetokens tag conf self tagdict getword 1 0 if usetagdict true else none none if not tag features self getfeaturesi word context prev prev2 tag conf self model predictfeatures returnconf output appendword tag conf if returnconf true else word tag prev2 prev prev tag return output def trainself sentences savelocnone nriter5 we d like to allow sentences to be either a list or an iterator the latter being especially important for a large training dataset because self maketagdictsentences runs regardless we make it populate self sentences a list with all the sentences this saves the overheard of just iterating through sentences to get the list by sentences listsentences self sentences list to be populated by self maketagdict self maketagdictsentences self model classes self classes for iter in rangenriter c 0 n 0 for sentence in self sentences words tags zipsentence prev prev2 self start context self start self normalizew for w in words self end for i word in enumeratewords guess self tagdict getword if not guess feats self getfeaturesi word context prev prev2 guess self model predictfeats self model updatetagsi guess feats prev2 prev prev guess c guess tagsi n 1 random shuffleself sentences logging infofiter iter cnpcc n we don t need the training sentences anymore and we don t want to waste space on them when we pickle the trained tagger self sentences none self model averageweights pickle as a binary file if saveloc is not none with opensaveloc wb as fout changed protocol from 1 to 2 to make pickling python 2 compatible pickle dumpself model weights self tagdict self classes fout 2 def loadself loc self model weights self tagdict self classes loadloc self model classes self classes def encodejsonobjself return self model weights self tagdict listself classes classmethod def decodejsonobjcls obj tagger clsloadfalse tagger model weights tagger tagdict tagger classes obj tagger classes settagger classes tagger model classes tagger classes return tagger def normalizeself word if in word and word0 return hyphen if word isdigit and lenword 4 return year if word and word0 isdigit return digits return word lower def getfeaturesself i word context prev prev2 def addname args features joinname tupleargs 1 i lenself start features defaultdictint it s useful to have a constant feature which acts sort of like a prior addbias addi suffix word3 addi pref1 word0 if word else addi1 tag prev addi2 tag prev2 addi tagi2 tag prev prev2 addi word contexti addi1 tagi word prev contexti addi1 word contexti 1 addi1 suffix contexti 13 addi2 word contexti 2 addi1 word contexti 1 addi1 suffix contexti 13 addi2 word contexti 2 return features def maketagdictself sentences counts defaultdictlambda defaultdictint for sentence in sentences self sentences appendsentence for word tag in sentence countswordtag 1 self classes addtag freqthresh 20 ambiguitythresh 0 97 for word tagfreqs in counts items tag mode maxtagfreqs items keylambda item item1 n sumtagfreqs values don t add rare words to the tag dictionary only add quite unambiguous words if n freqthresh and mode n ambiguitythresh self tagdictword tag def pcn d return n d 100 def loaddataconllformatfilename printread from file filename with openfilename rb as fin sentences sentence for line in fin readlines line line strip print line if lenline 0 sentences appendsentence sentence continue tokens line splitt word tokens1 tag tokens4 sentence appendword tag return sentences def getpretrainmodel train and test on english part of conll data wsj part of penn treebank train section 211 test section 23 tagger perceptrontagger training loaddataconllformatenglishptbtrain conll testing loaddataconllformatenglishptbtest conll printsize of training and testing sentence lentraining lentesting train and save the model tagger traintraining pickle printaccuracy tagger accuracytesting if name main getpretrainmodel pass this module is a port of the textblob averaged perceptron tagger matthew honnibal honnibal gh gmail com long duong longdt219 gmail com nltk port url https github com sloria textblob aptagger https www nltk org 2013 matthew honnibal nltk modifications 2015 the nltk project this module is provided under the terms of the mit license an averaged perceptron as implemented by matthew honnibal see more implementation details here https explosion ai blog part of speech pos tagger in python each feature gets its own weight vector so weights is a dict of dicts the accumulated values for the averaging these will be keyed by feature clas tuples the last time the feature was changed for the averaging also keyed by feature clas tuples tstamps is short for timestamps number of instances seen dot product the features and current weights and return the best label do a secondary alphabetic sort for stability compute the confidence update the feature weights average weights from all iterations save the pickled model weights load the pickled model weights greedy averaged perceptron tagger as implemented by matthew honnibal see more implementation details here https explosion ai blog part of speech pos tagger in python from nltk tag perceptron import perceptrontagger train the model tagger perceptrontagger load false tagger train today nn is vbz good jj day nn yes nns it prp beautiful jj tagger tag today is a beautiful day today nn is prp a prp beautiful jj day nn use the pretrain model the default constructor pretrain perceptrontagger pretrain tag the quick brown fox jumps over the lazy dog split the dt quick jj brown nn fox nn jumps vbz over in the dt lazy jj dog nn pretrain tag the red cat split the dt red jj cat nn param load load the pickled model upon instantiation tag tokenized sentences params tokens list of word type tokens list str train a model from sentences and save it at save_loc nr_iter controls the number of perceptron training iterations param sentences a list or iterator of sentences where each sentence is a list of words tags tuples param save_loc if not none saves a pickled model in this location param nr_iter number of training iterations we d like to allow sentences to be either a list or an iterator the latter being especially important for a large training dataset because self _make_tagdict sentences runs regardless we make it populate self _sentences a list with all the sentences this saves the overheard of just iterating through sentences to get the list by sentences list sentences to be populated by self _make_tagdict we don t need the training sentences anymore and we don t want to waste space on them when we pickle the trained tagger pickle as a binary file changed protocol from 1 to 2 to make pickling python 2 compatible param loc load a pickled model at location type loc str normalization used in pre processing all words are lower cased groups of digits of length 4 are represented as year other digits are represented as digits rtype str map tokens into a feature representation implemented as a hashable int dict if the features change a new model must be trained it s useful to have a constant feature which acts sort of like a prior make a tag dictionary for single tag words param sentences a list of list of word tag tuples don t add rare words to the tag dictionary only add quite unambiguous words print line train and test on english part of conll data wsj part of penn treebank train section 2 11 test section 23 train and save the model _get_pretrain_model
import logging
import pickle
import random
from collections import defaultdict

from nltk import jsontags
from nltk.data import find, load
from nltk.tag.api import TaggerI

try:
    import numpy as np
except ImportError:
    pass

PICKLE = "averaged_perceptron_tagger.pickle"


@jsontags.register_tag
class AveragedPerceptron:

    

    json_tag = "nltk.tag.perceptron.AveragedPerceptron"

    def __init__(self, weights=None):
        
        self.weights = weights if weights else {}
        self.classes = set()
        
        
        self._totals = defaultdict(int)
        
        
        
        self._tstamps = defaultdict(int)
        
        self.i = 0

    def _softmax(self, scores):
        s = np.fromiter(scores.values(), dtype=float)
        exps = np.exp(s)
        return exps / np.sum(exps)

    def predict(self, features, return_conf=False):
        
        scores = defaultdict(float)
        for feat, value in features.items():
            if feat not in self.weights or value == 0:
                continue
            weights = self.weights[feat]
            for label, weight in weights.items():
                scores[label] += value * weight

        
        best_label = max(self.classes, key=lambda label: (scores[label], label))
        
        conf = max(self._softmax(scores)) if return_conf == True else None

        return best_label, conf

    def update(self, truth, guess, features):
        

        def upd_feat(c, f, w, v):
            param = (f, c)
            self._totals[param] += (self.i - self._tstamps[param]) * w
            self._tstamps[param] = self.i
            self.weights[f][c] = w + v

        self.i += 1
        if truth == guess:
            return None
        for f in features:
            weights = self.weights.setdefault(f, {})
            upd_feat(truth, f, weights.get(truth, 0.0), 1.0)
            upd_feat(guess, f, weights.get(guess, 0.0), -1.0)

    def average_weights(self):
        
        for feat, weights in self.weights.items():
            new_feat_weights = {}
            for clas, weight in weights.items():
                param = (feat, clas)
                total = self._totals[param]
                total += (self.i - self._tstamps[param]) * weight
                averaged = round(total / self.i, 3)
                if averaged:
                    new_feat_weights[clas] = averaged
            self.weights[feat] = new_feat_weights

    def save(self, path):
        
        with open(path, "wb") as fout:
            return pickle.dump(dict(self.weights), fout)

    def load(self, path):
        
        self.weights = load(path)

    def encode_json_obj(self):
        return self.weights

    @classmethod
    def decode_json_obj(cls, obj):
        return cls(obj)


@jsontags.register_tag
class PerceptronTagger(TaggerI):

    

    json_tag = "nltk.tag.sequential.PerceptronTagger"

    START = ["-START-", "-START2-"]
    END = ["-END-", "-END2-"]

    def __init__(self, load=True):
        
        self.model = AveragedPerceptron()
        self.tagdict = {}
        self.classes = set()
        if load:
            AP_MODEL_LOC = "file:" + str(
                find("taggers/averaged_perceptron_tagger/" + PICKLE)
            )
            self.load(AP_MODEL_LOC)

    def tag(self, tokens, return_conf=False, use_tagdict=True):
        
        prev, prev2 = self.START
        output = []

        context = self.START + [self.normalize(w) for w in tokens] + self.END
        for i, word in enumerate(tokens):
            tag, conf = (
                (self.tagdict.get(word), 1.0) if use_tagdict == True else (None, None)
            )
            if not tag:
                features = self._get_features(i, word, context, prev, prev2)
                tag, conf = self.model.predict(features, return_conf)
            output.append((word, tag, conf) if return_conf == True else (word, tag))

            prev2 = prev
            prev = tag

        return output

    def train(self, sentences, save_loc=None, nr_iter=5):
        
        
        
        
        
        
        

        self._sentences = list()  
        self._make_tagdict(sentences)
        self.model.classes = self.classes
        for iter_ in range(nr_iter):
            c = 0
            n = 0
            for sentence in self._sentences:
                words, tags = zip(*sentence)

                prev, prev2 = self.START
                context = self.START + [self.normalize(w) for w in words] + self.END
                for i, word in enumerate(words):
                    guess = self.tagdict.get(word)
                    if not guess:
                        feats = self._get_features(i, word, context, prev, prev2)
                        guess, _ = self.model.predict(feats)
                        self.model.update(tags[i], guess, feats)
                    prev2 = prev
                    prev = guess
                    c += guess == tags[i]
                    n += 1
            random.shuffle(self._sentences)
            logging.info(f"Iter {iter_}: {c}/{n}={_pc(c, n)}")

        
        
        self._sentences = None

        self.model.average_weights()
        
        if save_loc is not None:
            with open(save_loc, "wb") as fout:
                
                pickle.dump((self.model.weights, self.tagdict, self.classes), fout, 2)

    def load(self, loc):
        

        self.model.weights, self.tagdict, self.classes = load(loc)
        self.model.classes = self.classes

    def encode_json_obj(self):
        return self.model.weights, self.tagdict, list(self.classes)

    @classmethod
    def decode_json_obj(cls, obj):
        tagger = cls(load=False)
        tagger.model.weights, tagger.tagdict, tagger.classes = obj
        tagger.classes = set(tagger.classes)
        tagger.model.classes = tagger.classes
        return tagger

    def normalize(self, word):
        
        if "-" in word and word[0] != "-":
            return "!HYPHEN"
        if word.isdigit() and len(word) == 4:
            return "!YEAR"
        if word and word[0].isdigit():
            return "!DIGITS"
        return word.lower()

    def _get_features(self, i, word, context, prev, prev2):
        

        def add(name, *args):
            features[" ".join((name,) + tuple(args))] += 1

        i += len(self.START)
        features = defaultdict(int)
        
        add("bias")
        add("i suffix", word[-3:])
        add("i pref1", word[0] if word else "")
        add("i-1 tag", prev)
        add("i-2 tag", prev2)
        add("i tag+i-2 tag", prev, prev2)
        add("i word", context[i])
        add("i-1 tag+i word", prev, context[i])
        add("i-1 word", context[i - 1])
        add("i-1 suffix", context[i - 1][-3:])
        add("i-2 word", context[i - 2])
        add("i+1 word", context[i + 1])
        add("i+1 suffix", context[i + 1][-3:])
        add("i+2 word", context[i + 2])
        return features

    def _make_tagdict(self, sentences):
        
        counts = defaultdict(lambda: defaultdict(int))
        for sentence in sentences:
            self._sentences.append(sentence)
            for word, tag in sentence:
                counts[word][tag] += 1
                self.classes.add(tag)
        freq_thresh = 20
        ambiguity_thresh = 0.97
        for word, tag_freqs in counts.items():
            tag, mode = max(tag_freqs.items(), key=lambda item: item[1])
            n = sum(tag_freqs.values())
            
            
            if n >= freq_thresh and (mode / n) >= ambiguity_thresh:
                self.tagdict[word] = tag


def _pc(n, d):
    return (n / d) * 100


def _load_data_conll_format(filename):
    print("Read from file: ", filename)
    with open(filename, "rb") as fin:
        sentences = []
        sentence = []
        for line in fin.readlines():
            line = line.strip()
            
            if len(line) == 0:
                sentences.append(sentence)
                sentence = []
                continue
            tokens = line.split("\t")
            word = tokens[1]
            tag = tokens[4]
            sentence.append((word, tag))
        return sentences


def _get_pretrain_model():
    
    
    
    tagger = PerceptronTagger()
    training = _load_data_conll_format("english_ptb_train.conll")
    testing = _load_data_conll_format("english_ptb_test.conll")
    print("Size of training and testing (sentence)", len(training), len(testing))
    
    tagger.train(training, PICKLE)
    print("Accuracy : ", tagger.accuracy(testing))


if __name__ == "__main__":
    
    pass