natural language toolkit interface to the crfsuite tagger c 20012023 nltk project long duong longdt219gmail com url https www nltk org for license information see license txt a module for pos tagging using crfsuite a module for pos tagging using crfsuite https pypi python orgpypipythoncrfsuite from nltk tag import crftagger ct crftagger doctest skip traindata university noun is verb a det good adj place noun dog noun eat verb meat noun ct traintraindata model crf tagger doctest skip ct tagsents dog is good cat eat meat doctest skip dog noun is verb good adj cat noun eat verb meat noun goldsentences dog noun is verb good adj cat noun eat verb meat noun ct accuracygoldsentences doctest skip 1 0 setting learned model file ct crftagger doctest skip ct setmodelfile model crf tagger doctest skip ct accuracygoldsentences doctest skip 1 0 initialize the crfsuite tagger param featurefunc the function that extracts features for each token of a sentence this function should take 2 parameters tokens and index which extract features at index position from tokens list see the build in getfeatures function for more detail param verbose output the debugging messages during training type verbose boolean param trainingopt pythoncrfsuite training options type trainingopt dictionary set of possible training options using lbfgs training algorithm feature minfreq the minimum frequency of features feature possiblestates force to generate possible state features feature possibletransitions force to generate possible transition features c1 coefficient for l1 regularization c2 coefficient for l2 regularization maxiterations the maximum number of iterations for lbfgs optimization nummemories the number of limited memories for approximating the inverse hessian matrix epsilon epsilon for testing the convergence of the objective period the duration of iterations to test the stopping criterion delta the threshold for the stopping criterion an lbfgs iteration stops when the improvement of the log likelihood over the last period iterations is no greater than this threshold linesearch the line search algorithm used in lbfgs updates morethuente more and thuente s method backtracking backtracking method with regular wolfe condition strongbacktracking backtracking method with strong wolfe condition maxlinesearch the maximum number of trials for the line search algorithm extract basic features about this word including current word is it capitalized does it have punctuation does it have a number suffixes up to length 3 note that we might include feature over previous word next word etc return a list which contains the features rtype liststr capitalization number punctuation suffix up to length 3 tag a list of sentences nb before using this function user should specify the modefile either by train a new model using train function use the pretrained model which is set via setmodelfile function params sentences list of sentences needed to tag type sentences listliststr return list of tagged sentences rtype listlisttuplestr str we need the list of sentences instead of the list generator for matching the input and output train the crf tagger using crfsuite params traindata is the list of annotated sentences type traindata list listtuplestr str params modelfile the model will be saved to this file now train the model the output should be modelfile save the model file tag a sentence using python crfsuite tagger nb before using this function user should specify the modefile either by train a new model using train function use the pretrained model which is set via setmodelfile function params tokens list of tokens needed to tag type tokens liststr return list of tagged tokens rtype listtuplestr str natural language toolkit interface to the crfsuite tagger c 2001 2023 nltk project long duong longdt219 gmail com url https www nltk org for license information see license txt a module for pos tagging using crfsuite a module for pos tagging using crfsuite https pypi python org pypi python crfsuite from nltk tag import crftagger ct crftagger doctest skip train_data university noun is verb a det good adj place noun dog noun eat verb meat noun ct train train_data model crf tagger doctest skip ct tag_sents dog is good cat eat meat doctest skip dog noun is verb good adj cat noun eat verb meat noun gold_sentences dog noun is verb good adj cat noun eat verb meat noun ct accuracy gold_sentences doctest skip 1 0 setting learned model file ct crftagger doctest skip ct set_model_file model crf tagger doctest skip ct accuracy gold_sentences doctest skip 1 0 initialize the crfsuite tagger param feature_func the function that extracts features for each token of a sentence this function should take 2 parameters tokens and index which extract features at index position from tokens list see the build in _get_features function for more detail param verbose output the debugging messages during training type verbose boolean param training_opt python crfsuite training options type training_opt dictionary set of possible training options using lbfgs training algorithm feature minfreq the minimum frequency of features feature possible_states force to generate possible state features feature possible_transitions force to generate possible transition features c1 coefficient for l1 regularization c2 coefficient for l2 regularization max_iterations the maximum number of iterations for l bfgs optimization num_memories the number of limited memories for approximating the inverse hessian matrix epsilon epsilon for testing the convergence of the objective period the duration of iterations to test the stopping criterion delta the threshold for the stopping criterion an l bfgs iteration stops when the improvement of the log likelihood over the last period iterations is no greater than this threshold linesearch the line search algorithm used in l bfgs updates morethuente more and thuente s method backtracking backtracking method with regular wolfe condition strongbacktracking backtracking method with strong wolfe condition max_linesearch the maximum number of trials for the line search algorithm extract basic features about this word including current word is it capitalized does it have punctuation does it have a number suffixes up to length 3 note that we might include feature over previous word next word etc return a list which contains the features rtype list str capitalization number punctuation suffix up to length 3 tag a list of sentences nb before using this function user should specify the mode_file either by train a new model using train function use the pre trained model which is set via set_model_file function params sentences list of sentences needed to tag type sentences list list str return list of tagged sentences rtype list list tuple str str we need the list of sentences instead of the list generator for matching the input and output train the crf tagger using crfsuite params train_data is the list of annotated sentences type train_data list list tuple str str params model_file the model will be saved to this file now train the model the output should be model_file save the model file tag a sentence using python crfsuite tagger nb before using this function user should specify the mode_file either by train a new model using train function use the pre trained model which is set via set_model_file function params tokens list of tokens needed to tag type tokens list str return list of tagged tokens rtype list tuple str str
import re
import unicodedata

from nltk.tag.api import TaggerI

try:
    import pycrfsuite
except ImportError:
    pass


class CRFTagger(TaggerI):
    

    def __init__(self, feature_func=None, verbose=False, training_opt={}):
        

        self._model_file = ""
        self._tagger = pycrfsuite.Tagger()

        if feature_func is None:
            self._feature_func = self._get_features
        else:
            self._feature_func = feature_func

        self._verbose = verbose
        self._training_options = training_opt
        self._pattern = re.compile(r"\d")

    def set_model_file(self, model_file):
        self._model_file = model_file
        self._tagger.open(self._model_file)

    def _get_features(self, tokens, idx):
        
        token = tokens[idx]

        feature_list = []

        if not token:
            return feature_list

        
        if token[0].isupper():
            feature_list.append("CAPITALIZATION")

        
        if re.search(self._pattern, token) is not None:
            feature_list.append("HAS_NUM")

        
        punc_cat = {"Pc", "Pd", "Ps", "Pe", "Pi", "Pf", "Po"}
        if all(unicodedata.category(x) in punc_cat for x in token):
            feature_list.append("PUNCTUATION")

        
        if len(token) > 1:
            feature_list.append("SUF_" + token[-1:])
        if len(token) > 2:
            feature_list.append("SUF_" + token[-2:])
        if len(token) > 3:
            feature_list.append("SUF_" + token[-3:])

        feature_list.append("WORD_" + token)

        return feature_list

    def tag_sents(self, sents):
        
        if self._model_file == "":
            raise Exception(
                " No model file is found !! Please use train or set_model_file function"
            )

        
        result = []
        for tokens in sents:
            features = [self._feature_func(tokens, i) for i in range(len(tokens))]
            labels = self._tagger.tag(features)

            if len(labels) != len(tokens):
                raise Exception(" Predicted Length Not Matched, Expect Errors !")

            tagged_sent = list(zip(tokens, labels))
            result.append(tagged_sent)

        return result

    def train(self, train_data, model_file):
        
        trainer = pycrfsuite.Trainer(verbose=self._verbose)
        trainer.set_params(self._training_options)

        for sent in train_data:
            tokens, labels = zip(*sent)
            features = [self._feature_func(tokens, i) for i in range(len(tokens))]
            trainer.append(features, labels)

        
        trainer.train(model_file)
        
        self.set_model_file(model_file)

    def tag(self, tokens):
        

        return self.tag_sents([tokens])[0]