natural language toolkit transformationbased learning c 20012023 nltk project marcus uneson marcus unesongmail com based on previous nltk2 version by christopher maloof edward loper steven bird url https www nltk org for license information see license txt run a demo with defaults see source comments for details or docstrings of any of the more specific demo functions exemplify reprrule see also strrule and rule formatverbose exemplify reprrule see also strrule and rule formatverbose exemplify rule formatverbose the features of a template takes a list of positions relative to the current word where the feature should be looked for conceptually joined by logical or for instance pos1 1 given a value v will hold whenever v is found one step to the left andor one step to the right for contiguous ranges a 2arg form giving inclusive end points can also be used pos3 1 is the same as the arg below templates can have more than a single feature show aggregate statistics per template little used templates are candidates for deletion much used templates may possibly be refined deleting unused templates is mostly about saving time andor space training is basically ot in the number of templates t also in terms of memory usage which often will be the limiting factor template expand and feature expand are class methods facilitating generating large amounts of templates see their documentation for details note training with 500 templates can easily fill all available even on relatively small corpora plot a learning curve the contribution on tagging accuracy of the individual rules note requires matplotlib writes a file with context for each erroneous word after tagging testing data serializes the learned tagger to a file in pickle format reloads it and validates the process discard rules with low accuracy this may hurt performance a bit but will often produce rules which are more interesting read to a human brill tagger demonstration param templates how many sentences of training and testing data to use type templates list of template param taggeddata maximum number of rule instances to create type taggeddata cint param numsents how many sentences of training and testing data to use type numsents cint param maxrules maximum number of rule instances to create type maxrules cint param minscore the minimum score for a rule in order for it to be considered type minscore cint param minacc the minimum score for a rule in order for it to be considered type minacc cfloat param train the fraction of the the corpus to be used for training 1all type train cfloat param trace the level of diagnostic tracing output to produce 04 type trace cint param randomize whether the training data should be a random subset of the corpus type randomize cbool param ruleformat rule output format one of str repr verbose type ruleformat cstr param incrementalstats if true will tag incrementally and collect stats for each rule rather slow type incrementalstats cbool param templatestats if true will print pertemplate statistics collected in training and optionally testing type templatestats cbool param erroroutput the file where errors will be saved type erroroutput cstring param serializeoutput the file where the learned tbl tagger will be saved type serializeoutput cstring param learningcurveoutput filename of plot of learning curves train and also test if available type learningcurveoutput cstring param learningcurvetake how many rules plotted type learningcurvetake cint param baselinebackofftagger the file where rules will be saved type baselinebackofftagger tagger param separatebaselinedata use a fraction of the training data exclusively for training baseline type separatebaselinedata cbool param cachebaselinetagger cache baseline tagger to this file only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions type cachebaselinetagger cstring note on separatebaselinedata if true reuse training data both for baseline and rule learner this is fast and fine for a demo but is likely to generalize worse on unseen data also cannot be sensibly used for learning curves on training data the baseline will be artificially high defaults some prebuilt template sets taken from typical systems or publications are available print a list with describetemplatesets for instance creating or reloading from cache a baseline tagger unigram tagger this is just a mechanism for getting deterministic output from the baseline between python versions creating a brill tagger printing the learned rules if learned silently printing template statistics optionally including comparison with the training data note if not separatebaselinedata then baseline accuracy will be artificially high writing error analysis to file serializing the tagger to a pickle file and reloading just to see it works train is the proportion of data used in training the rest is reserved for testing natural language toolkit transformation based learning c 2001 2023 nltk project marcus uneson marcus uneson gmail com based on previous nltk2 version by christopher maloof edward loper steven bird url https www nltk org for license information see license txt run a demo with defaults see source comments for details or docstrings of any of the more specific demo_ functions exemplify repr rule see also str rule and rule format verbose exemplify repr rule see also str rule and rule format verbose exemplify rule format verbose the feature s of a template takes a list of positions relative to the current word where the feature should be looked for conceptually joined by logical or for instance pos 1 1 given a value v will hold whenever v is found one step to the left and or one step to the right for contiguous ranges a 2 arg form giving inclusive end points can also be used pos 3 1 is the same as the arg below templates can have more than a single feature show aggregate statistics per template little used templates are candidates for deletion much used templates may possibly be refined deleting unused templates is mostly about saving time and or space training is basically o t in the number of templates t also in terms of memory usage which often will be the limiting factor template expand and feature expand are class methods facilitating generating large amounts of templates see their documentation for details note training with 500 templates can easily fill all available even on relatively small corpora plot a learning curve the contribution on tagging accuracy of the individual rules note requires matplotlib writes a file with context for each erroneous word after tagging testing data serializes the learned tagger to a file in pickle format reloads it and validates the process discard rules with low accuracy this may hurt performance a bit but will often produce rules which are more interesting read to a human brill tagger demonstration param templates how many sentences of training and testing data to use type templates list of template param tagged_data maximum number of rule instances to create type tagged_data c int param num_sents how many sentences of training and testing data to use type num_sents c int param max_rules maximum number of rule instances to create type max_rules c int param min_score the minimum score for a rule in order for it to be considered type min_score c int param min_acc the minimum score for a rule in order for it to be considered type min_acc c float param train the fraction of the the corpus to be used for training 1 all type train c float param trace the level of diagnostic tracing output to produce 0 4 type trace c int param randomize whether the training data should be a random subset of the corpus type randomize c bool param ruleformat rule output format one of str repr verbose type ruleformat c str param incremental_stats if true will tag incrementally and collect stats for each rule rather slow type incremental_stats c bool param template_stats if true will print per template statistics collected in training and optionally testing type template_stats c bool param error_output the file where errors will be saved type error_output c string param serialize_output the file where the learned tbl tagger will be saved type serialize_output c string param learning_curve_output filename of plot of learning curve s train and also test if available type learning_curve_output c string param learning_curve_take how many rules plotted type learning_curve_take c int param baseline_backoff_tagger the file where rules will be saved type baseline_backoff_tagger tagger param separate_baseline_data use a fraction of the training data exclusively for training baseline type separate_baseline_data c bool param cache_baseline_tagger cache baseline tagger to this file only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions type cache_baseline_tagger c string note on separate_baseline_data if true reuse training data both for baseline and rule learner this is fast and fine for a demo but is likely to generalize worse on unseen data also cannot be sensibly used for learning curves on training data the baseline will be artificially high defaults some pre built template sets taken from typical systems or publications are available print a list with describe_template_sets for instance creating or reloading from cache a baseline tagger unigram tagger this is just a mechanism for getting deterministic output from the baseline between python versions creating a brill tagger printing the learned rules if learned silently printing template statistics optionally including comparison with the training data note if not separate_baseline_data then baseline accuracy will be artificially high writing error analysis to file serializing the tagger to a pickle file and reloading just to see it works train is the proportion of data used in training the rest is reserved for testing cardinal numbers articles adjectives nouns formed from adjectives adverbs plural nouns gerunds past tense verbs nouns default
import os
import pickle
import random
import time

from nltk.corpus import treebank
from nltk.tag import BrillTaggerTrainer, RegexpTagger, UnigramTagger
from nltk.tag.brill import Pos, Word
from nltk.tbl import Template, error_list


def demo():
    
    postag()


def demo_repr_rule_format():
    
    postag(ruleformat="repr")


def demo_str_rule_format():
    
    postag(ruleformat="str")


def demo_verbose_rule_format():
    
    postag(ruleformat="verbose")


def demo_multiposition_feature():
    
    postag(templates=[Template(Pos([-3, -2, -1]))])


def demo_multifeature_template():
    
    postag(templates=[Template(Word([0]), Pos([-2, -1]))])


def demo_template_statistics():
    
    postag(incremental_stats=True, template_stats=True)


def demo_generated_templates():
    
    wordtpls = Word.expand([-1, 0, 1], [1, 2], excludezero=False)
    tagtpls = Pos.expand([-2, -1, 0, 1], [1, 2], excludezero=True)
    templates = list(Template.expand([wordtpls, tagtpls], combinations=(1, 3)))
    print(
        "Generated {} templates for transformation-based learning".format(
            len(templates)
        )
    )
    postag(templates=templates, incremental_stats=True, template_stats=True)


def demo_learning_curve():
    
    postag(
        incremental_stats=True,
        separate_baseline_data=True,
        learning_curve_output="learningcurve.png",
    )


def demo_error_analysis():
    
    postag(error_output="errors.txt")


def demo_serialize_tagger():
    
    postag(serialize_output="tagger.pcl")


def demo_high_accuracy_rules():
    
    postag(num_sents=3000, min_acc=0.96, min_score=10)


def postag(
    templates=None,
    tagged_data=None,
    num_sents=1000,
    max_rules=300,
    min_score=3,
    min_acc=None,
    train=0.8,
    trace=3,
    randomize=False,
    ruleformat="str",
    incremental_stats=False,
    template_stats=False,
    error_output=None,
    serialize_output=None,
    learning_curve_output=None,
    learning_curve_take=300,
    baseline_backoff_tagger=None,
    separate_baseline_data=False,
    cache_baseline_tagger=None,
):
    

    
    baseline_backoff_tagger = baseline_backoff_tagger or REGEXP_TAGGER
    if templates is None:
        from nltk.tag.brill import brill24, describe_template_sets

        
        
        
        templates = brill24()
    (training_data, baseline_data, gold_data, testing_data) = _demo_prepare_data(
        tagged_data, train, num_sents, randomize, separate_baseline_data
    )

    
    
    
    if cache_baseline_tagger:
        if not os.path.exists(cache_baseline_tagger):
            baseline_tagger = UnigramTagger(
                baseline_data, backoff=baseline_backoff_tagger
            )
            with open(cache_baseline_tagger, "w") as print_rules:
                pickle.dump(baseline_tagger, print_rules)
            print(
                "Trained baseline tagger, pickled it to {}".format(
                    cache_baseline_tagger
                )
            )
        with open(cache_baseline_tagger) as print_rules:
            baseline_tagger = pickle.load(print_rules)
            print(f"Reloaded pickled tagger from {cache_baseline_tagger}")
    else:
        baseline_tagger = UnigramTagger(baseline_data, backoff=baseline_backoff_tagger)
        print("Trained baseline tagger")
    if gold_data:
        print(
            "    Accuracy on test set: {:0.4f}".format(
                baseline_tagger.accuracy(gold_data)
            )
        )

    
    tbrill = time.time()
    trainer = BrillTaggerTrainer(
        baseline_tagger, templates, trace, ruleformat=ruleformat
    )
    print("Training tbl tagger...")
    brill_tagger = trainer.train(training_data, max_rules, min_score, min_acc)
    print(f"Trained tbl tagger in {time.time() - tbrill:0.2f} seconds")
    if gold_data:
        print("    Accuracy on test set: %.4f" % brill_tagger.accuracy(gold_data))

    
    if trace == 1:
        print("\nLearned rules: ")
        for (ruleno, rule) in enumerate(brill_tagger.rules(), 1):
            print(f"{ruleno:4d} {rule.format(ruleformat):s}")

    
    
    if incremental_stats:
        print(
            "Incrementally tagging the test data, collecting individual rule statistics"
        )
        (taggedtest, teststats) = brill_tagger.batch_tag_incremental(
            testing_data, gold_data
        )
        print("    Rule statistics collected")
        if not separate_baseline_data:
            print(
                "WARNING: train_stats asked for separate_baseline_data=True; the baseline "
                "will be artificially high"
            )
        trainstats = brill_tagger.train_stats()
        if template_stats:
            brill_tagger.print_template_statistics(teststats)
        if learning_curve_output:
            _demo_plot(
                learning_curve_output, teststats, trainstats, take=learning_curve_take
            )
            print(f"Wrote plot of learning curve to {learning_curve_output}")
    else:
        print("Tagging the test data")
        taggedtest = brill_tagger.tag_sents(testing_data)
        if template_stats:
            brill_tagger.print_template_statistics()

    
    if error_output is not None:
        with open(error_output, "w") as f:
            f.write("Errors for Brill Tagger %r\n\n" % serialize_output)
            f.write("\n".join(error_list(gold_data, taggedtest)).encode("utf-8") + "\n")
        print(f"Wrote tagger errors including context to {error_output}")

    
    if serialize_output is not None:
        taggedtest = brill_tagger.tag_sents(testing_data)
        with open(serialize_output, "w") as print_rules:
            pickle.dump(brill_tagger, print_rules)
        print(f"Wrote pickled tagger to {serialize_output}")
        with open(serialize_output) as print_rules:
            brill_tagger_reloaded = pickle.load(print_rules)
        print(f"Reloaded pickled tagger from {serialize_output}")
        taggedtest_reloaded = brill_tagger.tag_sents(testing_data)
        if taggedtest == taggedtest_reloaded:
            print("Reloaded tagger tried on test set, results identical")
        else:
            print("PROBLEM: Reloaded tagger gave different results on test set")


def _demo_prepare_data(
    tagged_data, train, num_sents, randomize, separate_baseline_data
):
    
    
    if tagged_data is None:
        print("Loading tagged data from treebank... ")
        tagged_data = treebank.tagged_sents()
    if num_sents is None or len(tagged_data) <= num_sents:
        num_sents = len(tagged_data)
    if randomize:
        random.seed(len(tagged_data))
        random.shuffle(tagged_data)
    cutoff = int(num_sents * train)
    training_data = tagged_data[:cutoff]
    gold_data = tagged_data[cutoff:num_sents]
    testing_data = [[t[0] for t in sent] for sent in gold_data]
    if not separate_baseline_data:
        baseline_data = training_data
    else:
        bl_cutoff = len(training_data) // 3
        (baseline_data, training_data) = (
            training_data[:bl_cutoff],
            training_data[bl_cutoff:],
        )
    (trainseqs, traintokens) = corpus_size(training_data)
    (testseqs, testtokens) = corpus_size(testing_data)
    (bltrainseqs, bltraintokens) = corpus_size(baseline_data)
    print(f"Read testing data ({testseqs:d} sents/{testtokens:d} wds)")
    print(f"Read training data ({trainseqs:d} sents/{traintokens:d} wds)")
    print(
        "Read baseline data ({:d} sents/{:d} wds) {:s}".format(
            bltrainseqs,
            bltraintokens,
            "" if separate_baseline_data else "[reused the training set]",
        )
    )
    return (training_data, baseline_data, gold_data, testing_data)


def _demo_plot(learning_curve_output, teststats, trainstats=None, take=None):
    testcurve = [teststats["initialerrors"]]
    for rulescore in teststats["rulescores"]:
        testcurve.append(testcurve[-1] - rulescore)
    testcurve = [1 - x / teststats["tokencount"] for x in testcurve[:take]]

    traincurve = [trainstats["initialerrors"]]
    for rulescore in trainstats["rulescores"]:
        traincurve.append(traincurve[-1] - rulescore)
    traincurve = [1 - x / trainstats["tokencount"] for x in traincurve[:take]]

    import matplotlib.pyplot as plt

    r = list(range(len(testcurve)))
    plt.plot(r, testcurve, r, traincurve)
    plt.axis([None, None, None, 1.0])
    plt.savefig(learning_curve_output)


NN_CD_TAGGER = RegexpTagger([(r"^-?[0-9]+(\.[0-9]+)?$", "CD"), (r".*", "NN")])

REGEXP_TAGGER = RegexpTagger(
    [
        (r"^-?[0-9]+(\.[0-9]+)?$", "CD"),  
        (r"(The|the|A|a|An|an)$", "AT"),  
        (r".*able$", "JJ"),  
        (r".*ness$", "NN"),  
        (r".*ly$", "RB"),  
        (r".*s$", "NNS"),  
        (r".*ing$", "VBG"),  
        (r".*ed$", "VBD"),  
        (r".*", "NN"),  
    ]
)


def corpus_size(seqs):
    return (len(seqs), sum(len(x) for x in seqs))


if __name__ == "__main__":
    demo_learning_curve()