natural language toolkit chunkers c 20012023 nltk project steven bird stevenbird1gmail com edward loper edlopergmail com url https www nltk org for license information see license txt classes and interfaces for identifying nonoverlapping linguistic groups such as base noun phrases in unrestricted text this task is called chunk parsing or chunking and the identified groups are called chunks the chunked text is represented using a shallow tree called a chunk structure a chunk structure is a tree containing tokens and chunks where each chunk is a subtree containing only tokens for example the chunk structure for base noun phrase chunks in the sentence i saw the big dog on the hill is sentence np i saw np the big dog on np the hill to convert a chunk structure back to a list of tokens simply use the chunk structure s leaves method this module defines chunkparseri a standard interface for chunking texts and regexpchunkparser a regularexpression based implementation of that interface it also defines chunkscore a utility class for scoring chunk parsers regexpchunkparser regexpchunkparser is an implementation of the chunk parser interface that uses regularexpressions over tags to chunk a text its parse method first constructs a chunkstring which encodes a particular chunking of the input text initially nothing is chunked parse regexpchunkparser then applies a sequence of regexpchunkrule rules to the chunkstring each of which modifies the chunking that it encodes finally the chunkstring is transformed back into a chunk structure which is returned regexpchunkparser can only be used to chunk a single kind of phrase for example you can use an regexpchunkparser to chunk the noun phrases in a text or the verb phrases in a text but you can not use it to simultaneously chunk both noun phrases and verb phrases in the same text this is a limitation of regexpchunkparser not of chunk parsers in general regexpchunkrules a regexpchunkrule is a transformational rule that updates the chunking of a text by modifying its chunkstring each regexpchunkrule defines the apply method which modifies the chunking encoded by a chunkstring the regexpchunkrule class itself can be used to implement any transformational rule based on regular expressions there are also a number of subclasses which can be used to implement simpler types of rules chunkrule chunks anything that matches a given regular expression striprule strips anything that matches a given regular expression unchunkrule will unchunk any chunk that matches a given regular expression mergerule can be used to merge two contiguous chunks splitrule can be used to split a single chunk into two smaller chunks expandleftrule will expand a chunk to incorporate new unchunked material on the left expandrightrule will expand a chunk to incorporate new unchunked material on the right tag patterns a regexpchunkrule uses a modified version of regular expression patterns called tag patterns tag patterns are used to match sequences of tags examples of tag patterns are r dtjjnn r nn r nn the differences between regular expression patterns and tag patterns are in tag patterns and act as parentheses so nn matches one or more repetitions of nn not nn followed by one or more repetitions of whitespace in tag patterns is ignored so dt nn is equivalent to dtnn in tag patterns is equivalent to so nn matches any single tag starting with nn the function tagpattern2repattern can be used to transform a tag pattern to an equivalent regular expression pattern efficiency preliminary tests indicate that regexpchunkparser can chunk at a rate of about 300 tokenssecond with a moderately complex rule set there may be problems if regexpchunkparser is used with more than 5 000 tokens at a time in particular evaluation of some regular expressions may cause the python regular expression engine to exceed its maximum recursion depth we have attempted to minimize these problems but it is impossible to avoid them completely we therefore recommend that you apply the chunk parser to a single sentence at a time emacs tip if you evaluate the following elisp expression in emacs it will colorize a chunkstring when you use an interactive python shell with emacs or xemacs cc let defconst comintmodefontlockkeywords 0 fontlockreferenceface 0 fontlockfunctionnameface addhook comintmodehook lambda turnonfontlock you can evaluate this code by copying it to a temporary buffer placing the cursor after the last close parenthesis and typing cx ce you should evaluate it before running the interactive session the change will last until you close emacs unresolved issues if we use the re module for regular expressions python s regular expression engine generates maximum recursion depth exceeded errors when processing very large texts even for regular expressions that should not require any recursion we therefore use the pre module instead but note that pre does not include unicode support so this module will not work with unicode strings note also that pre regular expressions are not quite as advanced as re ones e g no leftward zerolength assertions type chunktagpattern regexp var chunktagpattern a regular expression to test whether a tag pattern is valid standard treebank pos tagger use nltk s currently recommended named entity chunker to chunk the given list of tagged tokens use nltk s currently recommended named entity chunker to chunk the given list of tagged sentences each consisting of a list of tagged tokens natural language toolkit chunkers c 2001 2023 nltk project steven bird stevenbird1 gmail com edward loper edloper gmail com url https www nltk org for license information see license txt classes and interfaces for identifying non overlapping linguistic groups such as base noun phrases in unrestricted text this task is called chunk parsing or chunking and the identified groups are called chunks the chunked text is represented using a shallow tree called a chunk structure a chunk structure is a tree containing tokens and chunks where each chunk is a subtree containing only tokens for example the chunk structure for base noun phrase chunks in the sentence i saw the big dog on the hill is sentence np i saw np the big dog on np the hill to convert a chunk structure back to a list of tokens simply use the chunk structure s leaves method this module defines chunkparseri a standard interface for chunking texts and regexpchunkparser a regular expression based implementation of that interface it also defines chunkscore a utility class for scoring chunk parsers regexpchunkparser regexpchunkparser is an implementation of the chunk parser interface that uses regular expressions over tags to chunk a text its parse method first constructs a chunkstring which encodes a particular chunking of the input text initially nothing is chunked parse regexpchunkparser then applies a sequence of regexpchunkrule rules to the chunkstring each of which modifies the chunking that it encodes finally the chunkstring is transformed back into a chunk structure which is returned regexpchunkparser can only be used to chunk a single kind of phrase for example you can use an regexpchunkparser to chunk the noun phrases in a text or the verb phrases in a text but you can not use it to simultaneously chunk both noun phrases and verb phrases in the same text this is a limitation of regexpchunkparser not of chunk parsers in general regexpchunkrules a regexpchunkrule is a transformational rule that updates the chunking of a text by modifying its chunkstring each regexpchunkrule defines the apply method which modifies the chunking encoded by a chunkstring the regexpchunkrule class itself can be used to implement any transformational rule based on regular expressions there are also a number of subclasses which can be used to implement simpler types of rules chunkrule chunks anything that matches a given regular expression striprule strips anything that matches a given regular expression unchunkrule will un chunk any chunk that matches a given regular expression mergerule can be used to merge two contiguous chunks splitrule can be used to split a single chunk into two smaller chunks expandleftrule will expand a chunk to incorporate new unchunked material on the left expandrightrule will expand a chunk to incorporate new unchunked material on the right tag patterns a regexpchunkrule uses a modified version of regular expression patterns called tag patterns tag patterns are used to match sequences of tags examples of tag patterns are r dt jj nn r nn r nn the differences between regular expression patterns and tag patterns are in tag patterns and act as parentheses so nn matches one or more repetitions of nn not nn followed by one or more repetitions of whitespace in tag patterns is ignored so dt nn is equivalent to dt nn in tag patterns is equivalent to so nn matches any single tag starting with nn the function tag_pattern2re_pattern can be used to transform a tag pattern to an equivalent regular expression pattern efficiency preliminary tests indicate that regexpchunkparser can chunk at a rate of about 300 tokens second with a moderately complex rule set there may be problems if regexpchunkparser is used with more than 5 000 tokens at a time in particular evaluation of some regular expressions may cause the python regular expression engine to exceed its maximum recursion depth we have attempted to minimize these problems but it is impossible to avoid them completely we therefore recommend that you apply the chunk parser to a single sentence at a time emacs tip if you evaluate the following elisp expression in emacs it will colorize a chunkstring when you use an interactive python shell with emacs or xemacs c c let defconst comint mode font lock keywords 0 font lock reference face 0 font lock function name face add hook comint mode hook lambda turn on font lock you can evaluate this code by copying it to a temporary buffer placing the cursor after the last close parenthesis and typing c x c e you should evaluate it before running the interactive session the change will last until you close emacs unresolved issues if we use the re module for regular expressions python s regular expression engine generates maximum recursion depth exceeded errors when processing very large texts even for regular expressions that should not require any recursion we therefore use the pre module instead but note that pre does not include unicode support so this module will not work with unicode strings note also that pre regular expressions are not quite as advanced as re ones e g no leftward zero length assertions type chunk_tag_pattern regexp var chunk_tag_pattern a regular expression to test whether a tag pattern is valid standard treebank pos tagger use nltk s currently recommended named entity chunker to chunk the given list of tagged tokens use nltk s currently recommended named entity chunker to chunk the given list of tagged sentences each consisting of a list of tagged tokens
from nltk.chunk.api import ChunkParserI
from nltk.chunk.regexp import RegexpChunkParser, RegexpParser
from nltk.chunk.util import (
    ChunkScore,
    accuracy,
    conllstr2tree,
    conlltags2tree,
    ieerstr2tree,
    tagstr2tree,
    tree2conllstr,
    tree2conlltags,
)
from nltk.data import load


_BINARY_NE_CHUNKER = "chunkers/maxent_ne_chunker/english_ace_binary.pickle"
_MULTICLASS_NE_CHUNKER = "chunkers/maxent_ne_chunker/english_ace_multiclass.pickle"


def ne_chunk(tagged_tokens, binary=False):
    
    if binary:
        chunker_pickle = _BINARY_NE_CHUNKER
    else:
        chunker_pickle = _MULTICLASS_NE_CHUNKER
    chunker = load(chunker_pickle)
    return chunker.parse(tagged_tokens)


def ne_chunk_sents(tagged_sentences, binary=False):
    
    if binary:
        chunker_pickle = _BINARY_NE_CHUNKER
    else:
        chunker_pickle = _MULTICLASS_NE_CHUNKER
    chunker = load(chunker_pickle)
    return chunker.parse_sents(tagged_sentences)