natural language toolkit tagged corpus reader c 20012023 nltk project edward loper edlopergmail com steven bird stevenbird1gmail com jacob perkins japerkgmail com url https www nltk org for license information see license txt a reader for corpora whose documents contain partofspeechtagged words reader for simple partofspeech tagged corpora paragraphs are assumed to be split using blank lines sentences and words can be tokenized using the default tokenizers or by custom tokenizers specified as parameters to the constructor words are parsed using nltk tag str2tuple by default is used as the separator i e words should have the form word1tag1 word2tag2 word3tag3 but custom separators may be specified as parameters to the constructor part of speech tags are casenormalized to upper case construct a new tagged corpus reader for a set of documents located at the given root directory example usage root path to corpus reader taggedcorpusreaderroot txt doctest skip param root the root directory for this corpus param fileids a list or regexp specifying the fileids in this corpus return the given files as a list of words and punctuation symbols rtype liststr return the given files as a list of sentences or utterances each encoded as a list of word strings rtype listliststr return the given files as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word strings rtype listlistliststr return the given files as a list of tagged words and punctuation symbols encoded as tuples word tag rtype listtuplestr str return the given files as a list of sentences each encoded as a list of word tag tuples rtype listlisttuplestr str return the given files as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word tag tuples rtype listlistlisttuplestr str a reader for partofspeech tagged corpora whose documents are divided into categories based on their file identifiers initialize the corpus reader categorization arguments catpattern catmap and catfile are passed to the categorizedcorpusreader constructor the remaining arguments are passed to the taggedcorpusreader a specialized corpus view for tagged documents it can be customized via flags to divide the tagged corpus documents up by sentence or paragraph and to include or omit part of speech tags taggedcorpusview objects are typically taggedcorpusreader not directly by nltk users reads one paragraph at a time block for parastr in self parablockreaderstream para for sentstr in self senttokenizer tokenizeparastr sent str2tuples self sep for s in self wordtokenizer tokenizesentstr if self tagmappingfunction sent w self tagmappingfunctiont for w t in sent if not self tagged sent w for w t in sent if self groupbysent para appendsent else para extendsent if self groupbypara block appendpara else block extendpara return block needs to implement simplified tags class macmorphocorpusreadertaggedcorpusreader def initself root fileids encodingutf8 tagsetnone taggedcorpusreader init self root fileids sep wordtokenizerlinetokenizer senttokenizerregexptokenizer n parablockreaderself readblock encodingencoding tagsettagset def readblockself stream return readregexpblockstream r r class timittaggedcorpusreadertaggedcorpusreader def initself args kwargs taggedcorpusreader init self parablockreaderreadtimitblock args kwargs def parasself raise notimplementederroruse sents instead def taggedparasself raise notimplementederroruse taggedsents instead natural language toolkit tagged corpus reader c 2001 2023 nltk project edward loper edloper gmail com steven bird stevenbird1 gmail com jacob perkins japerk gmail com url https www nltk org for license information see license txt a reader for corpora whose documents contain part of speech tagged words reader for simple part of speech tagged corpora paragraphs are assumed to be split using blank lines sentences and words can be tokenized using the default tokenizers or by custom tokenizers specified as parameters to the constructor words are parsed using nltk tag str2tuple by default is used as the separator i e words should have the form word1 tag1 word2 tag2 word3 tag3 but custom separators may be specified as parameters to the constructor part of speech tags are case normalized to upper case construct a new tagged corpus reader for a set of documents located at the given root directory example usage root path to corpus reader taggedcorpusreader root txt doctest skip param root the root directory for this corpus param fileids a list or regexp specifying the fileids in this corpus return the given file s as a list of words and punctuation symbols rtype list str return the given file s as a list of sentences or utterances each encoded as a list of word strings rtype list list str return the given file s as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word strings rtype list list list str return the given file s as a list of tagged words and punctuation symbols encoded as tuples word tag rtype list tuple str str return the given file s as a list of sentences each encoded as a list of word tag tuples rtype list list tuple str str return the given file s as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word tag tuples rtype list list list tuple str str a reader for part of speech tagged corpora whose documents are divided into categories based on their file identifiers initialize the corpus reader categorization arguments cat_pattern cat_map and cat_file are passed to the categorizedcorpusreader constructor the remaining arguments are passed to the taggedcorpusreader a specialized corpus view for tagged documents it can be customized via flags to divide the tagged corpus documents up by sentence or paragraph and to include or omit part of speech tags taggedcorpusview objects are typically taggedcorpusreader not directly by nltk users reads one paragraph at a time needs to implement simplified tags a corpus reader for the mac_morpho corpus each line contains a single tagged word using _ as a separator sentence boundaries are based on the end sentence tag _ paragraph information is not included in the corpus so each paragraph returned by self paras and self tagged_paras contains a single sentence a corpus reader for tagged sentences that are included in the timit corpus
import os

from nltk.corpus.reader.api import *
from nltk.corpus.reader.timit import read_timit_block
from nltk.corpus.reader.util import *
from nltk.tag import map_tag, str2tuple
from nltk.tokenize import *


class TaggedCorpusReader(CorpusReader):
    

    def __init__(
        self,
        root,
        fileids,
        sep="/",
        word_tokenizer=WhitespaceTokenizer(),
        sent_tokenizer=RegexpTokenizer("\n", gaps=True),
        para_block_reader=read_blankline_block,
        encoding="utf8",
        tagset=None,
    ):
        
        CorpusReader.__init__(self, root, fileids, encoding)
        self._sep = sep
        self._word_tokenizer = word_tokenizer
        self._sent_tokenizer = sent_tokenizer
        self._para_block_reader = para_block_reader
        self._tagset = tagset

    def words(self, fileids=None):
        
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    False,
                    False,
                    False,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    None,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def sents(self, fileids=None):
        
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    False,
                    True,
                    False,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    None,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def paras(self, fileids=None):
        
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    False,
                    True,
                    True,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    None,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def tagged_words(self, fileids=None, tagset=None):
        
        if tagset and tagset != self._tagset:
            tag_mapping_function = lambda t: map_tag(self._tagset, tagset, t)
        else:
            tag_mapping_function = None
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    True,
                    False,
                    False,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    tag_mapping_function,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def tagged_sents(self, fileids=None, tagset=None):
        
        if tagset and tagset != self._tagset:
            tag_mapping_function = lambda t: map_tag(self._tagset, tagset, t)
        else:
            tag_mapping_function = None
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    True,
                    True,
                    False,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    tag_mapping_function,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def tagged_paras(self, fileids=None, tagset=None):
        
        if tagset and tagset != self._tagset:
            tag_mapping_function = lambda t: map_tag(self._tagset, tagset, t)
        else:
            tag_mapping_function = None
        return concat(
            [
                TaggedCorpusView(
                    fileid,
                    enc,
                    True,
                    True,
                    True,
                    self._sep,
                    self._word_tokenizer,
                    self._sent_tokenizer,
                    self._para_block_reader,
                    tag_mapping_function,
                )
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )


class CategorizedTaggedCorpusReader(CategorizedCorpusReader, TaggedCorpusReader):
    

    def __init__(self, *args, **kwargs):
        
        CategorizedCorpusReader.__init__(self, kwargs)
        TaggedCorpusReader.__init__(self, *args, **kwargs)

    def tagged_words(self, fileids=None, categories=None, tagset=None):
        return super().tagged_words(self._resolve(fileids, categories), tagset)

    def tagged_sents(self, fileids=None, categories=None, tagset=None):
        return super().tagged_sents(self._resolve(fileids, categories), tagset)

    def tagged_paras(self, fileids=None, categories=None, tagset=None):
        return super().tagged_paras(self._resolve(fileids, categories), tagset)


class TaggedCorpusView(StreamBackedCorpusView):
    

    def __init__(
        self,
        corpus_file,
        encoding,
        tagged,
        group_by_sent,
        group_by_para,
        sep,
        word_tokenizer,
        sent_tokenizer,
        para_block_reader,
        tag_mapping_function=None,
    ):
        self._tagged = tagged
        self._group_by_sent = group_by_sent
        self._group_by_para = group_by_para
        self._sep = sep
        self._word_tokenizer = word_tokenizer
        self._sent_tokenizer = sent_tokenizer
        self._para_block_reader = para_block_reader
        self._tag_mapping_function = tag_mapping_function
        StreamBackedCorpusView.__init__(self, corpus_file, encoding=encoding)

    def read_block(self, stream):
        
        block = []
        for para_str in self._para_block_reader(stream):
            para = []
            for sent_str in self._sent_tokenizer.tokenize(para_str):
                sent = [
                    str2tuple(s, self._sep)
                    for s in self._word_tokenizer.tokenize(sent_str)
                ]
                if self._tag_mapping_function:
                    sent = [(w, self._tag_mapping_function(t)) for (w, t) in sent]
                if not self._tagged:
                    sent = [w for (w, t) in sent]
                if self._group_by_sent:
                    para.append(sent)
                else:
                    para.extend(sent)
            if self._group_by_para:
                block.append(para)
            else:
                block.extend(para)
        return block



class MacMorphoCorpusReader(TaggedCorpusReader):
    

    def __init__(self, root, fileids, encoding="utf8", tagset=None):
        TaggedCorpusReader.__init__(
            self,
            root,
            fileids,
            sep="_",
            word_tokenizer=LineTokenizer(),
            sent_tokenizer=RegexpTokenizer(".*\n"),
            para_block_reader=self._read_block,
            encoding=encoding,
            tagset=tagset,
        )

    def _read_block(self, stream):
        return read_regexp_block(stream, r".*", r".*_\.")


class TimitTaggedCorpusReader(TaggedCorpusReader):
    

    def __init__(self, *args, **kwargs):
        TaggedCorpusReader.__init__(
            self, para_block_reader=read_timit_block, *args, **kwargs
        )

    def paras(self):
        raise NotImplementedError("use sents() instead")

    def tagged_paras(self):
        raise NotImplementedError("use tagged_sents() instead")