natural language toolkit plaintext corpus reader c 20012023 nltk project steven bird stevenbird1gmail com edward loper edlopergmail com nitin madnani nmadnaniumiacs umd edu url https www nltk org for license information see license txt a reader for corpora that consist of plaintext documents reader for corpora that consist of plaintext documents paragraphs are assumed to be split using blank lines sentences and words can be tokenized using the default tokenizers or by custom tokenizers specified as parameters to the constructor this corpus reader can be customized e g to skip preface sections of specific document formats by creating a subclass and overriding the corpusview class variable the corpus view class used by this reader subclasses of plaintextcorpusreader may specify alternative corpus view classes e g to skip the preface sections of documents def init self root fileids wordtokenizerwordpuncttokenizer senttokenizernltk data lazyloadertokenizerspunktenglish pickle parablockreaderreadblanklineblock encodingutf8 r construct a new plaintext corpus reader for a set of documents located at the given root directory example usage root usrlocalsharenltkdatacorporawebtext reader plaintextcorpusreaderroot txt doctest skip param root the root directory for this corpus param fileids a list or regexp specifying the fileids in this corpus param wordtokenizer tokenizer for breaking sentences or paragraphs into words param senttokenizer tokenizer for breaking paragraphs into words param parablockreader the block reader used to divide the corpus into paragraph blocks return the given files as a list of words and punctuation symbols rtype liststr return the given files as a list of sentences or utterances each encoded as a list of word strings rtype listliststr return the given files as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word strings rtype listlistliststr a reader for plaintext corpora whose documents are divided into categories based on their file identifiers initialize the corpus reader categorization arguments catpattern catmap and catfile are passed to the categorizedcorpusreader constructor the remaining arguments are passed to the plaintextcorpusreader constructor fixme is there a better way how to not hardcode this possibly add a language kwargs to categorizedplaintextcorpusreader to override the senttokenizer reader for europarl corpora that consist of plaintext documents documents are divided into chapters instead of paragraphs as for regular plaintext documents chapters are separated using blank lines everything is inherited from plaintextcorpusreader except that since the corpus is preprocessed and pretokenized the word tokenizer should just split the line at whitespaces for the same reason the sentence tokenizer should just split the paragraph at line breaks there is a new chapters method that returns chapters instead instead of paragraphs the paras method inherited from plaintextcorpusreader is made nonfunctional to remove any confusion between chapters and paragraphs for europarl return the given files as a list of chapters each encoded as a list of sentences which are in turn encoded as lists of word strings rtype listlistliststr natural language toolkit plaintext corpus reader c 2001 2023 nltk project steven bird stevenbird1 gmail com edward loper edloper gmail com nitin madnani nmadnani umiacs umd edu url https www nltk org for license information see license txt a reader for corpora that consist of plaintext documents reader for corpora that consist of plaintext documents paragraphs are assumed to be split using blank lines sentences and words can be tokenized using the default tokenizers or by custom tokenizers specified as parameters to the constructor this corpus reader can be customized e g to skip preface sections of specific document formats by creating a subclass and overriding the corpusview class variable the corpus view class used by this reader subclasses of plaintextcorpusreader may specify alternative corpus view classes e g to skip the preface sections of documents construct a new plaintext corpus reader for a set of documents located at the given root directory example usage root usr local share nltk_data corpora webtext reader plaintextcorpusreader root txt doctest skip param root the root directory for this corpus param fileids a list or regexp specifying the fileids in this corpus param word_tokenizer tokenizer for breaking sentences or paragraphs into words param sent_tokenizer tokenizer for breaking paragraphs into words param para_block_reader the block reader used to divide the corpus into paragraph blocks return the given file s as a list of words and punctuation symbols rtype list str return the given file s as a list of sentences or utterances each encoded as a list of word strings rtype list list str return the given file s as a list of paragraphs each encoded as a list of sentences which are in turn encoded as lists of word strings rtype list list list str read 20 lines at a time a reader for plaintext corpora whose documents are divided into categories based on their file identifiers initialize the corpus reader categorization arguments cat_pattern cat_map and cat_file are passed to the categorizedcorpusreader constructor the remaining arguments are passed to the plaintextcorpusreader constructor fixme is there a better way how to not hardcode this possibly add a language kwargs to categorizedplaintextcorpusreader to override the sent_tokenizer reader for europarl corpora that consist of plaintext documents documents are divided into chapters instead of paragraphs as for regular plaintext documents chapters are separated using blank lines everything is inherited from plaintextcorpusreader except that since the corpus is pre processed and pre tokenized the word tokenizer should just split the line at whitespaces for the same reason the sentence tokenizer should just split the paragraph at line breaks there is a new chapters method that returns chapters instead instead of paragraphs the paras method inherited from plaintextcorpusreader is made non functional to remove any confusion between chapters and paragraphs for europarl read 20 lines at a time return the given file s as a list of chapters each encoded as a list of sentences which are in turn encoded as lists of word strings rtype list list list str
import nltk.data
from nltk.corpus.reader.api import *
from nltk.corpus.reader.util import *
from nltk.tokenize import *


class PlaintextCorpusReader(CorpusReader):
    

    CorpusView = StreamBackedCorpusView
    

    def __init__(
        self,
        root,
        fileids,
        word_tokenizer=WordPunctTokenizer(),
        sent_tokenizer=nltk.data.LazyLoader("tokenizers/punkt/english.pickle"),
        para_block_reader=read_blankline_block,
        encoding="utf8",
    ):
        r
        CorpusReader.__init__(self, root, fileids, encoding)
        self._word_tokenizer = word_tokenizer
        self._sent_tokenizer = sent_tokenizer
        self._para_block_reader = para_block_reader

    def words(self, fileids=None):
        
        return concat(
            [
                self.CorpusView(path, self._read_word_block, encoding=enc)
                for (path, enc, fileid) in self.abspaths(fileids, True, True)
            ]
        )

    def sents(self, fileids=None):
        
        if self._sent_tokenizer is None:
            raise ValueError("No sentence tokenizer for this corpus")

        return concat(
            [
                self.CorpusView(path, self._read_sent_block, encoding=enc)
                for (path, enc, fileid) in self.abspaths(fileids, True, True)
            ]
        )

    def paras(self, fileids=None):
        
        if self._sent_tokenizer is None:
            raise ValueError("No sentence tokenizer for this corpus")

        return concat(
            [
                self.CorpusView(path, self._read_para_block, encoding=enc)
                for (path, enc, fileid) in self.abspaths(fileids, True, True)
            ]
        )

    def _read_word_block(self, stream):
        words = []
        for i in range(20):  
            words.extend(self._word_tokenizer.tokenize(stream.readline()))
        return words

    def _read_sent_block(self, stream):
        sents = []
        for para in self._para_block_reader(stream):
            sents.extend(
                [
                    self._word_tokenizer.tokenize(sent)
                    for sent in self._sent_tokenizer.tokenize(para)
                ]
            )
        return sents

    def _read_para_block(self, stream):
        paras = []
        for para in self._para_block_reader(stream):
            paras.append(
                [
                    self._word_tokenizer.tokenize(sent)
                    for sent in self._sent_tokenizer.tokenize(para)
                ]
            )
        return paras


class CategorizedPlaintextCorpusReader(CategorizedCorpusReader, PlaintextCorpusReader):
    

    def __init__(self, *args, **kwargs):
        
        CategorizedCorpusReader.__init__(self, kwargs)
        PlaintextCorpusReader.__init__(self, *args, **kwargs)





class PortugueseCategorizedPlaintextCorpusReader(CategorizedPlaintextCorpusReader):
    def __init__(self, *args, **kwargs):
        CategorizedCorpusReader.__init__(self, kwargs)
        kwargs["sent_tokenizer"] = nltk.data.LazyLoader(
            "tokenizers/punkt/portuguese.pickle"
        )
        PlaintextCorpusReader.__init__(self, *args, **kwargs)


class EuroparlCorpusReader(PlaintextCorpusReader):

    

    def _read_word_block(self, stream):
        words = []
        for i in range(20):  
            words.extend(stream.readline().split())
        return words

    def _read_sent_block(self, stream):
        sents = []
        for para in self._para_block_reader(stream):
            sents.extend([sent.split() for sent in para.splitlines()])
        return sents

    def _read_para_block(self, stream):
        paras = []
        for para in self._para_block_reader(stream):
            paras.append([sent.split() for sent in para.splitlines()])
        return paras

    def chapters(self, fileids=None):
        
        return concat(
            [
                self.CorpusView(fileid, self._read_para_block, encoding=enc)
                for (fileid, enc) in self.abspaths(fileids, True)
            ]
        )

    def paras(self, fileids=None):
        raise NotImplementedError(
            "The Europarl corpus reader does not support paragraphs. Please use chapters() instead."
        )