natural language toolkit categorized sentences corpus reader c 20012023 nltk project pierpaolo pantone 24alsecondogmail com url https www nltk org for license information see license txt corpusreader structured for corpora that contain one instance on each row this corpusreader is specifically used for the subjectivity dataset and the sentence polarity dataset subjectivity dataset information s bo pang and lillian lee url https www cs cornell edupeoplepabomoviereviewdata distributed with permission related papers bo pang and lillian lee a sentimental education sentiment analysis using subjectivity summarization based on minimum cuts proceedings of the acl 2004 sentence polarity dataset information s bo pang and lillian lee url https www cs cornell edupeoplepabomoviereviewdata related papers bo pang and lillian lee seeing stars exploiting class relationships for sentiment categorization with respect to rating scales proceedings of the acl 2005 a reader for corpora in which each row represents a single instance mainly a sentence istances are divided into categories based on their file identifiers see categorizedcorpusreader since many corpora allow rows that contain more than one sentence it is possible to specify a sentence tokenizer to retrieve all sentences instead than all rows examples using the subjectivity dataset from nltk corpus import subjectivity subjectivity sents23 doctest normalizewhitespace television made him famous but his biggest hits happened off screen subjectivity categories obj subj subjectivity wordscategories subj smart and alert thirteen examples using the sentence polarity dataset from nltk corpus import sentencepolarity sentencepolarity sents doctest normalizewhitespace simplistic silly and tedious it s so laddish and juvenile only teenage boys could possibly find it funny sentencepolarity categories neg pos param root the root directory for the corpus param fileids a list or regexp specifying the fileids in the corpus param wordtokenizer a tokenizer for breaking sentences or paragraphs into words default whitespacetokenizer param senttokenizer a tokenizer for breaking paragraphs into sentences param encoding the encoding that should be used to read the corpus param kwargs additional parameters passed to categorizedcorpusreader return all sentences in the corpus or in the specified files param fileids a list or regexp specifying the ids of the files whose sentences have to be returned param categories a list specifying the categories whose sentences have to be returned return the given files as a list of sentences each sentence is tokenized using the specified wordtokenizer rtype listliststr return all words and punctuation symbols in the corpus or in the specified files param fileids a list or regexp specifying the ids of the files whose words have to be returned param categories a list specifying the categories whose words have to be returned return the given files as a list of words and punctuation symbols rtype liststr natural language toolkit categorized sentences corpus reader c 2001 2023 nltk project pierpaolo pantone 24alsecondo gmail com url https www nltk org for license information see license txt corpusreader structured for corpora that contain one instance on each row this corpusreader is specifically used for the subjectivity dataset and the sentence polarity dataset subjectivity dataset information s bo pang and lillian lee url https www cs cornell edu people pabo movie review data distributed with permission related papers bo pang and lillian lee a sentimental education sentiment analysis using subjectivity summarization based on minimum cuts proceedings of the acl 2004 sentence polarity dataset information s bo pang and lillian lee url https www cs cornell edu people pabo movie review data related papers bo pang and lillian lee seeing stars exploiting class relationships for sentiment categorization with respect to rating scales proceedings of the acl 2005 a reader for corpora in which each row represents a single instance mainly a sentence istances are divided into categories based on their file identifiers see categorizedcorpusreader since many corpora allow rows that contain more than one sentence it is possible to specify a sentence tokenizer to retrieve all sentences instead than all rows examples using the subjectivity dataset from nltk corpus import subjectivity subjectivity sents 23 doctest normalize_whitespace television made him famous but his biggest hits happened off screen subjectivity categories obj subj subjectivity words categories subj smart and alert thirteen examples using the sentence polarity dataset from nltk corpus import sentence_polarity sentence_polarity sents doctest normalize_whitespace simplistic silly and tedious it s so laddish and juvenile only teenage boys could possibly find it funny sentence_polarity categories neg pos param root the root directory for the corpus param fileids a list or regexp specifying the fileids in the corpus param word_tokenizer a tokenizer for breaking sentences or paragraphs into words default whitespacetokenizer param sent_tokenizer a tokenizer for breaking paragraphs into sentences param encoding the encoding that should be used to read the corpus param kwargs additional parameters passed to categorizedcorpusreader return all sentences in the corpus or in the specified file s param fileids a list or regexp specifying the ids of the files whose sentences have to be returned param categories a list specifying the categories whose sentences have to be returned return the given file s as a list of sentences each sentence is tokenized using the specified word_tokenizer rtype list list str return all words and punctuation symbols in the corpus or in the specified file s param fileids a list or regexp specifying the ids of the files whose words have to be returned param categories a list specifying the categories whose words have to be returned return the given file s as a list of words and punctuation symbols rtype list str read 20 lines at a time
from nltk.corpus.reader.api import *
from nltk.tokenize import *


class CategorizedSentencesCorpusReader(CategorizedCorpusReader, CorpusReader):
    

    CorpusView = StreamBackedCorpusView

    def __init__(
        self,
        root,
        fileids,
        word_tokenizer=WhitespaceTokenizer(),
        sent_tokenizer=None,
        encoding="utf8",
        **kwargs
    ):
        

        CorpusReader.__init__(self, root, fileids, encoding)
        CategorizedCorpusReader.__init__(self, kwargs)
        self._word_tokenizer = word_tokenizer
        self._sent_tokenizer = sent_tokenizer

    def sents(self, fileids=None, categories=None):
        
        fileids = self._resolve(fileids, categories)
        if fileids is None:
            fileids = self._fileids
        elif isinstance(fileids, str):
            fileids = [fileids]
        return concat(
            [
                self.CorpusView(path, self._read_sent_block, encoding=enc)
                for (path, enc, fileid) in self.abspaths(fileids, True, True)
            ]
        )

    def words(self, fileids=None, categories=None):
        
        fileids = self._resolve(fileids, categories)
        if fileids is None:
            fileids = self._fileids
        elif isinstance(fileids, str):
            fileids = [fileids]
        return concat(
            [
                self.CorpusView(path, self._read_word_block, encoding=enc)
                for (path, enc, fileid) in self.abspaths(fileids, True, True)
            ]
        )

    def _read_sent_block(self, stream):
        sents = []
        for i in range(20):  
            line = stream.readline()
            if not line:
                continue
            if self._sent_tokenizer:
                sents.extend(
                    [
                        self._word_tokenizer.tokenize(sent)
                        for sent in self._sent_tokenizer.tokenize(line)
                    ]
                )
            else:
                sents.append(self._word_tokenizer.tokenize(line))
        return sents

    def _read_word_block(self, stream):
        words = []
        for sent in self._read_sent_block(stream):
            words.extend(sent)
        return words