<jupyter_start><jupyter_text>Accessing Text Corpora and Lexical Resources<jupyter_code>from nltk.book import *<jupyter_output>*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by <NAME> 1851
text2: Sense and Sensibility by <NAME> 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . <NAME> 1908<jupyter_text>Accessing Text Corpora  NLTK includes a small selection of texts from the Project Gutenberg electronic textarchive, which contains some 25,000 free electronic books, hosted at http://www.gutenberg.org/.<jupyter_code>import nltk
nltk.corpus.gutenberg.fileids()
emma = nltk.corpus.gutenberg.words('austen-emma.txt')
from __future__ import division

for fileid in gutenberg.fileids():
    num_chars = len(gutenberg.raw(fileid))
    num_words = len(gutenberg.words(fileid))
    num_sents = len(gutenberg.sents(fileid))
    num_vocab = len(set([w.lower() for w in gutenberg.words(fileid)]))
    print(int(num_chars/num_words), int(num_words/num_sents), int(num_words/num_vocab),fileid)<jupyter_output>4 24 26 austen-emma.txt
4 26 16 austen-persuasion.txt
4 28 22 austen-sense.txt
4 33 79 bible-kjv.txt
4 19 5 blake-poems.txt
4 19 14 bryant-stories.txt
4 17 12 burgess-busterbrown.txt
4 20 12 carroll-alice.txt
4 20 11 chesterton-ball.txt
4 22 11 chesterton-brown.txt
4 18 10 chesterton-thursday.txt
4 20 24 edgeworth-parents.txt
4 25 15 melville-moby_dick.txt
4 52 10 milton-paradise.txt
4 11 8 shakespeare-caesar.txt
4 12 7 shakespeare-hamlet.txt
4 12 6 shakespeare-macbeth.txt
4 36 12 whitman-leaves.txt<jupyter_text>The raw() function gives us the contents of the file withoutany linguistic processing. So, for example, len(gutenberg.raw('blake-poems.txt') tellsus how many letters occur in the text, including the spaces between words. Thesents() function divides the text up into its sentences, where each sentence is a list ofwords:<jupyter_code>macbeth_sentences = gutenberg.sents('shakespeare-macbeth.txt')
macbeth_sentences
macbeth_sentences[1037]
longest_len = max([len(s) for s in macbeth_sentences])
[s for s in macbeth_sentences if len(s) == longest_len]<jupyter_output><empty_output><jupyter_text>Web and Chat Text  NLTK’s small collectionof web text includes content from a Firefox discussion forum, conversationsoverheard in New York, the movie script of Pirates of the Carribean, personal advertisements,and wine reviews:<jupyter_code>from nltk.corpus import webtext
for fileid in webtext.fileids():
    print(fileid, webtext.raw(fileid)[:65], '...')<jupyter_output>firefox.txt Cookie Manager: "Don't allow sites that set removed cookies to se ...
grail.txt SCENE 1: [wind] [clop clop clop] 
KING ARTHUR: Whoa there!  [clop ...
overheard.txt White guy: So, do you have any plans for this evening?
Asian girl ...
pirates.txt PIRATES OF THE CARRIBEAN: DEAD MAN'S CHEST, by <NAME> & Terr ...
singles.txt 25 SEXY MALE, seeks attrac older single lady, for discreet encoun ...
wine.txt Lovely delicate, fragrant Rhone wine. Polished leather and strawb ...<jupyter_text>There is also a corpus of instant messaging chat sessions, originally collected by theNaval Postgraduate School for research on automatic detection of Internet predators.<jupyter_code>from nltk.corpus import nps_chat
chatroom = nps_chat.posts('10-19-20s_706posts.xml')
chatroom[123]<jupyter_output><empty_output><jupyter_text>Brown Corpus  The Brown Corpus was the first million-word electronic corpus of English, created in1961 at Brown University. This corpus contains text from 500 sources, and the sourceshave been categorized by genre, such as news, editorial, and so on. Table 2-1 gives anexample of each genre (for a complete list, see http://icame.uib.no/brown/bcm-los.html).<jupyter_code>from nltk.corpus import brown
brown.categories()
brown.words(categories='news')
brown.words(fileids=['cg22'])
brown.sents(categories=['news', 'editorial', 'reviews'])
from nltk.corpus import brown
news_text = brown.words(categories='news')
fdist = nltk.FreqDist([w.lower() for w in news_text])
modals = ['can', 'could', 'may', 'might', 'must', 'will']

for m in modals:
    print(m + ':', fdist[m])
modals = ['what', 'when', 'where','who', 'why']

for m in modals:
    print(m + ':', fdist[m])
cfd = nltk.ConditionalFreqDist(
    (genre, word)
    for genre in brown.categories()
    for word in brown.words(categories=genre))

genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor']
modals = ['can', 'could', 'may', 'might', 'must', 'will']
cfd.tabulate(conditions=genres, samples=modals)<jupyter_output>can could   may might  must  will 
           news    93    86    66    38    50   389 
       religion    82    59    78    12    54    71 
        hobbies   268    58   131    22    83   264 
science_fiction    16    49     4    12     8    16 
        romance    74   193    11    51    45    43 
          humor    16    30     8     8     9    13<jupyter_text>Reuters Corpus  The Reuters Corpus contains 10,788 news documents totaling 1.3 million words. Thedocuments have been classified into 90 topics, and grouped into two sets, called “training”and “test”; thus, the text with fileid 'test/14826' is a document drawn from thetest set. This split is for training and testing algorithms that automatically detect thetopic of a document, as we will see in Chapter 6.<jupyter_code>from nltk.corpus import reuters
reuters.fileids()
reuters.categories()
reuters.categories('training/9865')
 reuters.categories(['training/9865', 'training/9880'])
reuters.fileids('barley')
reuters.fileids(['barley', 'corn'])
#['test/14832', 'test/14858', 'test/15033', 'test/15043', 'test/15106',
#'test/15287', 'test/15341', 'test/15618', 'test/15618', 'test/15648', ...]
reuters.words('training/9865')[:14]
reuters.words(['training/9865', 'training/9880'])
#['FRENCH', 'FREE', 'MARKET', 'CEREAL', 'EXPORT', ...]
reuters.words(categories='barley')
#['FRENCH', 'FREE', 'MARKET', 'CEREAL', 'EXPORT', ...]
reuters.words(categories=['barley', 'corn'])
#['THAI', 'TRADE', 'DEFICIT', 'WIDENS', 'IN', 'FIRST', ...]<jupyter_output><empty_output><jupyter_text>Inaugural Address Corpus<jupyter_code>from nltk.corpus import inaugural
inaugural.fileids()
#['1789-Washington.txt', '1793-Washington.txt', '1797-Adams.txt', ...]
[fileid[:4] for fileid in inaugural.fileids()]
#['1789', '1793', '1797', '1801', '1805', '1809', '1813', '1817', '1821', ...]
#To get the year out of the filename, we extracted the first four characters, using fileid[:4].
cfd = nltk.ConditionalFreqDist(
    (target, fileid[:4]) for fileid in inaugural.fileids() for w in inaugural.words(fileid) for target in ['america', 'citizen'] if w.lower().startswith(target))

cfd.plot()<jupyter_output><empty_output><jupyter_text>Annotated Text Corpora<jupyter_code>nltk.corpus.cess_esp.words()
#['El', 'grupo', 'estatal', 'Electricit\xe9_de_France', ...]
nltk.corpus.floresta.words()
#['Um', 'revivalismo', 'refrescante', 'O', '7_e_Meio', ...]
nltk.corpus.indian.words('hindi.pos')
#['\xe0\xa4\xaa\xe0\xa5\x82\xe0\xa4\xb0\xe0\xa5\x8d\xe0\xa4\xa3',
#'\xe0\xa4\xaa\xe0\xa5\x8d\xe0\xa4\xb0\xe0\xa4\xa4\xe0\xa4\xbf\xe0\xa4\xac\xe0\xa4
#\x82\xe0\xa4\xa7', ...]
nltk.corpus.udhr.fileids()
#['Abkhaz-Cyrillic+Abkh', 'Abkhaz-UTF8', 'Achehnese-Latin1', 'Achuar-Shiwiar-Latin1',
#'Adja-UTF8', 'Afaan_Oromo_Oromiffa-Latin1', 'Afrikaans-Latin1', 'Aguaruna-Latin1',
#'Akuapem_Twi-UTF8', 'Albanian_Shqip-Latin1', 'Amahuaca', 'Amahuaca-Latin1', ...]
nltk.corpus.udhr.words('Javanese-Latin1')[11:]
#[u'Saben', u'umat', u'manungsa', u'lair', u'kanthi', ...]<jupyter_output><empty_output><jupyter_text>Let’s use a conditional frequencydistribution to examine the differences in word lengths for a selection of languagesincluded in the udhr corpus. The output is shown in Figure 2-2 (run the program yourselfto see a color plot). Note that True and False are Python’s built-in Boolean values.<jupyter_code>from nltk.corpus import udhr
languages = ['Chickasaw', 'English', 'German_Deutsch','Greenlandic_Inuktikut', 'Hungarian_Magyar', 'Ibibio_Efik']
cfd = nltk.ConditionalFreqDist(
    (lang, len(word))
    for lang in languages
    for word in udhr.words(lang + '-Latin1'))

cfd.plot(cumulative=True)
raw_text= udhr.raw('German_Deutsch'+ '-Latin1')
nltk.FreqDist(raw_text).plot()<jupyter_output><empty_output><jupyter_text>Text Corpus Structure  Basic corpus functionality defined in NLTK: More documentation can be found usinghelp(nltk.corpus.reader) and by reading the online Corpus HOWTO at http://www.nltk.org/howto. |Example |Description||-------|---------||fileids() |The files of the corpus||fileids([categories])| The files of the corpus corresponding to these categories||categories()| The categories of the corpus||categories([fileids])| The categories of the corpus corresponding to these files||raw()| The raw content of the corpus||raw(fileids=[f1,f2,f3]) |The raw content of the specified files||raw(categories=[c1,c2]) |The raw content of the specified categories||words()| The words of the whole corpus||words(fileids=[f1,f2,f3])| The words of the specified fileids||words(categories=[c1,c2])| The words of the specified categories||sents()| The sentences of the specified categories||sents(fileids=[f1,f2,f3])| The sentences of the specified fileids||sents(categories=[c1,c2]) |The sentences of the specified categories||abspath(fileid) |The location of the given file on disk||encoding(fileid)| The encoding of the file (if known)||open(fileid) |Open a stream for reading the given corpus file||root()| The path to the root of locally installed corpus||readme() |The contents of the README file of the corpus|<jupyter_code>raw = gutenberg.raw("burgess-busterbrown.txt")
raw[1:20]
words = gutenberg.words("burgess-busterbrown.txt")
words[1:20]
sents = gutenberg.sents("burgess-busterbrown.txt")
sents[1:20]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1314540059347181, 'fraction_numerical': 0.043719090009891194, 'mean_word_length': 5.176542455711668, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '618799', 'license': 'MIT', 'n_tokens_mistral': 3845, 'n_tokens_neox': 3482, 'n_words': 1018, 'path': 'NLP with Python/Chapter02/Chapter2 Part 1.ipynb', 'repo_name': 'Stella-Sirius/NLP-Learning', 'size': '10137'}