<jupyter_start><jupyter_text>1. ○ Create a variable phrase containing a list of words. Experiment with the operationsdescribed in this chapter, including addition, multiplication, indexing, slicing,and sorting.<jupyter_code>a = ['Create', 'a', 'variable', 'phrase', 'containing', 'a', 'list', 'of', 'words']
a[3]
print(a*4)
a.index('a')
a.count('a')
a.sort()
print(a)<jupyter_output>['Create', 'a', 'a', 'containing', 'list', 'of', 'phrase', 'variable', 'words']<jupyter_text>2. ○ Use the corpus module to explore austen-persuasion.txt. How many wordtokens does this book have? How many word types?<jupyter_code>emma = gutenberg.words('austen-persuasion.txt')

print(len(emma))
len(sorted(set([w.lower() for w in emma])))<jupyter_output>98171<jupyter_text>3. ○ Use the Brown Corpus reader nltk.corpus.brown.words() or the Web Text Corpusreader nltk.corpus.webtext.words() to access some sample text in two differentgenres.<jupyter_code>nltk.corpus.webtext.fileids()
nltk.corpus.webtext.words('wine.txt')<jupyter_output><empty_output><jupyter_text>4. ○ Read in the texts of the State of the Union addresses, using the state_union corpus reader. Count occurrences of men, women, and people in each document. What hashappened to the usage of these words over time?<jupyter_code>from nltk.corpus import state_union
cfd = nltk.ConditionalFreqDist(
    (target, fileid[:4])
    for fileid in state_union.fileids()
    for w in state_union.words(fileid)
    for target in ['men', 'women', 'people']
    if w.lower().startswith(target))
cfd.plot()<jupyter_output><empty_output><jupyter_text>5. ○ Investigate the holonym-meronym relations for some nouns. Remember thatthere are three kinds of holonym-meronym relation, so you need to use member_meronyms(), part_meronyms(), substance_meronyms(), member_holonyms(),part_holonyms(), and substance_holonyms().<jupyter_code>wn.synset('piano.n.01').part_meronyms()
wn.synset('table.n.01').member_meronyms()
wn.synset('cloth.n.01').substance_meronyms()
wn.synset('letter.n.01').member_holonyms()
wn.synset('letter.n.01').part_holonyms()
wn.synsets('piano')
wn.synset('piano.n.01').lemma_names()
wn.synset('piano.n.02').lemma_names()
wn.synset('piano.n.01').definition()
for synset in wn.synsets('piano'):
    print(synset.lemma_names())
piano= wn.synset('piano.n.01')
piano_t = piano.hyponyms()
sorted(lemma.name() for synset in piano_t for lemma in synset.lemmas())<jupyter_output><empty_output><jupyter_text>6. ○ In the discussion of comparative wordlists, we created an object called translate, which you could look up using words in both German and Italian in order to get corresponding words in English. What problem might arise with this approach?Can you suggest a way to avoid this problem?<jupyter_code>from nltk.corpus import swadesh
de2en = swadesh.entries(['de', 'en'])
es2en = swadesh.entries(['es', 'en']) 
de2en = [(d + '-de', e) for (d, e) in swadesh.entries(['de', 'en'])]
translate = dict(de2en)
translate.update(dict(de2en))
translate.update(dict(es2en))
translate['Hund-de']<jupyter_output><empty_output><jupyter_text>7. ○ According to Strunk and White’s Elements of Style, the word however, used atthe start of a sentence, means “in whatever way” or “to whatever extent,” and not“nevertheless.” They give this example of correct usage: However you advise him,he will probably do as he thinks best. (http://www.bartleby.com/141/strunk3.html)Use the concordance tool to study actual usage of this word in the various texts wehave been considering. See also the LanguageLog posting “Fossilized prejudicesabout ‘however’” at http://itre.cis.upenn.edu/~myl/languagelog/archives/001913.html.<jupyter_code>from nltk.corpus import inaugural
text = nltk.Text(nltk.corpus.inaugural.words())
text.concordance("however")
#rule follows Strunk and White’s Elements of Style<jupyter_output><empty_output><jupyter_text>8. ◑ Define a conditional frequency distribution over the Names Corpus that allowsyou to see which initial letters are more frequent for males versus females (seeFigure 2-7).<jupyter_code>names = nltk.corpus.names
cfd = nltk.ConditionalFreqDist(
    (fileid, name[0])
    for fileid in names.fileids()
    for name in names.words(fileid))
cfd.plot()<jupyter_output><empty_output><jupyter_text>9. ◑ Pick a pair of texts and study the differences between them, in terms of vocabulary,vocabulary richness, genre, etc. Can you find pairs of words that have quitedifferent meanings across the two texts, such as monstrous in Moby Dick and inSense and Sensibility?<jupyter_code>def lexical_diversity(text):
    return len(set(text)) / len(text)
print(lexical_diversity('1789-Washington.txt'))
print(lexical_diversity('1941-Roosevelt.txt'))
print(lexical_diversity('2017-Trump.txt'))
print(lexical_diversity('2013-Obama.txt'))
print(lexical_diversity('2013-Obama.txt'))
Was_text = nltk.Text(inaugural.words('1789-Washington.txt'))
Oba_text = nltk.Text(inaugural.words('2013-Obama.txt'))
austen = nltk.Text(gutenberg.words('austen-persuasion.txt'))
Was_text.concordance("country")
Oba_text.concordance("country")
austen.concordance("country")
nltk.Text(Was_text).similar('God')
nltk.Text(Oba_text).similar('God')
nltk.Text(austen).similar('God')
nltk.Text(Was_text).similar('country')
nltk.Text(Oba_text).similar('country')
nltk.Text(austen).similar('country')<jupyter_output>moment wives
time world consequence room other way same astonishment window day
best acquaintance idea town subject others question case gentleman
pleasure<jupyter_text>10. ◑ Read the BBC News article: “UK’s Vicky Pollards ‘left behind’” at http://news.bbc.co.uk/1/hi/education/6173441.stm. The article gives the following statisticabout teen language: “the top 20 words used, including yeah, no, but and like,account for around a third of all words.” How many word types account for a thirdof all word tokens, for a variety of text sources? What do you conclude about thisstatistic? Read more about this on LanguageLog, at http://itre.cis.upenn.edu/~myl/languagelog/archives/003993.html.<jupyter_code>#everyday language is much less diverse than written. 
#Moreover, it is not a surprise, that language of young people is not as diverse. 
#The thing is that earlier, there were no means to investigate the issue, 
#since the written text is usually novels written by educated people.<jupyter_output><empty_output><jupyter_text>11. ◑ Investigate the table of modal distributions and look for other patterns. Try toexplain them in terms of your own impressionistic understanding of the differentgenres. Can you find other closed classes of words that exhibit significant differencesacross different genres? 12. ◑ The CMU Pronouncing Dictionary contains multiple pronunciations for certainwords. How many distinct words does it contain? What fraction of words in thisdictionary have more than one possible pronunciation?<jupyter_code>prondict = nltk.corpus.cmudict.dict()
words = 0
for key in prondict:
    if len(prondict[key]) > 1:
        words +=1
print(words)
print(len(prondict))
print(words/ len(prondict))<jupyter_output>9241
123455
0.07485318537118789<jupyter_text>13. ◑ What percentage of noun synsets have no hyponyms? You can get all noun synsetsusing wn.all_synsets('n').<jupyter_code>all_s = list(wn.all_synsets('n'))
hyp = [s for s in all_s if s.hyponyms() == []]
print((len(hyp)/len(all_s))*100)<jupyter_output>79.67119283931072<jupyter_text>14. ◑ Define a function supergloss(s) that takes a synset s as its argument and returnsa string consisting of the concatenation of the definition of s, and the definitionsof all the hypernyms and hyponyms of s.<jupyter_code>def supergloss(s):
    word_def = s.definition() + '\n'
    for hypernym in s.hypernyms():
        word_def = word_def + hypernym.name() + ': ' + hypernym.definition() + '\n'
    for hyponym in s.hyponyms():
        word_def = word_def + hyponym.name() + ': ' + hyponym.definition() + '\n'
    return word_def
print(supergloss(wn.synset('piano.n.01')))<jupyter_output>a keyboard instrument that is played by depressing keys that cause hammers to strike tuned strings and produce sounds
keyboard_instrument.n.01: a musical instrument that is played by means of a keyboard
percussion_instrument.n.01: a musical instrument in which the sound is produced by one object striking another
stringed_instrument.n.01: a musical instrument in which taut strings provide the source of sound
grand_piano.n.01: a piano with the strings on a horizontal harp-shaped frame; usually supported by three legs
mechanical_piano.n.01: a mechanically operated piano that uses a roll of perforated paper to activate the keys
upright.n.02: a piano with a vertical sounding board<jupyter_text>15. ◑ Write a program to find all words that occur at least three times in the BrownCorpus.<jupyter_code>freq = FreqDist(w.lower() for w in brown.words() if w.isalpha())
sorted(w for w in set(brown.words()) if freq[w] >= 3)<jupyter_output><empty_output><jupyter_text>16. ◑ Write a program to generate a table of lexical diversity scores (i.e., token/typeratios), as we saw in Table 1-1. Include the full set of Brown Corpus genres (nltk.corpus.brown.categories()). Which genre has the lowest diversity (greatest number of tokens per type)? Is this what you would have expected?<jupyter_code>def lexical_diversity(text):
    return len(set(text)) / len(text)
for genre in nltk.corpus.brown.categories():
    print(genre, lexical_diversity(brown.words(categories=genre)))<jupyter_output>adventure 0.1279743878169075
belles_lettres 0.10642071451679992
editorial 0.16054152327770924
fiction 0.1358194136199042
government 0.11667641228232811
hobbies 0.14493897625842492
humor 0.23125144042406084
learned 0.09268890745953554
lore 0.13148804612915801
mystery 0.12212912592488936
news 0.14314696580941583
religion 0.1617553745018909
reviews 0.21192020440251572
romance 0.12070492131044529
science_fiction 0.22342778161713892<jupyter_text>17. ◑ Write a function that finds the 50 most frequently occurring words of a text that are not stopwords.<jupyter_code>def frequent_words(text):
    stopwords = nltk.corpus.stopwords.words('english')
    freq = FreqDist(w for w in text if w.lower() not in stopwords and w.isalpha())
    return freq.most_common(50)

print(frequent_words(emma))<jupyter_output>[('Anne', 497), ('could', 444), ('would', 351), ('Captain', 297), ('Mrs', 291), ('Elliot', 288), ('Mr', 256), ('must', 228), ('one', 221), ('Wentworth', 218), ('much', 205), ('Lady', 191), ('good', 181), ('little', 175), ('said', 173), ('Charles', 166), ('might', 166), ('never', 153), ('time', 151), ('think', 149), ('Russell', 148), ('Sir', 144), ('Walter', 141), ('well', 141), ('Mary', 138), ('man', 133), ('nothing', 131), ('Musgrove', 130), ('know', 127), ('Miss', 123), ('see', 123), ('soon', 121), ('father', 117), ('great', 116), ('first', 112), ('two', 112), ('Louisa', 111), ('quite', 110), ('always', 109), ('though', 107), ('without', 106), ('Bath', 99), ('every', 96), ('made', 96), ('long', 95), ('say', 93), ('seemed', 93), ('thought', 90), ('Elizabeth', 89), ('last', 88)]<jupyter_text>18. ◑ Write a program to print the 50 most frequent bigrams (pairs of adjacent words)of a text, omitting bigrams that contain stopwords.<jupyter_code>def frequent_bigrams(text):
    stopwords = nltk.corpus.stopwords.words('english')
    bigrams = list(nltk.bigrams(text))
    print(bigrams)
    freq = FreqDist(b for b in bigrams if b not in stopwords)
    return freq.most_common(50)<jupyter_output><empty_output><jupyter_text>19. ◑ Write a program to create a table of word frequencies by genre, like the one givenin Section 2.1 for modals. Choose your own words and try to find words whosepresence (or absence) is typical of a genre. Discuss your findings.<jupyter_code>cfd = nltk.ConditionalFreqDist(
    (genre, word)
    for genre in brown.categories()
    for word in brown.words(categories=genre))
genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor']
pron = ['I', 'you', 'we', 'she', 'he', 'it', 'they']
cfd.tabulate(conditions=genres, samples=pron)<jupyter_output>I  you   we  she   he   it they 
           news  179   55   77   42  451  363  205 
       religion  155  100  176   10  137  264  115 
        hobbies  154  383  100   21  155  476  177 
science_fiction   98   81   30   36  139  129   53 
        romance  951  456   78  496  702  573  168 
          humor  239  131   32   58  146  162   70<jupyter_text>20. ◑ Write a function word_freq() that takes a word and the name of a section of theBrown Corpus as arguments, and computes the frequency of the word in that sectionof the corpus.<jupyter_code>def word_freq(word, genre):
    f_d = nltk.FreqDist(brown.words(categories=genre))
    return f_d[word]
word_freq('We', 'news')<jupyter_output><empty_output><jupyter_text>21. ◑ Write a program to guess the number of syllables contained in a text, makinguse of the CMU Pronouncing Dictionary.<jupyter_code>def syll_numb(text):
    prondict = nltk.corpus.cmudict.dict()
    word = 0
    for w in text:
        if w in prondict.keys():
            word += len(prondict[w][0])
    print(word)

syll_numb(emma)<jupyter_output>265006<jupyter_text>22. ◑ Define a function hedge(text) that processes a text and produces a new versionwith the word 'like' between every third word. 23. ● Zipf’s Law: Let f(w) be the frequency of a word w in free text. Suppose that allthe words of a text are ranked according to their frequency, with the most frequentword first. Zipf’s Law states that the frequency of a word type is inverselyproportional to its rank (i.e., f × r = k, for some constant k). For example, the 50thmost common word type should occur three times as frequently as the 150th mostcommon word type.a. Write a function to process a large text and plot word frequency against wordrank using pylab.plot. Do you confirm Zipf’s law? (Hint: it helps to use alogarithmic scale.) What is going on at the extreme ends of the plotted line?b. Generate random text, e.g., using random.choice("abcdefg "), taking care toinclude the space character. You will need to import random first. Use the stringconcatenation operator to accumulate characters into a (very) long string.Then tokenize this string, generate the Zipf plot as before, and compare thetwo plots. What do you make of Zipf’s Law in the light of this? 24. ● Modify the text generation program in Example 2-1 further, to do the followingtasks: a. Store the n most likely words in a list words, then randomly choose a wordfrom the list using random.choice(). (You will need to import random first.)b. Select a particular genre, such as a section of the Brown Corpus or a Genesistranslation, one of the Gutenberg texts, or one of the Web texts. Train themodel on this corpus and get it to generate random text. You may have toexperiment with different start words. How intelligible is the text? Discuss thestrengths and weaknesses of this method of generating random text.c. Now train your system using two distinct genres and experiment with generatingtext in the hybrid genre. Discuss your observations.<jupyter_code>import random
def generate_model(text, n):
    fdist = FreqDist(text)
    likely = fdist.most_common(n)
    bigrams = nltk.bigrams(text)
    cfd = nltk.ConditionalFreqDist(bigrams)
    for i in range(n):
        print(random.choice(likely)[0], end=' ')

generate_model(emma, 100)<jupyter_output>Sir Mr from this ' Mrs a his this were so my " is and no ," which you might have little an to He his might this and that in an ," Lady was the , his said they which were that such who did should ; on never and He ," her me Captain Charles not It him were than as " his all Mrs is being no Charles ! such and no been than said an is any do ! It have Walter s than s in good could but think he Elliot will Mr was -<jupyter_text>25. ● Define a function find_language() that takes a string as its argument and returnsa list of languages that have that string as a word. Use the udhr corpus and limityour searches to files in the Latin-1 encoding.<jupyter_code>from nltk.corpus import udhr
def find_language(word):
    lang = []
    for fileid in udhr.fileids():
        if fileid.endswith('-Latin1') and word in udhr.words(fileid):
            lang.append(fileid)
    return lang
find_language("is")<jupyter_output><empty_output><jupyter_text>26. ● What is the branching factor of the noun hypernym hierarchy? I.e., for everynoun synset that has hyponyms—or children in the hypernym hierarchy—howmany do they have on average? You can get all noun synsets using wn.all_synsets('n').<jupyter_code>noun_hyp = 0
sum_hyp = 0
for synset in wn.all_synsets('n'):
    if synset.hyponyms() != []:
        noun_hyp += 1
        sum_hyp += len(synset.hyponyms())
print(sum_hyp/noun_hyp)<jupyter_output>4.543820763194153
{'fraction_non_alphanumeric': 0.10846245530393325, 'fraction_numerical': 0.045828367103694874, 'mean_word_length': 5.10440160058203, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '273319', 'license': 'MIT', 'n_tokens_mistral': 6065, 'n_tokens_neox': 5361, 'n_words': 1914, 'path': 'Ch 2.ipynb', 'repo_name': 'kalashnikova-elena/NLTK_Solutions', 'size': '16780'}