<jupyter_start><jupyter_text>Part-of-Speech tagging, N-grams & Plotting  AssignmentUse the same text as last week (Wizard of Oz books from Gutenberg), or find and use another text source with at least 6 documents.  Make sure to clean up the text (lowercase, remove stopwords, etc) before looking for n-grams.1.	Perform part of speech tagging.   Bonus: What are some of the most frequently-used nouns and verbs?2.	Count 2-grams, 3-grams, and 4-grams instead of words (1-grams). Plot the Zipfian distribution of the n-grams. What are the top 20 phrases for the 2-grams?  Are 3- and 4-grams even useful here?3.	Create word clouds of the n-grams.4.	Choose some n-grams, and find the top most similar n-grams of the same length.  You could use string similarity, like Levenshtein distance, or you could use semantic similarity, like topic modeling (e.g. LSA), or you could use the .similarity() function from spaCy in Python, which uses word embeddings.5.	Bonus:  Look for interesting word collocations in your texts.  In NLTK (Python) this is under nltk.collocations. NLTK uses PMI (https://en.wikipedia.org/wiki/Pointwise_mutual_information) to find rare word collocations.  Imports<jupyter_code>import re  # python's regular expression package
import string #string operations
import nltk
from nltk.tokenize import sent_tokenize, word_tokenize #Tokeningation
from nltk.corpus import stopwords #stop words
from nltk.stem import WordNetLemmatizer #lemmatization
from nltk import FreqDist #frequency distribution
import matplotlib.pyplot as plt #plotting
from nltk import pos_tag #Part of speech tagging
from nltk import bigrams, trigrams, ngrams #bigrams, trigrams and n-grams
from wordcloud import WordCloud #for creating word clouds
from nltk.corpus import wordnet as wn #for finding similarity between words
from nltk.collocations import * #for bigram etc. collocation finders and associatution measures<jupyter_output><empty_output><jupyter_text>Functions<jupyter_code>def read(filepath):
    """Reads a text file and returns it as a string"""
    with open(filepath) as f:      
        return f.read()
def extract(raw_text):
    """Receives raw text as input, removes header and footer and returns content of the original book """
    regex = re.compile(r'\*\*\*(START|END)(.*)\*\*\*') #compile regex
    regex.findall(raw_text) #find all lines starting with ***START or ***END
    #find location of the strings marking the beginning and the end of the book
    mo = regex.search(raw_text) #match object for the first string
    start = mo.end() + 1  #Start where the ***START OF THE BOOK String ends
    mo = regex.search(raw_text, pos=start) #Perform second search begining where the first search ended 
    #mo (match object) contains the second string
    end = mo.start() #mark end of the main text at the begginning of the second string
    # slice out the content of the book (text between the two strings)
    book = raw_text[start:end+1] #book contains the main text
    return book

def tokenize_nostops(book_text):
    """Returns a list of cleaned, stemmed tokens"""
    #Tokenize into words and change to lower case
    tokens_lower = [word.lower() for word in word_tokenize(book_text)]
    # remove punctation 
    string.punctuation
    tokens_nopunct = [word for word in tokens_lower if word not in string.punctuation]
    #remove stop words
    stop_words = stopwords.words('english') #list of stop words
    tokens_nostop = [word for word in tokens_nopunct if word not in stop_words]
    #Lemmatization
    wnl = WordNetLemmatizer()
    tokens_nostop_lem  = [wnl.lemmatize(word) for word in tokens_nostop if word.isalpha()] #lemmatize only words with all aphapetic characters
    return tokens_nostop_lem<jupyter_output><empty_output><jupyter_text>Main<jupyter_code># path to the text file relative to the python script file 
filepath = 'wizard-of-oz.txt'
#Read the file from the disk
raw=read(filepath) #Call the read() function
#Display a slice of the text to make sure it is loaded correctly
raw[:800]
#Extract the main text of the book from the file
book_text=extract(raw) 
#Display a slice of the text to validate the result
book_text[:800]


#Text cleaning - resulting list of tokens without stop words
cleantokens_nostops = tokenize_nostops(book_text)
#Display partial results to check correctness
cleantokens_nostops[100:150]<jupyter_output><empty_output><jupyter_text>Part-of-Speech (POS) tagging<jupyter_code>#Perform part of speech tagging on cleaned list of token words
pos_words= pos_tag(cleantokens_nostops, tagset='universal')
#Display first 20 words with tags to check results
pos_words[:20]
#Find most common (word, tag) combinations (for all parts of speech)
pos_words_fd = FreqDist(pos_words)
common_pos= pos_words_fd.most_common()
common_pos[:20]
# challenge: function tinfding the most common words by part of speech

def find_most_common(fd, pos, num=10):
    """Find the most common part of speech in an NLTK FreqDist and return a list with given length"""
    tup_fd = nltk.FreqDist(tup for (tup, count) in fd)
    part_list = [word for (word, part) in tup_fd if pos in part]
    return part_list[:10]

# Find 10 most common verbs
found_verbs= find_most_common(common_pos, 'V')
found_verbs
# Find 10 most common nouns
found_nouns = find_most_common(common_pos, 'N')
found_nouns<jupyter_output><empty_output><jupyter_text>N-Grams<jupyter_code># count bigrams
bigrams= [tok for tok in bigrams(cleantokens_nostops)]
len(bigrams)
# count trigrams
trigrams= [tok for tok in trigrams(cleantokens_nostops)]
len(trigrams)
# count 4-grams
fourgrams = ngrams(cleantokens_nostops, 4)
len(list(fourgrams))

#Display first 10 bigrams
bigrams[:10]

#Display forst 10 trigrams
trigrams[:10]

# create a frequency distribution for bigrams
fd_bigrams = FreqDist(bigrams)
# plot frequency distribution for bigrams
plt.figure(figsize=(15, 5))
fd_bigrams.plot(50)<jupyter_output><empty_output><jupyter_text>The graph above shows that bigrams in this text do not follow Zipfian distribution. For example, the most popular bigram (tin, woodman) is reteated 20 times in the book. According to Zipfian distrbtuion, we would expect to encounter the second popular bigram about 10 times, but it is used 19 times. The thrid popular bigram is found 16 times instead of about 7.<jupyter_code>#Display 20 most common bigrams
fd_bigrams.most_common(20)
#Display 20 most common trigrams
fd_trigrams = FreqDist(trigrams)
fd_trigrams.most_common(20)
#Display 20 most common fourgrams
fd_fourgrams = FreqDist(ngrams(cleantokens_nostops, 4))
fd_fourgrams.most_common(20)<jupyter_output><empty_output><jupyter_text>Trigrams and fourgrams' distributions are even further away from the Zipfian distriibution.  The most popular trigram and fourgrams are found in the text only 3 times each. So, Zipfian distribution is not useful for analysing text in this case.   Create a Wordcloud<jupyter_code>#The following word cloud ewxample was adapted from: https://amueller.github.io/word_cloud/auto_examples/simple.html#sphx-glr-auto-examples-simple-py 

#Creating word cloud to visually present the most popular n-grams in the test

import os
from os import path
from wordcloud import WordCloud

# get data directory (using getcwd() is needed to support running example in generated IPython notebook)
d = path.dirname(__file__) if "__file__" in locals() else os.getcwd()

# Generate a word cloud image using the full cleaned text of the book
wordcloud = WordCloud().generate(book_text)

# Display the generated image:
# the matplotlib way:
import matplotlib.pyplot as plt
#plt.imshow(wordcloud, interpolation='bilinear')
#plt.axis("off")

# lower max_font_size
wordcloud = WordCloud(max_font_size=40).generate(book_text)
# make your figure larger and turn off the axes
plt.figure(figsize=(20, 10))
plt.axis('off')
plt.imshow(wordcloud, interpolation='bilinear')
plt.show()<jupyter_output><empty_output><jupyter_text>Similarity<jupyter_code># find the synset for 'woodman'
woodman_syn = wn.synsets('woodman')
woodman_syn<jupyter_output><empty_output><jupyter_text>The sysnet for "woodman" includes 2 words - woodsman and woodworker.<jupyter_code># find the synset for 'man'
man_syn = wn.synsets('man')
man_syn<jupyter_output><empty_output><jupyter_text>The sysnet for "man" is larger - 13 options.<jupyter_code># extract similarity on sysnet for nthe first elements for woodman and man
woodman = woodman_syn[0]
woodman.path_similarity(man_syn[0])<jupyter_output><empty_output><jupyter_text>The similarity score using path_similarity is relatively low - 0.16(6).<jupyter_code># Use nested for loops to compare the each synset of woodman and man
for wm in woodman_syn:
    for m in man_syn:
        score = wm.path_similarity(m)
        print("Similarity  score for : ", wm, "and ", m, " is ", score, ".")<jupyter_output>Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.01')  is  0.16666666666666666 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('serviceman.n.01')  is  0.14285714285714285 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.03')  is  0.2 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('homo.n.02')  is  0.07692307692307693 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.05')  is  0.125 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.06')  is  0.16666666666666666 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('valet.n.01')  is  0.125 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.08')  is  0.16666666666666666 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.09')  is  0.1 .
Similarity  score for :  Synset('woodsman.n.01') and  Synset('man.n.10')  is  0.08333333333333333 .
Similarity  score for :  Synset('woodsman.n.01') a[...]<jupyter_text>As the output above shows,  overall similarity scored are relatively low, they range from none to 0.2. The pair that has the highest similarity score equal to 0.2 is  Synset('woodsman.n.01') and  Synset('man.n.03').    Collocations (Optional Bonus)<jupyter_code>#Find bigrams collocations in text
bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(cleantokens_nostops)
scored = finder.score_ngrams(bigram_measures.raw_freq)
collocation2 =sorted(bigram for bigram, score in scored)  
#display first 15 collocations found
collocation2[:15]
# find trigrams collocations
trigram_measures = nltk.collocations.TrigramAssocMeasures()
finder3 = TrigramCollocationFinder.from_words(cleantokens_nostops)
scored3 = finder.score_ngrams(trigram_measures.raw_freq)
set(trigram for trigram, score in scored3) == set(nltk.trigrams(cleantokens_nostops))
#Display top 15 results
sorted(finder3.nbest(trigram_measures.raw_freq, 15))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08671160772001107, 'fraction_numerical': 0.025856496444731737, 'mean_word_length': 5.206303724928366, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '280365', 'license': 'MIT', 'n_tokens_mistral': 3819, 'n_tokens_neox': 3461, 'n_words': 1190, 'path': 'Text Analytics/MSDS_3_Part-of-Speech_Ngrams_Similarity_Collocatio.ipynb', 'repo_name': 'nweakly/MSDS_class_projects_python', 'size': '10829'}