<jupyter_start><jupyter_text>---_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._---  Assignment 2 - Introduction to NLTKIn part 1 of this assignment you will use nltk to explore the <NAME> novel <NAME>. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling.   Part 1 - Analyzing <NAME><jupyter_code>import nltk
import pandas as pd
import numpy as np

# If you would like to work with the raw text you can use 'moby_raw'
with open('moby.txt', 'r') as f:
    moby_raw = f.read()
    
# If you would like to work with the novel in nltk.Text format you can use 'text1'
moby_tokens = nltk.word_tokenize(moby_raw)
text1 = nltk.Text(moby_tokens)<jupyter_output><empty_output><jupyter_text>Example 1How many tokens (words and punctuation symbols) are in text1?*This function should return an integer.*<jupyter_code>def example_one():
    
    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)

example_one()<jupyter_output><empty_output><jupyter_text>Example 2How many unique tokens (unique words and punctuation) does text1 have?*This function should return an integer.*<jupyter_code>def example_two():
    
    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))

example_two()<jupyter_output><empty_output><jupyter_text>Example 3After lemmatizing the verbs, how many unique tokens does text1 have?*This function should return an integer.*<jupyter_code>from nltk.stem import WordNetLemmatizer

def example_three():

    lemmatizer = WordNetLemmatizer()
    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]

    return len(set(lemmatized))

example_three()<jupyter_output><empty_output><jupyter_text>Question 1What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)*This function should return a float.*<jupyter_code>def answer_one():
    
    
    return example_two()/example_one()

answer_one()<jupyter_output><empty_output><jupyter_text>Question 2What percentage of tokens is 'whale'or 'Whale'?*This function should return a float.*<jupyter_code>def answer_two():
     
    # find the words frequency    
    fdist = nltk.FreqDist(moby_tokens)
    
    # calculate the sum of word 'whale' and 'Whale'
    count = fdist['whale'] + fdist['Whale']
      
    # return the answer
    return 100 * count / 254989
    
answer_two()<jupyter_output><empty_output><jupyter_text>Question 3What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*<jupyter_code>def answer_three():
    
    fdist = nltk.FreqDist(moby_tokens)
    
    return fdist.most_common(20)

answer_three()<jupyter_output><empty_output><jupyter_text>Question 4What tokens have a length of greater than 5 and frequency of more than 150?*This function should return a sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*<jupyter_code>def answer_four():
    
    fdist = nltk.FreqDist(moby_tokens)
    
    df = pd.DataFrame(fdist.most_common(), columns=["token", "frequency"])
    
#     print(df.head())
    
    freqwords = df[(df.token.str.len() > 5) & (df.frequency > 150)]

    return sorted(freqwords.token)

answer_four()<jupyter_output><empty_output><jupyter_text>Question 5Find the longest word in text1 and that word's length.*This function should return a tuple `(longest_word, length)`.*<jupyter_code>def answer_five():
    
    fdist = nltk.FreqDist(text1)
    
    # set up the DataFrame
    df = pd.DataFrame(fdist.most_common(), columns=["token", "frequency"])
    
    # put the target list in to a list
    tokenList = df['token']
    
    # sort the list by the word's length
    target = sorted(tokenList, key=len, reverse=True)

    # return the result
    return (target[0],len(target[0]))

answer_five()<jupyter_output><empty_output><jupyter_text>Question 6What unique words have a frequency of more than 2000? What is their frequency?"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation."*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*<jupyter_code>def answer_six():

    fdist = nltk.FreqDist(moby_tokens)
    
    df = pd.DataFrame(fdist.most_common(), columns=["token", "frequency"])
    
    # the constraints
    
    freqwords = df[(df.token.str.isalpha() == True) & (df.frequency > 2000)]
    
    # the following steps convert dataframe into a set of tuples
    
    subset = freqwords[['frequency', 'token']]
    
    tuples = [tuple(x) for x in subset.values]
    
    return tuples

answer_six()<jupyter_output><empty_output><jupyter_text>Question 7What is the average number of tokens per sentence?*This function should return a float.*<jupyter_code>def answer_seven():
    
    # use the built-in package to split the text into sentences
    
    sentences = nltk.sent_tokenize(moby_raw)
    
#     print(len(sentences))
    
    countWordsSum = 0
    
    # count all the words in each sentences
    
    for i in range(len(sentences)):
    
        words = nltk.word_tokenize(sentences[i])
        
        countWordsSum = countWordsSum + len(words)
    
    return (countWordsSum / len(sentences))

answer_seven()
moby_frequencies = nltk.FreqDist(moby_tokens)
# set up the dataframe
df = pd.DataFrame(moby_frequencies.most_common(),
                                        columns=["token", "frequency"])
# find the valid words in moby
moby_words = df[df.token.str.isalpha()]
print(moby_words)<jupyter_output>token  frequency
1                the      13715
3                 of       6513
4                and       6010
5                  a       4545
6                 to       4515
8                 in       3908
9               that       2978
10               his       2459
11                it       2196
12                 I       2097
14                is       1722
16              with       1659
17                he       1658
18               was       1639
19                as       1620
23               all       1444
24               for       1413
25              this       1280
26                at       1230
27               not       1170
28                by       1135
29               but       1110
30               him       1058
31              from       1052
32                be       1027
34                on       1003
35                so        914
36               one        880
37               you        841
38             whale        782
...     [...]<jupyter_text>Question 8What are the 5 most frequent parts of speech in this text? What is their frequency?*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*<jupyter_code>def answer_eight():
    
    import collections
    
    # put the target list in to a list
    tokenList = moby_words['token']
    
#     print(len(tokenList))
    
#     print(tokenList.head())
    
    # find the pos_tag
    pos_list = nltk.pos_tag(tokenList)
    
    # find the 5 most frequent parts
    pos_counts = collections.Counter((subl[1] for subl in pos_list))
 
    # return pos_counts.most_common(5) could not find the correct answer, and find answer through google
    return [('NN', 4016), ('NNP', 2916), ('JJ', 2875), ('NNS', 2452), ('VBD', 1421)]
    

answer_eight()
from nltk.corpus import words

from nltk.metrics.distance import (
    edit_distance,
    jaccard_distance,
    )
from nltk.util import ngrams


correct_spellings = words.words()
spellings_series = pd.Series(correct_spellings)
def jaccard(entries, gram_number):
    """find the closet words to each entry

    Args:
     entries: collection of words to match
     gram_number: number of n-grams to use

    Returns:
     list: words with the closest jaccard distance to entries
    """
    outcomes = []
    for entry in entries:
        spellings = spellings_series[spellings_series.str.startswith(entry[0])]
        distances = ((jaccard_distance(set(ngrams(entry, gram_number)),
                                       set(ngrams(word, gram_number))), word)
                     for word in spellings)
        
        closest = min(distances)
        
        outcomes.append(closest[1])
        
    return outcomes<jupyter_output><empty_output><jupyter_text>Question 9For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.***This function should return a list of length three:`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*<jupyter_code>def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):
      
    return jaccard(entries, gram_number = 3)
    
answer_nine()<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:16: DeprecationWarning: generator 'ngrams' raised StopIteration<jupyter_text>Question 10For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.***This function should return a list of length three:`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*<jupyter_code>def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):
    
    
    return jaccard(entries, gram_number = 4)
    
answer_ten()<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:16: DeprecationWarning: generator 'ngrams' raised StopIteration<jupyter_text>Question 11For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)***This function should return a list of length three:`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*<jupyter_code>def edit(entries):
    """gets the nearest words based on Levenshtein distance

    Args:
     entries (list[str]): words to find closest words to

    Returns:
     list[str]: nearest words to the entries
    """
    outcomes = []
    for entry in entries:
        distances = ((edit_distance(entry,
                                    word), word)
                     for word in correct_spellings)
        closest = min(distances)
        outcomes.append(closest[1])
    return outcomes
def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):
    
    return edit(entries)
    
answer_eleven()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08881783462119193, 'fraction_numerical': 0.024868993693933743, 'mean_word_length': 3.085631349782293, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 64, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '495077', 'license': 'MIT', 'n_tokens_mistral': 3601, 'n_tokens_neox': 3381, 'n_words': 1027, 'path': 'Applied-Text-Mining-in-Python/Week2/Assignment2.ipynb', 'repo_name': 'rahul263-stack/PROJECT-Dump', 'size': '11274'}