<jupyter_start><jupyter_text>Exercises: Introduction to NLP in Python===![](http://static1.squarespace.com/static/538cea80e4b00f1fad490c1b/54668a77e4b00fb778d22a34/54668d8ae4b00fb778d2859c/1416007413413/NLTK.png)<jupyter_code>reset -fs
import nltk
nltk?<jupyter_output><empty_output><jupyter_text>---What is NLTK?---Natural Language Toolkit (nltk) is a Python package for NLP__Pros__:- Common- Functionality__Cons__:- Too academic- Too slow- Awkward API ---What are Corpus / Corpora?---> a collection of written texts, especially the entire works of a particular author or a body of writing on a particular subject.The rest of the world call them "documents". ---0) Download `nltk` corpora---![](images/download_all_the_corpora.png)<jupyter_code># Genesis is the first book in the Bible
nltk.download('genesis')<jupyter_output>[nltk_data] Downloading package genesis to /Users/brian/nltk_data...
[nltk_data]   Package genesis is already up-to-date!<jupyter_text>----<jupyter_code># Double check that you have the corpora
from nltk.corpus import genesis

assert genesis.words()[:3] == ['In', 'the', 'beginning']<jupyter_output><empty_output><jupyter_text>---1) Zipf's Law:---Zipf's Law (not a real "law") is a NLP phenomena:>  the frequency of any word is inversely proportional to its rank in the frequency table. or in Plain English:> a rank r word occurs 1/r times as often as the most frequent word, so the rank 2 word occurs half as often as the rank 1 word, the rank 3 word one-third as often, the rank 4 word one-fourth as often,...> the most frequent word will occur about twice as often as the second most frequent word For example:|Word  |  Rank | Frequency |  Count ||:-------:|:------:| :---:| :---: || “the”   | 1st | 1/10 | 10K || "of" | 2nd | 1/20 |  5K || "and" |  3rd | 1/30 | 3.3K |  __Word by frequency count:__![](images/graph-zipf.png) ------NLP (or Data Science) WorkFlow------1. Ask the right question2. Acquire the relevant data3. Process: clean and organize4. Model: Apply Statistics, Machine Learning, and Deep Learning5. Deliver: Add value, report, presentation, or code/API -------TODO: Take the words from "Genesis" and put the words counts into a dictionaryClick here for a hint…It is one line of code if you use an import from Python's Standard Library TODO: Sort word counts from high to low.Click here for a hint…Use the keyword arguments for `sorted`<jupyter_code>assert word_counts_sorted[0]  == (',', 22643)
assert word_counts_sorted[1]  == ('.', 11069)
assert word_counts_sorted[-1] == ('kista', 1)<jupyter_output><empty_output><jupyter_text>Take a small sample to visualize<jupyter_code>sample = word_counts_sorted[:15]
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1215088864708016, 'fraction_numerical': 0.03844758795792528, 'mean_word_length': 4.721991701244813, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '411537', 'license': 'Apache-2.0', 'n_tokens_mistral': 988, 'n_tokens_neox': 897, 'n_words': 317, 'path': '01_weclome/3_lab_intro_to_nlp.ipynb', 'repo_name': 'chrispaulca/nlp-course', 'size': '2757'}