<jupyter_start><jupyter_text>Metadata```Course:   DS 5001 Module:   01 LabTopic:    First ForayAuthor:   <NAME>Purpose:  We take a raw text file from Project Gutenberg and convert it into a dataframe of tokens. We then explore some properties of the data. The main idea is to get acquainted with the primary process of convert text into analytical form.```  Set Up<jupyter_code>import pandas as pd
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>Import File<jupyter_code>lines = open('pg105.txt', 'r').readlines()
lines[:5]
lines[-5:]<jupyter_output><empty_output><jupyter_text>Convert to Dataframe<jupyter_code>text = pd.DataFrame(lines)
text.sample(10)
text.columns = ['line_str']
text.head()
text.index.name = 'line_num'
text.head()<jupyter_output><empty_output><jupyter_text>Extract Simple Features<jupyter_code>text['len'] = text.line_str.str.len()
text.len.describe()
text.len.hist();<jupyter_output><empty_output><jupyter_text>**Why two humps?** What might this bimodal distribution indicate? Let's look at the first hump for characters.<jupyter_code>text[text['len'] < 5].sample(10)<jupyter_output><empty_output><jupyter_text>Import AgainNow that we know what line breaks mean, we can use this information to import the file with a more accurate structure. Note also that we could have inferred this from visual inspection, too. But the principle that statistical features can provide evidence for structure remains -- we will use this throughout the course.  Interpret line breaks `\n`<jupyter_code>chunk_pat = '\n\n'
chunks = open('pg105.txt', 'r').read().split(chunk_pat)
text = pd.DataFrame(chunks, columns=['chunk_str'])
text.index.name = 'chunk_id'
text.head()
text.shape<jupyter_output><empty_output><jupyter_text>Remove remaining breaks<jupyter_code>text.chunk_str = text.chunk_str.str.replace('\n+', ' ', regex=True).str.strip()
text.head()<jupyter_output><empty_output><jupyter_text>Convert Lines to Tokens`K: A dataframe of tokens`<jupyter_code>K = text.chunk_str.str.split(' ', expand=True).stack().to_frame('token_str')
K.index.names = ['chunk_num','token_num']
K.shape
K.head()
# Broken down into steps
# text.chunk_str.str.split(' ')
# text.chunk_str.str.split(' ', expand=True)
# text.chunk_str.str.split(' ', expand=True).stack()
# text.chunk_str.str.split(' ', expand=True).stack().to_frame('token_str')
K.iloc[100:120, :]<jupyter_output><empty_output><jupyter_text>Do Some Cleaning<jupyter_code>K['term_str'] = K.token_str.str.replace(r'\W+', '', regex=True).str.lower()
K.sample(10)<jupyter_output><empty_output><jupyter_text>Extract a Vocabulary`v: A table of terms` (As opposed to tokens, which are term _instances_.)<jupyter_code>V = K.term_str.value_counts().to_frame('n')
V.index.name = 'term_str'
V.head(10)<jupyter_output><empty_output><jupyter_text>Visualize Frequest Words<jupyter_code>V.plot(figsize=(10,5), fontsize=14, rot=45, legend=False);
V.n.head(20).sort_values().plot.barh(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>The TheWhy is "the" the most frequent word?Consider that "the" is "[The Most Powerful Word in the English Language](http://www.bbc.com/culture/story/20200109-is-this-the-most-powerful-word-in-the-english-language)."> ... ‘the’ lies at the heart of English grammar, having a function rather than a meaning. Words are split into two categories: expressions with a semantic meaning and functional words like ‘the’, ‘to’, ‘for’, with a job to do. ‘The’ can function in multiple ways. This is typical, explains <NAME>, assistant professor in linguistics at New York University: “a super high-usage word will often develop a real flexibility”, with different subtle uses that make it hard to define. Helping us understand what is being referred to, ‘the’ makes sense of nouns as a subject or an object. So even someone with a rudimentary grasp of English can tell the difference between ‘I ate an apple’ and ‘I ate the apple’.Note: function vs. meaning ...> **Function words are very specific to each language.**So, someone who is a native Hindi or Russian speaker is going to have to think very differently when constructing a sentence in English. Murphy says that she has noticed, for instance, that sometimes her Chinese students hedge their bets and include ‘the’ where it is not required. Conversely, Smith describes Russian friends who are so unsure when to use ‘the’ that they sometimes leave a little pause: ‘I went into... bank. I picked up... pen.’ English speakers learning a language with no equivalent of ‘the’ also struggle and might overcompensate by using words like ‘this’ and ‘that’ instead.  Word Dispersion Plots<jupyter_code>def word_plot(term_str):
    global K
    term_str = term_str.lower()
    (K.term_str == term_str)\
        .reset_index(drop=True).astype('int')\
        .to_frame(term_str)\
        .plot(figsize=(20, .5), legend=False, title=term_str)
word_plot('the')
word_plot('superior')
word_plot('delicacy')
word_plot('anne')
word_plot('walter')
word_plot('wentworth')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12192216044479746, 'fraction_numerical': 0.010524225575853852, 'mean_word_length': 5.966804979253112, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '514460', 'license': 'MIT', 'n_tokens_mistral': 1649, 'n_tokens_neox': 1589, 'n_words': 494, 'path': 'M01_Intro/M01_03_first-foray.ipynb', 'repo_name': 'avnimalik/DS5001-2022-01', 'size': '5047'}