<jupyter_start><jupyter_text>Text Preprocessing In this notebook we are going to perform a few basic text preprocessing steps on the extracted data.<jupyter_code># Import some import-worthy things 
import numpy as np
import pandas as pd 
import re
# For some Plottery on the fly
import matplotlib.pyplot as plt
from matplotlib.ticker import StrMethodFormatter

# Some stuff the help me with the debugging 
_GLOBAL_DEBUG_ = False
# Note to self - I really need to start using Python Logger!!!<jupyter_output><empty_output><jupyter_text>Load the data in a Pandas DataFrame Initially we will work with just 1,000 entries and scale up later.<jupyter_code># Let's load a small subset of the June data. 
# We will work on it, test it, verify it
# Once confident, we will apply all the steps to the whole dataset.

"""
header : int, list of int, default ‘infer’
    Row number(s) to use as the column names, and the start of the data. 
index_col : int, str, sequence of int / str, or False, default None
    Column(s) to use as the row labels of the DataFrame, either given as string name or column index.
usecols : list-like or callable, optional
    Return a subset of the columns. If list-like, all elements must either be positional 
    (i.e. integer indices into the document columns) or strings that correspond to column names 
    provided either by the user in names or inferred from the document header row(s).    
prefix : str, optional
    Prefix to add to column numbers when no header, e.g. ‘X’ for X0, X1, …
nrows : int, optional
    Number of rows of file to read. Useful for reading pieces of large files.    
"""
raw_text_df = pd.read_csv('./data/60_days_of_udacity_june.csv', 
                          header=0, usecols=['message'], nrows=1000)

# This little bit of pandas hackery will ensure that we get to see all the columns
# and rows, whenever a dataframe is printed 
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

if _GLOBAL_DEBUG_:
    print(raw_text_df.shape)

raw_text_df.head(10)<jupyter_output><empty_output><jupyter_text>Preprocess TextWe are going to preprocess text now. Following preprocessing steps are going to get applied, 1. Lowercase all text messages 2. Replace URLs with a space 3. Replace usernames with a space 4. Replace all special characters with a space<jupyter_code># 1
# Lowercase all message strings 
raw_text_df['message'] = raw_text_df['message'].str.lower()
raw_text_df.head()
# 2 
# Replace URLs with a space " "
# This nasty regex string covers ALL the edge cases for a URL 
re_string = r'(https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]+\.[^\s]{2,}|www\.[a-zA-Z0-9]+\.[^\s]{2,})'

# Before we go around removing links from the data, 
# we need to have a way to VERIFY if our regex removal 
# system as worked or not. 
# VERIFICATION is a very important aspect of data preproccessing. 

# A simple method would be find the messages containing links using the same regex string 
# After removal, we should not find anything using the same string. 
print(f'Messages containing links = {sum(raw_text_df["message"].str.contains(re_string, regex=True).astype(int))}')

# Let's remove the links 
raw_text_df['message'] = raw_text_df['message'].replace(to_replace=re_string, value=' ', regex=True)

# Let's try again and see if we can find any messages containing links to verify 
# A value of 0 should indicate that we have been successful! 
print(f'Messages containing links after removal = {sum(raw_text_df["message"].str.contains(re_string, regex=True).astype(int))}')

# 3
# Usernames don't really add any semantic meaning for the machine 
# Therefore, replace all usernames with a space 
raw_text_df['message'] = raw_text_df['message'].replace(to_replace=r'@(\S)*', value=' ', regex=True)

raw_text_df['message'].head(10)
# 4
# Remove all the special characters except for letters and numbers with a space
raw_text_df['message'] = raw_text_df['message'].replace(to_replace=r'[^0-9a-z]+', value=' ', regex=True)

raw_text_df['message'].head()<jupyter_output><empty_output><jupyter_text>Text Preprocessing Routine Looks like we did a good job up there. Time to put everything together into a _one-routine-to-process-them-all_.<jupyter_code>def text_preprocess(df, column='message'):
    """
    ==============================================================================
    EDIT: After training the GPT-2 on the text data a number of times I have 
        figured that removing special characters, usernames, and punctuation 
        is taking the flavor out of the generated text. GPT-2 is smart enough 
        to know where to use what, so I don't need to filter these out. 
        I only have to remove the URLs and newline '\n' character. \n is causing
        havoc when writing the DF to a CSV file. 
        
        Also, we need to remove the stupid 'mojibakes'. 
    ==============================================================================
    
    Text preprocessing routine. Performs the following 
    steps on the data,
    1. Lowercase all text messages
    2. Replace URLs with a space
    3. Replace usernames with a space
    4. Replace all special characters with a space    
    
    Parameters
    ----------
    df : Pandas DataFrame   
        Unprocessed dataframe
    
    column : string 
        Column name that is going to be processed
   
    Returns
    -------
    df : Post processed dataframe 
    """
    # Debug Flag
    _LOCAL_DEBUG_ = True
    
    # 1
    # Lowercase all message strings 
    df[column] = df[column].astype(str).str.lower()
    # 2 
    # Replace URLs with a space " "
    re_string = r'(https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]+\.[^\s]{2,}|www\.[a-zA-Z0-9]+\.[^\s]{2,})'
    df[column] = df[column].replace(to_replace=re_string, value=' ', regex=True)
    # 3
    # Replace all usernames with a space 
    # df[column] = df[column].replace(to_replace=r'@(\S)*', value=' ', regex=True)
    # 4
    # Remove all the special characters except for letters and numbers, with a space
    # df[column] = df[column].replace(to_replace=r'[^0-9a-z:@]+', value=' ', regex=True)
    
    # Only need to remove the newline \n character.  
    df[column] = df[column].replace(to_replace=r'\n', value=' ', regex=True)
    # Remove mojibakes with corret replacement (I could have used ftfy here, but I didn't)
    df[column] = df[column].replace(to_replace='â€¦', value='…', regex=True)
    df[column] = df[column].replace(to_replace='â€“', value='–', regex=True)
    df[column] = df[column].replace(to_replace='â€™', value='’', regex=True)
    df[column] = df[column].replace(to_replace='â€œ', value='“', regex=True)
   
    if _GLOBAL_DEBUG_ and _LOCAL_DEBUG_:
        print(df[column].head(25))
        print(df[column][1000:1020])
  
    return df<jupyter_output><empty_output><jupyter_text>Let's do a quick test run of our new and shiny text_preprocess() routine.<jupyter_code>filtered_df = text_preprocess(raw_text_df, 'message')

filtered_df.head()<jupyter_output><empty_output><jupyter_text>Preprocessing for GPT-2With our text_preprocess routine working fine. We are left with the following tasks, 1. Load the full text corpus from both the files2. Preprocess the complete text corpus 3. Combine the processed text corpus into a single file<jupyter_code># 1
# Load the data from both the files in two separate dataframes
raw_text_df_1 = pd.read_csv('./data/60_days_of_udacity_june.csv', header=0, usecols=['message'])
raw_text_df_2 = pd.read_csv('./data/60_days_of_udacity_july_13.csv', header=0, usecols=['message'])

print(raw_text_df_1.shape)
print(raw_text_df_2.shape)
print(f'Total Messages in the corpus = {raw_text_df_1.shape[0]+raw_text_df_2.shape[0]}')
# 2
# Preprocess both the dataframes separately 
filtered_df_1 = text_preprocess(raw_text_df_1, 'message')
filtered_df_2 = text_preprocess(raw_text_df_2, 'message')

print('Done!')
# 3
# Combine both the dataframes 
text_corpus_df = pd.concat([filtered_df_1, filtered_df_2], ignore_index=True)

# Trust, but verify! 
print(f'Total Messages in filtered_df_1 and filtered_df_2 = {filtered_df_1.shape[0]+filtered_df_2.shape[0]}')
print(f'Total Messages in the text_corpus_df = {text_corpus_df.shape[0]}')
print('Above numbers should be same. ')
# Also, we save our processed data 
text_corpus_df.to_csv('./data/complete_text_corpus.csv')

print('File Write Complete!')<jupyter_output>File Write Complete!<jupyter_text>Filtering out the Messages Our work is not done yet. We need to filter out sentences that are too short.<jupyter_code># Calculate the length of each message 
text_corpus_df['m_length'] = text_corpus_df['message'].astype(str).apply(lambda row: len(row))

if _GLOBAL_DEBUG_:
    display(text_corpus_df[['message', 'm_length']].head())
# 'Describe' our m_length column 
# Divide the message lengths in 'Quintiles'
text_corpus_df['m_length'].describe(percentiles=[0.2, 0.4, 0.6, 0.8])<jupyter_output><empty_output><jupyter_text>Above we can see the problem with our data. 1. From a simple visual inspection and a little bit of common sense I can tell that 99% of the messages with length smaller than 100 are not related to daily updates. 2. Messages with length equal to 24 are just bot messages announcing someone has joined the channel. In short, We DO NOT HAVE ENOUGH DATA! Nevertheless we will finish what we started. We will filter out all messages less than 75 characters long. If you look at the quintiles above, that should be roughly about half of all the messages that we have. _Sad but True_.<jupyter_code>text_corpus_df = text_corpus_df[ (text_corpus_df.m_length > 75) ]

text_corpus_df['m_length'].describe(percentiles=[0.2, 0.4, 0.6, 0.8])
# Let's draw a histogram to visualize the distribution of our data  

ax = text_corpus_df.hist(column='m_length', bins=10, grid=False, figsize=(12,8), color='#86bf91', zorder=2, rwidth=0.9)

ax = ax[0]
for x in ax:
    # Despine
    x.spines['right'].set_visible(False)
    x.spines['top'].set_visible(False)
    x.spines['left'].set_visible(False)
    # Switch off ticks
    x.tick_params(axis="both", which="both", bottom="off", top="off", labelbottom="on", left="off", right="off", labelleft="on")
    # Draw horizontal axis lines
    vals = x.get_yticks()
    for tick in vals:
        x.axhline(y=tick, linestyle='dashed', alpha=0.4, color='#eeeeee', zorder=1)
    # Remove title
    x.set_title("")
    # Set x-axis label
    x.set_xlabel("Message Length", labelpad=20, weight='bold', size=12)
    # Set y-axis label
    x.set_ylabel("No of Messages", labelpad=20, weight='bold', size=12)
    # Format y-axis label
    x.yaxis.set_major_formatter(StrMethodFormatter('{x:,g}'))<jupyter_output><empty_output><jupyter_text>Some thoughts on the preprocessing results above, 1. Out of the 13,000 odd messages, ~40% were either bot announcements or were too short to be useful. 2. Almost 99% of the messages have a character length of less than 2000. 3. We keep the outliers in our dataset since our data is pretty darn small anyway.<jupyter_code>text_corpus_df.head()
# To train the GPT-2 model, we need the CSV file containing only a single column. 
# In our case, we need to keep 'message' and ignore all the other columns, 
text_corpus_df['message'].to_csv('./data/final_text_corpus.csv', index=False)

print('File Write Complete!')<jupyter_output>C:\Anaconda3\envs\my_NLP\lib\site-packages\ipykernel_launcher.py:3: FutureWarning: The signature of `Series.to_csv` was aligned to that of `DataFrame.to_csv`, and argument 'header' will change its default value from False to True: please pass an explicit value to suppress this warning.
  This is separate from the ipykernel package so we can avoid doing imports until
{'fraction_non_alphanumeric': 0.12932406399195912, 'fraction_numerical': 0.0165843035430103, 'mean_word_length': 4.342281879194631, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '370116', 'license': 'Apache-2.0', 'n_tokens_mistral': 4002, 'n_tokens_neox': 3816, 'n_words': 1273, 'path': 'Pranjal Chaubey/60-days-of-udacity-sixty-ai/02 Data Preprocessing/Text Preprocessing.ipynb', 'repo_name': 'bharatnishant/UdacityOpenSource', 'size': '11939'}