<jupyter_start><jupyter_text>Build Yelp Review CorpusThis notebook outlines the steps to load, preprocess, and clean yelp review text.It was heavily inspired by: https://towardsdatascience.com/a-practitioners-guide-to-natural-language-processing-part-i-processing-understanding-text-9f4abfd13e72<jupyter_code>import psycopg2
import nltk
import unicodedata
import pandas as pd
import pprint
import pickle
import re
import os
import copy

pp = pprint.PrettyPrinter(indent=4).pprint

data_dir = '/home/tlappas/data_science/Yelp-Ratings/data/processed'<jupyter_output><empty_output><jupyter_text>Move to ./Yelp-Ratings/data/corpus folder ...This needs to be fixed. Breaks if run more than once.path = os.getcwd()print('Notebook path: {}'.format(path))os.chdir('..')data_path = os.path.join(os.getcwd(), 'corpus')print('Corpus path: {}'.format(data_path))if os.path.exists(data_path) == False:    os.mkdir(data_path)os.chdir(data_path)  Get DataText is stored in yelp db. Query and store in a pandas DataFrame.<jupyter_code>conn = psycopg2.connect('dbname=yelp user=tlappas host=/var/run/postgresql')
cur = conn.cursor()
cur.execute("""
    select review.review_text 
    from review, business, user_info
    where review.user_id = user_info.user_id
    and business.business_id = review.business_id
    and business.categories LIKE '%Restaurants%'
    and length(user_info.elite) != 0 
    limit 1
""")

cols = ['review_text']

data = pd.DataFrame(cur.fetchall(), columns=cols)<jupyter_output><empty_output><jupyter_text>Example Instance print('DataFrame shape: {}\n'.format(data.shape))print('First instance: \n')print(data.loc[0])  Example Review print(data.loc[1, 'review_text'])  Basic Text Pre-Preprocessing1. Remove any non-ASCII characters2. Replace any characters that aren't alphanumeric/whitespace/"'"3. Convert all letters to lowercase4. Replace whitespace with ' '.<jupyter_code>for i, text in enumerate(data.loc[:,'review_text']):

    text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf8', 'ignore')
    text = re.sub(r"[^A-Za-z0-9\s']", '', text)
    text = text.lower()
    text = re.sub(r'[\n|\r|\n\r|\r\n]', ' ', text)
    data.loc[i,'review_text'] = text
print(data.loc[0,'review_text'])<jupyter_output>get the a wreck  it has everything roast beef turkey and ham oh my  i like to add oil and italian seasoning with hot giardiniera if you really want to thrill your tastebuds   love the selection of zapp's chips here goes great with your sammie op<jupyter_text>Save Corpus - Cleaned Text<jupyter_code>processing = ['ascii-only', 'whitespace alphanumeric only', 'lowercase', 'remove whitespace chars']
with open(os.path.join(data_dir, 'reviews-clean.pkl'), 'wb') as f:
    pickle.dump([data, processing], f)<jupyter_output><empty_output><jupyter_text>Remove Stopwords<jupyter_code>stops = nltk.corpus.stopwords.words('english')

for i, text in data.loc[:,'review_text'].iterrows():
    text = [word for word in text.split() if word not in stops]
    text = ' '.join(text)
    data.at[i,'review_text'] = text
print(data.at[0,'review_text'])<jupyter_output>get wreck everything roast beef turkey ham oh like add oil italian seasoning hot giardiniera really want thrill tastebuds love selection zapp's chips goes great sammie op<jupyter_text>Save Corpus - Cleaned + No Stopwords<jupyter_code>processing.append('no stopwords')
with open(os.path.join(data_dir, 'reviews-clean-nostop.pkl'), 'wb') as f:
    pickle.dump([data, processing], f)<jupyter_output><empty_output><jupyter_text>Remove neutral stopwordsSince we're attempting to differentiate negative reviews from positive reviews negations may be important. For example 'not', 'no', 'hasn't', etc.<jupyter_code>neg_stops = [
    'no', 'not', 'nor', 'don\'', 'don\'t', 'ain', 
    'ain\'t', 'aren\'t', 'aren', 'couldn', 'couldn\'t', 
    'didn', 'didn\'t', 'doesn', 'doesn\'t', 'hadn', 
    'hadn\'t', 'hasn', 'hasn\'t', 'haven', 'haven\'t',
    'isn', 'isn\'t', 'mightn', 'mightn\'t', 'mustn', 
    'mustn\'t', 'needn', 'needn\'t', 'shan', 'shan\'t',
    'shouldn', 'shouldn\'t', 'wasn', 'wasn\'t', 'weren',
    'weren\'t', 'won', 'won\'t', 'wouldn', 'wouldn\'t'
]

no_neg_stops = [word for word in stops if word in stops and word not in neg_stops]

data_neg_stops = []
processing = []

with open(os.path.join(data_dir, 'reviews-clean.pkl'), 'rb') as f:
    [data_neg_stops, processing] = pickle.load(f)
    
for i, text in data_neg_stops.iterrows():
    text = [word for word in text.split() if word not in stops]
    text = ' '.join(text)
    data.at[i,'review_text'] = text
print(data.at[0,'review_text'])<jupyter_output>review_text<jupyter_text>Save Corpus - Cleaned + No Neutral Stopwords<jupyter_code>processing.append('Remove all pos/neutral stopwords')
with open(os.path.join(data_dir, 'reviews-clean-neg-stops.pkl'), 'wb') as f:
    pickle.dump([data, processing], f)<jupyter_output><empty_output><jupyter_text>Lemmatize TextThe wordnet lemmatizer only lemmatizes a single pos at a time. Default pos param is 'N' (noun). [Set of pos tags](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html).[Here's an example](https://www.machinelearningplus.com/nlp/lemmatization-examples-python/wordnetlemmatizerwithappropriatepostag) that ties in the nltk pos tagger to identify the pos, then use the lemmatizer to lem just that type. Which will save a huge amount of time, since it won't need to go through every one.[Wordnet documentation](http://www.nltk.org/howto/wordnet.html)<jupyter_code>wnl = nltk.WordNetLemmatizer()

for i, text in data.loc[:,'review_text'].iterrows():
    text = text.split()
    text = [wnl.lemmatize(word, pos='n') for word in text]
    text = [wnl.lemmatize(word, pos='v') for word in text]
    text = [wnl.lemmatize(word, pos='a') for word in text]
    text = [wnl.lemmatize(word, pos='r') for word in text]
    text = ' '.join(text)
    data.at[i,'review_text'] = text
print(data.at[0,'review_text'])<jupyter_output>review_text<jupyter_text>Save Corpus - Cleaned + No Stopwords + Lemmatized<jupyter_code>processing.append('lemmatize: NVJR')
with open(os.path.join(data_dir, 'reviews-clean-nostop-lemmed.pkl'), 'wb') as f:
    pickle.dump([data, processing], f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14935897435897436, 'fraction_numerical': 0.004967948717948718, 'mean_word_length': 5.272361809045226, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '365448', 'license': 'MIT', 'n_tokens_mistral': 2270, 'n_tokens_neox': 2191, 'n_words': 537, 'path': 'notebooks/tsl-05-text-processing-pipeline.ipynb', 'repo_name': 'georgetown-analytics/Labor-Violations', 'size': '6240'}