<jupyter_start><jupyter_text>NLP_Session_2_AmazonReviews_Example Amazon Review Polarity Dataset DOWNLOAD DATA FROM HEREhttps://www.kaggle.com/kritanjalijain/amazon-reviews/download Extract the train and test CSV files to your google drive location and configure that in the code OVERVIEWContains 34,686,770 Amazon reviews from 6,643,669 users on 2,441,053 products, from the Stanford Network Analysis Project (SNAP). This subset contains 1,800,000 training samples and 200,000 testing samples in each polarity sentiment. ORIGINThe Amazon reviews dataset consists of reviews from amazon. The data span a period of 18 years, including ~35 million reviews up to March 2013. Reviews include product and user information, ratings, and a plaintext review. For more information, please refer to the following paper: <NAME> and <NAME>. Hidden factors and hidden topics: understanding rating dimensions with review text. RecSys, 2013. DESCRIPTIONThe Amazon reviews polarity dataset is constructed by taking review score 1 and 2 as negative, and 4 and 5 as positive. Samples of score 3 is ignored. In the dataset, class 1 is the negative and class 2 is the positive. Each class has 1,800,000 training samples and 200,000 testing samples.If you need help extracting the train.csv and test.csv files check out the starter code.The files train.csv and test.csv contain all the training samples as comma-separated values.The CSVs contain polarity, title, text. These 3 columns in them, correspond to class index (1 or 2), review title and review text.polarity - 1 for negative and 2 for positivetitle - review headingtext - review bodyThe review title and text are escaped using double quotes ("), and any internal double quote is escaped by 2 double quotes (""). New lines are escaped by a backslash followed with an "n" character, that is "\n". CITATIONThe Amazon reviews polarity dataset is constructed by <NAME> (<EMAIL>). It is used as a text classification benchmark in the following paper: <NAME>, <NAME>, <NAME>. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015). About TextBLOB libraryTextBlob is a Python (2 and 3) library for processing textual data. It provides a consistent API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, and more.Reference : https://textblob.readthedocs.io/en/dev/  Reference1. https://www.kaggle.com/kritanjalijain/amazon-reviews2. https://www.kaggle.com/sindhuguttal/amazon-review-using-nlp/data2. https://textblob.readthedocs.io/en/dev/  Importing the libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tarfile 
import seaborn as sns
import matplotlib.style as style
import matplotlib as mpl
import re
import string
import itertools
import collections
from wordcloud import WordCloud
import nltk 
from nltk.util import ngrams
from nltk.corpus import stopwords,RegexpTokenizer
from textblob import TextBlob
from nltk.stem import WordNetLemmatizer<jupyter_output><empty_output><jupyter_text>Setting up NLP Libraries and corpus<jupyter_code>nltk.download('stopwords')
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')<jupyter_output>[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!<jupyter_text>Mounting the data source<jupyter_code>from google.colab import drive
drive.mount('/content/drive',force_remount=True)<jupyter_output>Mounted at /content/drive<jupyter_text>Configuring the input, output and process folders<jupyter_code># Model Input and output folders
# Setup in Google drive
# '/content/drive/MyDrive/yourlocation/input/'
model_input_folder='/content/drive/MyDrive/yourlocation/input/'
model_output_folder='/content/drive/MyDrive/yourlocation/output/'
input_train_file=model_input_folder+'train.csv'
input_test_file=model_input_folder+'test.csv'
output_nlp_detail=model_output_folder+'nlp_details.csv'<jupyter_output><empty_output><jupyter_text>Loading the training dataset<jupyter_code># check out what the data looks like before you get started
# look at the training data set
train_df = pd.read_csv(input_train_file, header=None)
print(train_df.head())
train_df.shape<jupyter_output><empty_output><jupyter_text>Reducing the size of the dataframe for demonstration purposes<jupyter_code># Reducing the size of the dataframe
train_df=train_df.loc[1:10000]
for col in train_df.columns:
    print(col)
#checking a null values
train_df.isnull().sum()
#droping null vlaues
train_df.dropna()
train_df.isnull().count()<jupyter_output><empty_output><jupyter_text>Quick look at the dataset loaded<jupyter_code>train_df.info()
train_df.drop([0],axis=1,inplace=True)
for col in train_df.columns:
    print(col)
train_df.drop([1],axis=1,inplace=True)
train_df.shape
train_df.sample(5)
# Checking for Null Values
train_df[train_df[2].isnull()]<jupyter_output><empty_output><jupyter_text>Obtaining the review lengths<jupyter_code>train = train_df.copy()
train[2].apply(str)
train["review_length"] = train[2].apply(lambda w : len(re.findall(r'\w+', w)))<jupyter_output><empty_output><jupyter_text>Getting some statistics around the review length<jupyter_code>train['review_length'].describe()<jupyter_output><empty_output><jupyter_text>Doing some graphical plots<jupyter_code>sns.boxplot(data = train , x="review_length")
plt.xlabel('Number of Words')
plt.title('Review Length, Including Stop Words')
plt.show()
sns.distplot(train['review_length'], kde = False)
plt.xlabel('Distribution of Review Length')
plt.title('Review Length, Including Stop Words')
plt.show()<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>If we want to do better we must pre-process data like1. Converting to lower case2. Removing punctuation3. Removing Numbers4. Removing trailing spaces5. Removing extra whitespaces<jupyter_code>train_clean = train.copy()
stop_words = stopwords.words("english")<jupyter_output><empty_output><jupyter_text>Function to clean text<jupyter_code># Function for cleaning text
def clean(s):
    s = s.lower()                   #Converting to lower case
    s = re.sub(r'[^\w\s]', ' ', s)  #Removing punctuation
    s = re.sub(r'[\d+]', ' ', s)    #Removing Numbers
    s = s.strip()                   #Removing trailing spaces
    s = re.sub(' +', ' ', s)        #Removing extra whitespaces
    return s
train_clean["Reviews"] = train_clean[2].apply(lambda x: clean(x))
train_clean.sample(2)<jupyter_output><empty_output><jupyter_text>1. Removal of STOP WORDS<jupyter_code># Removal of Stop Words
train_clean["Reviews"] = train_clean["Reviews"].apply(lambda x: " ".join(x for x in x.split() if x not in stop_words))
import pandas as pd
reviews = pd.Series(train_clean["Reviews"].tolist()).astype(str)
plt.figure(figsize = (9, 9))
rev_wcloud_all = WordCloud(width = 900, height = 900, colormap = 'plasma', max_words = 150).generate(''.join(reviews))
plt.imshow(rev_wcloud_all)
plt.tight_layout(pad = 0.2)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>Detailed NLP Analysis<jupyter_code>tokenizer = RegexpTokenizer(r'\w+')
train_clean["review_token"] = train_clean["Reviews"].apply(lambda x: tokenizer.tokenize(x))
# Sentiment analysis
train_clean["sentiment_polarity"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).sentiment.polarity)
train_clean["sentiment_subjectivity"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).sentiment.subjectivity)
# Pos breakdown
train_clean["pos"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).tags)
train_clean["words"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).words)
train_clean["sentences"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).sentences)
#returns a list of tuples of n successive words
train_clean["ngrams_3"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).ngrams(n=3))
train_clean["ngrams_5"] = train_clean["Reviews"].apply(lambda x: TextBlob(x).ngrams(n=5))
#writing the detailed analysis
train_clean.to_csv()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10236401193481753, 'fraction_numerical': 0.014918521918751435, 'mean_word_length': 6.323529411764706, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '667728', 'license': 'MIT', 'n_tokens_mistral': 2881, 'n_tokens_neox': 2704, 'n_words': 803, 'path': 'AmazonReviews_Example/NLP_Session_2_AmazonReviews_Example.ipynb', 'repo_name': 'drshyamsundaram/nlp', 'size': '8753'}