<jupyter_start><jupyter_text>Preliminary Naive Bayes Model for Sentiment Polarity Import necessary dependencies<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from tqdm import tqdm
tqdm.pandas()

# For Tokenizing
import re
import string
import itertools
from spellchecker import SpellChecker
from contractions import contractions_dict

from nltk import download, word_tokenize
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.stem import WordNetLemmatizer

# For Text Classification
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from nltk.sentiment import SentimentAnalyzer
from nltk.corpus import subjectivity
from nltk.classify import NaiveBayesClassifier<jupyter_output><empty_output><jupyter_text>Download Necessary Corpus for tokenizer, and sentiment analyzers to work.<jupyter_code>download('punkt')
download('stopwords')
download('vader_lexicon')<jupyter_output>[nltk_data] Downloading package punkt to /home/lyqht/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /home/lyqht/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     /home/lyqht/nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!<jupyter_text>The Dataset Import the dataset and rename some columns<jupyter_code>df = pd.read_csv("../data/googleplaystore_user_reviews.csv")
df.rename(columns = {'Translated_Review':'Review'}, inplace=True)
df.sample(4)<jupyter_output><empty_output><jupyter_text>TerminologySource of definitions: Bing Liu. Sentiment Analysis and Opinion Mining, Morgan & Claypool Publishers, May 2012- Sentiment Polarity: Can be split into rational or emotional sentiment. Hence resulting in the common design of 5 sentiment ratings.    - emotional negative (-2)    - rational negative (-1)    - neutral (0): In practice, neutral often means no opinion or sentiment expressed.    - rational positive (+1)    - emotional positive (+2)- Sentiment Subjectivity: Subjective sentences expresses some personal feelings, views, or beliefs. They may not express any sentiment.  We see that the original dataset also has labels of the sentiment level, polarity and subjectivity by the original author. We will build our own model then compare the values to the values in the original dataset for comparison. Getting the reviews<jupyter_code>raw_reviews = df["Review"].copy()
raw_reviews.sample(5)<jupyter_output><empty_output><jupyter_text>Text Mining  Preprocessing We will drop NA types in the reviews data, since they do not contain any text. We will also remove duplicates. They probably got into the dataset because of imperfect data scraping.<jupyter_code>reviews = raw_reviews.dropna(axis=0).drop_duplicates()
print("Old data frame length:", len(raw_reviews), "\nNew data frame length:",  
       len(reviews), "\nNumber of rows with at least 1 NA value: ", 
       (len(raw_reviews)-len(reviews)))<jupyter_output>Old data frame length: 64295 
New data frame length: 27994 
Number of rows with at least 1 NA value:  36301<jupyter_text>These are the preprocessing steps that we will be doing for every review.1. Converting text to lowercase2. Remove html tags3. Expand Contractions    - Dictionary source from : https://github.com/pemagrg1/Text-Pre-Processing-in-Python/blob/master/individual_python_files/contractions.py4. Remove punctuation5. Using NLKT's `PunktSentenceTokenizer` to create tokens6. Word Normalization for Tokens    - Correcting mispelled words    - Stop word removal    - Lemmatization<jupyter_code>lemmatizer = WordNetLemmatizer()
stopword = set(stopwords.words('english'))
spell = SpellChecker()

# Regex Patterns
html_pattern = re.compile('<[^<]+?>')
lengthening_pattern = re.compile(r"(.)\1{2,}")
contractions_pattern = re.compile('({})'.format('|'.join(contractions_dict.keys())),
                                      flags=re.IGNORECASE | re.DOTALL)

def remove_html_tags(s):
    global html_pattern
    return html_pattern.sub('', s)
    
def remove_punctuation(s):
    return s.translate(str.maketrans('', '', string.punctuation))    

def expand_contractions(text):
    global contractions_pattern
    def expand_match(contraction):
        match = contraction.group(0)
        first_char = match[0]
        expanded_contraction = contractions_dict.get(match) \
            if contractions_dict.get(match) \
            else contractions_dict.get(match.lower())
        expanded_contraction = expanded_contraction
        return expanded_contraction

    expanded_text = contractions_pattern.sub(expand_match, text)
    expanded_text = re.sub("'", "", expanded_text)
    return expanded_text

def reduce_lengthening(s):
    global lengthening_pattern
    return lengthening_pattern.sub(r"\1\1", s)

def lemmatize(w):
    global lemmatizer
    return lemmatizer.lemmatize(w)

def normalize(s):
    global spell
    tokens = word_tokenize(s)
    tokens = [lemmatize(word) for word in tokens]
    return tokens


def preprocess(s):
    s = s.lower()
    s = remove_html_tags(s)
    s = expand_contractions(s)
    s = remove_punctuation(s)
    s = reduce_lengthening(s)
    words = normalize(s)
    
    return words<jupyter_output><empty_output><jupyter_text>Applying preprocessing to all the reviews<jupyter_code>tokens = reviews.apply(preprocess)
processed_df = pd.DataFrame(reviews)
processed_df["Tokens"] = tokens
processed_df.sample(5)<jupyter_output><empty_output><jupyter_text>Saving this data for reuse<jupyter_code>processed_df.to_csv("../data/tokenized.csv")<jupyter_output><empty_output><jupyter_text>Visualizing the most common tokens and saving the plot<jupyter_code>fig = plt.figure(figsize = (10,4))
plt.gcf().subplots_adjust(bottom=0.15)

word_counts = list(itertools.chain(*tokens))
freq_dist = FreqDist(word_counts)
freq_dist.plot(30, cumulative=False)

plt.show()
fig.savefig('images/freqDist.png', bbox_inches = "tight")<jupyter_output><empty_output><jupyter_text>Some steps that we did not include for preprocessing that could affect our model:- Removing numbers: sometimes in reviews we see that users explain their rationale of giving a specific number of star rating, so we did not want to remove them.- Removing chinese reviews and apps: We assumed that China does not have access to Google Playstore, so there will be lesser chinese reviews and apps made in Chinese, but this is certainly not true. However, filtering them out is a hassle atm so we did not do this step.- Spell Checking every token: We tried this, but it causes the computation to be taking too long! So we ignore it for now...  Modelling We will use `VADER SentimentIntensityAnalyzer` for calculating polarity scores for each review. The scoring is computed as following from https://github.com/cjhutto/vaderSentiment:- The `compound` score is computed by summing the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). This is the most useful metric if you want a single unidimensional measure of sentiment for a given sentence. Calling it a 'normalized, weighted composite score' is accurate.    - positive sentiment: compound score >= 0.05    - neutral sentiment: (compound score > -0.05) and (compound score < 0.05)    - negative sentiment: compound score <= -0.05The `pos`, `neu`, and `neg` scores are ratios for proportions of text that fall in each category (so these should all add up to be 1... or close to it with float operation). These are the most useful metrics if you want multidimensional measures of sentiment for a given sentence.<jupyter_code>analyzer = SentimentIntensityAnalyzer()
def sentiment_polarity(s):
    global analyzer
    polarity_scores = analyzer.polarity_scores(s)
    compound_score = polarity_scores["compound"]
    if compound_score >= 0.5:
        label = "Positive"
    elif compound_score > -0.05 and compound_score < 0.05:
        label = "Neutral"
    else:
        label = "Negative"
    return label, polarity_scores["neu"], polarity_scores["pos"], polarity_scores["neg"], compound_score
df = processed_df
df["Sentiment"], df["Neutral Proportion"], df["Positive Proportion"], df["Negative Proportion"], df["Compound Score"] =  zip(*df["Review"].apply(sentiment_polarity))
df.sample(3)
df.to_csv("../data/reviews_naive_polarity.csv")<jupyter_output><empty_output><jupyter_text>Now we will classify the reviews based on subjectivity using a `NaiveBayesClassifier`. First, we split the data for testing and training.<jupyter_code>test_size = int(0.2*len(reviews))
x_train = df[:test_size]
x_test = df[test_size:]<jupyter_output><empty_output><jupyter_text>Next, we build a vocabulary consisting of all the words that the training set of reviews have. From the output, you can see that there is still much to be improved upon for the tokens that were obtained despite the preprocessing.<jupyter_code>vocabulary = set(word_counts)
print("Number of words in vocabulary: {}".format(len(vocabulary)))
vocabulary<jupyter_output>Number of words in vocabulary: 22322<jupyter_text>Then for each data point (tokens with the pos/neg tag), we will build a dictionary that says whether it has particular features/ words from the vocabulary.<jupyter_code>def features_with_labels(row):
    return ({i:(i in row["Review"]) for i in vocabulary}, row["Sentiment"])

features = x_train.progress_apply(features_with_labels, axis=1)
classifier = NaiveBayesClassifier.train(features)
classifier.show_most_informative_features()
def features(row):
    return {i:(i in row["Review"]) for i in vocabulary}

test_features = x_test.progress_apply(features,axis=1)<jupyter_output>72%|███████▏  | 16024/22396 [37:36<14:46,  7.19it/s]
{'fraction_non_alphanumeric': 0.0897671568627451, 'fraction_numerical': 0.011029411764705883, 'mean_word_length': 5.159119496855346, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 54, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '381038', 'license': 'MIT', 'n_tokens_mistral': 3057, 'n_tokens_neox': 2892, 'n_words': 950, 'path': 'analysis/prelim_nlp_model.ipynb', 'repo_name': 'lyqht/googleplaystore-analytics', 'size': '9792'}