<jupyter_start><jupyter_text>This notebook builds ontoo the previous one by trying out this workflow but with only comments of 100-300 words, to see how that works on things.This notebook also wants to further improve the sentiment analysis, so will try to use ScatterText and DeepMoji maybe<jupyter_code>import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler

import matplotlib.pyplot as plt
review_json_path = 'Books_5.json'

size = 1000000
review = pd.read_json(review_json_path, lines=True,
                      chunksize=size)


for chunk_review in review:
    sample_list = pd.DataFrame(chunk_review[['reviewerID','asin','overall']])
    
    print('run')

chunk_review.sample(30)
reviews = chunk_review[['reviewerID','asin','reviewText','overall']]
reviews.dtypes
reviews['reviewText'] = reviews['reviewText'].astype('str')
reviews['reviewLen'] = reviews['reviewText'].str.split().map(lambda x: len(x))

reviews.head(10)
reviews = reviews.loc[(reviews['reviewLen'] > 100) & (reviews['reviewLen'] < 300)]

reviews<jupyter_output><empty_output><jupyter_text>Got my starting dataframe with the reviews. Should filter them a bit before I work on them. After I filter out some comments, I will need to do a couple things:1. Processing, tokenizing, stemming, leminization, those kind of word processing things. Can remove some of the noisy irrelevant stuff2. Try out the different sentiment analysis packages3. Other NLP techniques? Don't know4. Iterate on these Let's try out Vader first to see how it performs. Since I used this before, just to get a first stab at it.<jupyter_code>reviews_samp = reviews.sample(500)

reviews_samp
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
analyser = SentimentIntensityAnalyzer()

def sentiment_analyzer_scores(sentence):
    score = analyser.polarity_scores(sentence)
    return score

def create_sentiments_vader(reviews):

    reviews['text_sentiment'] = reviews['reviewText'].apply(sentiment_analyzer_scores)

    reviews['compound'] = [d.get('compound') for d in reviews['text_sentiment']]
    reviews['neg'] = [d.get('neg') for d in reviews['text_sentiment']]
    reviews['neu'] = [d.get('neu') for d in reviews['text_sentiment']]
    reviews['pos'] = [d.get('pos') for d in reviews['text_sentiment']]
    
    return reviews

reviews_samp_sent = create_sentiments_vader(reviews_samp)

reviews_samp_sent<jupyter_output><empty_output><jupyter_text>Seems a little iffy on first pass. Should kind of check more. Let's try textblob package<jupyter_code># Create quick lambda functions to find the polarity and subjectivity of each routine
# Terminal / Anaconda Navigator: conda install -c conda-forge textblob
from textblob import TextBlob

pol = lambda x: TextBlob(x).sentiment.polarity
sub = lambda x: TextBlob(x).sentiment.subjectivity

reviews_samp_sent['polarity'] = reviews_samp_sent['reviewText'].apply(pol)
reviews_samp_sent['subjectivity'] = reviews_samp_sent['reviewText'].apply(sub)
reviews_samp_sent<jupyter_output><empty_output><jupyter_text>Seems quite iffy too, and results are quite different from Vader it seems. Definetely need to try out processing of text. Let's try flair first, since it seems to work fairly well<jupyter_code>from flair.models import TextClassifier
from flair.data import Sentence
import re

def isfloat(value):
  try:
    float(value)
    return True
  except ValueError:
    return False

sia = TextClassifier.load('en-sentiment')
def flair_prediction(x):
    sentence = Sentence(x)
    sia.predict(sentence)
    score = str(sentence.labels[0])
    score = score.replace('(',' ').replace(')',' ')
    number = [float(s) for s in score.split() if isfloat(s) is True]
    
    if "POSITIVE" in score:
        return number[0]
    elif "NEGATIVE" in score:
        return -number[0]
from sklearn.preprocessing import MinMaxScaler

reviews_samp_sent["flair_sentiment"] = reviews_samp_sent["reviewText"].apply(flair_prediction)
scaler = MinMaxScaler()

reviews_samp_sent['flair_scaled'] = scaler.fit_transform(reviews_samp_sent[["flair_sentiment"]])
reviews_samp_sent.head(30)<jupyter_output><empty_output><jupyter_text>This is really troublesome flair sentinment rating. Need to transform it. So will try a few transformations out here.- First gonna try an evenly spaced out flair rating, see if that works.<jupyter_code>review_transform = reviews_samp_sent[['overall','flair_sentiment']]
from sklearn.preprocessing import PowerTransformer, QuantileTransformer, FunctionTransformer
pt = PowerTransformer()

review_transform['flair_power'] = pt.fit_transform(X = review_transform[['flair_sentiment']])

review_transform
qt = QuantileTransformer()

review_transform['flair_quantile'] = qt.fit_transform(X = review_transform[['flair_sentiment']])

review_transform
qt = QuantileTransformer(output_distribution = 'normal')

review_transform['flair_quantile_normal'] = qt.fit_transform(X = review_transform[['flair_sentiment']])

review_transform
    import matplotlib.pyplot as plt

    plt.hist(review_transform['flair_quantile'])
    plt.show()<jupyter_output><empty_output><jupyter_text>Need to smoooth out the curve a bit...<jupyter_code>reviews_samp_sent.tail(40)<jupyter_output><empty_output><jupyter_text>To better test out how this performs, let's sample one row and see how the scores are compared to the full text<jupyter_code>samp_text = reviews_samp_sent.sample(1)
pd.set_option('display.max_colwidth', None)
print(samp_text['reviewText'])

print(samp_text)
import scattertext as st
reviews_sample = reviews_samp_sent.copy()

reviews_sample
reviews_sample['Liked'] = reviews_sample.overall.apply(lambda x: x > 3).map({True: 'Liked', False: 'Not Liked'})

reviews_sample['Liked']
corpus = st.CorpusFromPandas(
    reviews_sample,
    category_col = 'Liked',
    text_col = 'reviewText',
    nlp=st.whitespace_nlp_with_sentences
).build()

html = st.produce_scattertext_explorer(
        corpus,
        category="Liked",
        category_name='Liked',
        not_category_name='Not Liked',
        minimum_term_frequency=10,
        pmi_threshold_coefficient=5,
        width_in_pixels=1000,
        metadata=reviews_sample['asin']
        )

open('reviews_scatter.html', 'wb').write(html.encode('utf-8'));<jupyter_output><empty_output><jupyter_text>Scatterplot is pretty neat, let's see if it holds for the Flair Data lol.<jupyter_code>reviews_sample['Liked'] = reviews_sample.flair_scaled.apply(lambda x: x > 0.5).map({True: 'Liked', False: 'Not Liked'})

reviews_sample['Liked']
corpus = st.CorpusFromPandas(
    reviews_sample,
    category_col = 'Liked',
    text_col = 'reviewText',
    nlp=st.whitespace_nlp_with_sentences
).build()

html = st.produce_scattertext_explorer(
        corpus,
        category="Liked",
        category_name='Liked',
        not_category_name='Not Liked',
        minimum_term_frequency=10,
        pmi_threshold_coefficient=5,
        width_in_pixels=1000,
        metadata=reviews_sample['asin']
        )

open('reviews_scatter_flair.html', 'wb').write(html.encode('utf-8'));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10653308875405672, 'fraction_numerical': 0.00818399887117257, 'mean_word_length': 5.587360594795539, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '284819', 'license': 'MIT', 'n_tokens_mistral': 2374, 'n_tokens_neox': 2296, 'n_words': 552, 'path': '5 - amazon_recommendation_system/notebooks/NLP_EDA_Sentiments-V4.ipynb', 'repo_name': 'kenhua15/Metis-Projects', 'size': '7087'}