<jupyter_start><jupyter_text>Sentiment analysisYour time to shine! We will use data from `Amazon Reviews: Unlocked Mobile Phones`, available on Kaggle platform and described as:> PromptCloud extracted 400 thousand reviews of unlocked mobile phones sold on Amazon.com to find out insights with respect to reviews, ratings, price and their relationships. \[...\] Data was acquired in December, 2016 by the crawlers build to deliver \[their\] data extraction services.([source](https://www.kaggle.com/PromptCloudHQ/amazon-reviews-unlocked-mobile-phones/data))  Load useful librairies and data<jupyter_code>import pandas as pd
import numpy as np

import nltk

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import roc_auc_score
# Load data
data = pd.read_csv('./data/644_1225_compressed_Amazon_Unlocked_Mobile.csv.zip', compression="zip")

# Let's keep only a fraction of the data to speed up computations
data = data.sample(frac=0.1, random_state=10)

# Drop missing values
data.dropna(inplace=True)

# Remove any 'neutral' ratings equal to 3
data = data[data['Rating'] != 3]

# Consider 4 and 5 as positive ratings (encoded as 1)
# and 1 and 2 as negative ones (encoded as 0)
data['positive_rating'] = np.where(data['Rating'] > 3, 1, 0)
data.sample(3)<jupyter_output><empty_output><jupyter_text>Data explorationBefore diving into the sentiment analysis, what can you tell me about the data?<jupyter_code>print("Dataset contains {} instances of {} variables.".format(data.shape[0], data.shape[1]))

print(
    "It contains {} positive ratings ({:.1%} of all)".format(
        data[data.positive_rating == 1].shape[0],
        data[data.positive_rating == 1].shape[0] / data.shape[0],
    )
)
print(
    "Examples of positive ratings: \n    {}\n    {}".format(
        data[data.positive_rating == 1].sample(1).Reviews.iloc[0],
        data[data.positive_rating == 1].sample(1).Reviews.iloc[0],
    )
)
print(
    "\nExamples of negative ratings: \n    {}\n    {}".format(
        data[data.positive_rating == 0].sample(1).Reviews.iloc[0],
        data[data.positive_rating == 0].sample(1).Reviews.iloc[0],
    )
)<jupyter_output>Examples of positive ratings: 
    The service was perfect, the phone very good but my only problem is with the battery.I leave it during the night on, and in the morning is 30% less charged..I have to charge it during mid-day in order to be up & running until the night.Has anyone else faced that?
    Only thing I would not link it with my email account, all my contacts were deleted. Not the phone is was google!

Examples of negative ratings: 
    I bought the phone because it said it was unlocked and it would work worldwide, turns out that the bandwidth that the phone can detect does not support the 3g network in europe, at least in Germany where i live. I bought it expecting it to work properly and here it can only detect gsm signal. So all of the features on the phone work slower and i can't really use it like i wanted. So if you want an international phone do not buy it, if it is for the states i guess it is ok!
    I knew I was buying a used phone but I wasn't expecting there to b[...]<jupyter_text>Suggestions from workshop participants<jupyter_code>data.skew()
data["Review Votes"].describe()
data.info()
import matplotlib
data['Reviews'].str.len().clip(0,1000).hist()<jupyter_output><empty_output><jupyter_text>Sentiment analysis<jupyter_code># Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    data['Reviews'], data['positive_rating'], random_state=0
)<jupyter_output><empty_output><jupyter_text>Let's investigate positive / negative review classification. With CountVectorizer* What would be the AUC score (on test data) of a classifier using CountVectorizer and a Logistic Regression model (with `max_iter=1500`)?<jupyter_code># Fit the CountVectorizer to the training data
vect = CountVectorizer().fit(X_train)

# transform the documents in the training data to a document-term matrix
X_train_vectorized = vect.transform(X_train)

# Train the model
model = LogisticRegression(max_iter=1500)
model.fit(X_train_vectorized, y_train)

# Predict the transformed test documents
predictions = model.predict(vect.transform(X_test))

print("AUC = {:.3f}".format(roc_auc_score(y_test, predictions)))<jupyter_output>AUC = 0.897<jupyter_text>* With the classifier built above, which 20 tokens are the most associated with negative reviews?<jupyter_code># get the feature names as numpy array
feature_names = np.array(vect.get_feature_names())

# Sort the coefficients from the model
sorted_coef_index = model.coef_[0].argsort()

# Find the 20 smallest coefficients
print("Smallest Coefs:\n{}\n".format(feature_names[sorted_coef_index[:20]]))<jupyter_output>Smallest Coefs:
['worst' 'terrible' 'slow' 'junk' 'poor' 'sucks' 'horrible' 'useless'
 'waste' 'disappointed' 'broke' 'broken' 'sucked' 'defective' 'garbage'
 'wasn' 'freezes' 'wouldn' 'died' 'doesn']<jupyter_text>* With the classifier built above, which 20 tokens are the most associated with positive reviews?<jupyter_code># Find the 20 largest coefficients
print("Largest Coefs: \n{}".format(feature_names[sorted_coef_index[:-21:-1]]))<jupyter_output>Largest Coefs: 
['excelent' 'excelente' 'excellent' 'perfectly' 'love' 'perfect' 'exactly'
 'great' 'best' 'awesome' 'loves' 'amazing' 'wonderful' 'fantastic' 'nice'
 'exelente' 'good' 'bueno' 'loving' 'perfecto']<jupyter_text>With TF-IDF Vectorizer* What would be the AUC score (on test data) of a classifier using TfidfVectorizer and a Logistic Regression model (with `max_iter=1500`)?<jupyter_code># TfidfVectorizer specifiying a minimum document frequency of 3
vect = TfidfVectorizer(min_df=3).fit(X_train)

# transform the documents in the training data to a document-term matrix
X_train_vectorized = vect.transform(X_train)

# Train the model
model = LogisticRegression(max_iter=1500)
model.fit(X_train_vectorized, y_train)

# Predict the transformed test documents
predictions = model.predict(vect.transform(X_test))

print("AUC = {:.3f}".format(roc_auc_score(y_test, predictions)))<jupyter_output>AUC = 0.890<jupyter_text>* Which tokens have the 10 smallest and 10 largest TF-IDF coefficients?<jupyter_code>feature_names = np.array(vect.get_feature_names())

sorted_tfidf_index = X_train_vectorized.max(0).toarray()[0].argsort()

print("Smallest tfidf:\n{}\n".format(feature_names[sorted_tfidf_index[:10]]))
print("Largest tfidf: \n{}".format(feature_names[sorted_tfidf_index[:-11:-1]]))<jupyter_output>Smallest tfidf:
['peppy' 'puffin' 'funddo' 'oblong' 'solute' 'benzene' 'corrosive'
 'callsaccessories' 'calories' 'stewards']

Largest tfidf: 
['case' 'amazing' 'hi' 'too' 'top' 'tops' 'very' 'exelente' 'exelent'
 'christopher']<jupyter_text>* With the classifier built above, which 20 tokens are the most associated with negative reviews?<jupyter_code># Sort the coefficients from the model
sorted_coef_index = model.coef_[0].argsort()

# Find the 20 smallest coefficients
print("Smallest Coefs:\n{}\n".format(feature_names[sorted_coef_index[:20]]))<jupyter_output>Smallest Coefs:
['not' 'slow' 'disappointed' 'worst' 'terrible' 'return' 'never' 'doesn'
 'horrible' 'waste' 'poor' 'months' 'broken' 'sucks' 'broke' 'wasn' 'work'
 'junk' 'off' 'useless']<jupyter_text>* With the classifier built above, which 20 tokens are the most associated with positive reviews?<jupyter_code># Find the 20 largest coefficients
print("Largest Coefs: \n{}".format(feature_names[sorted_coef_index[:-21:-1]]))<jupyter_output>Largest Coefs: 
['great' 'love' 'excellent' 'good' 'best' 'perfect' 'price' 'awesome'
 'far' 'perfectly' 'excelente' 'nice' 'amazing' 'easy' 'excelent'
 'exactly' 'loves' 'fast' 'happy' 'my']<jupyter_text>Testing our sentiment analyizerAre our classifiers able to discriminate the following reviews:* 'not an issue, phone is working'* 'an issue, phone is not working'?<jupyter_code>print(model.predict(vect.transform(['not an issue, phone is working',
                                    'an issue, phone is not working'])))<jupyter_output>[0 0]<jupyter_text>What seems to be the issue?  Toward improving our positive / negative review classifier_Hint to move forward_ : Consider the option `ngram_range` of `CountVectorizer`<jupyter_code># In addition to words alone, let's consider 2-word combinations
vect = CountVectorizer(min_df=5, ngram_range=(1,2)).fit(X_train)

X_train_vectorized = vect.transform(X_train)

model = LogisticRegression(max_iter=1500)
model.fit(X_train_vectorized, y_train)

predictions = model.predict(vect.transform(X_test))

print("AUC = {:.3f}".format(roc_auc_score(y_test, predictions)))
feature_names = np.array(vect.get_feature_names())

sorted_coef_index = model.coef_[0].argsort()

print('Smallest Coefs:\n{}\n'.format(feature_names[sorted_coef_index[:10]]))
print('Largest Coefs: \n{}'.format(feature_names[sorted_coef_index[:-11:-1]]))<jupyter_output>Smallest Coefs:
['no good' 'junk' 'poor' 'slow' 'worst' 'broken' 'not good' 'terrible'
 'defective' 'horrible']

Largest Coefs: 
['excellent' 'excelente' 'excelent' 'perfect' 'great' 'love' 'awesome'
 'no problems' 'good' 'best']<jupyter_text>Is the issue now fixed?<jupyter_code>print(model.predict(vect.transform(['not an issue, phone is working',
                                    'an issue, phone is not working'])))<jupyter_output>[1 0]
{'fraction_non_alphanumeric': 0.11836178904367864, 'fraction_numerical': 0.015188017178171153, 'mean_word_length': 5.844444444444444, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 48, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '196878', 'license': 'MIT', 'n_tokens_mistral': 3298, 'n_tokens_neox': 3092, 'n_words': 934, 'path': 'slides_and_solution/part_2_sentiment_analysis_a_solution.ipynb', 'repo_name': 'pyladieshamburg/getting-started-with-nlp', 'size': '9547'}