<jupyter_start><jupyter_text>1. Read Dataset<jupyter_code>import pandas as pd
Ebuss_df = pd.read_csv("sample30.csv")
#from google.colab import drive
#drive.mount('/content/drive')

<KEY>
Ebuss_df.head()<jupyter_output><empty_output><jupyter_text>2. Exploratory data analysis<jupyter_code>null_data = Ebuss_df.isnull().sum()
null_data
import matplotlib.pyplot as plt
import seaborn as sns
fig = plt.figure(figsize=(50, 20))
data_plot = plt.plot(null_data.index,null_data,label='Null Count Across Attributes',color='r')
plt.title('Null Frequencies',fontsize=14)
plt.xlabel('Attributes',color = 'g',fontsize=14)
plt.ylabel('Null Count',color = 'b',fontsize=14,labelpad=15)
plt.legend()
plt.tick_params(labelsize=15);
plt.show()

#!pip install mpld3
#import mpld3
#labels = ['point {0}'.format(i + 1) for i in range(100)]
#tooltip = mpld3.plugins.PointLabelTooltip(data_plot, labels=labels)
#mpld3.plugins.connect(fig, tooltip)
#mpld3.show()<jupyter_output><empty_output><jupyter_text>Observations :> missing values are maximum for below mentioned attributes.a. reviews_didPurchase b. reviews_userCityc. reviews_userProvince  We have to predict review_rating based on the review_text. So we'll be ultimately concentrating there but let's perform EDA on other attributes to understand the dataset  Top 10 brands<jupyter_code>Ebuss_df["brand"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Top 10 categories<jupyter_code>Ebuss_df["categories"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Top 10 manufacturer<jupyter_code>Ebuss_df["manufacturer"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Top 10 Product name<jupyter_code>Ebuss_df["name"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Whether a particular user has purchased the product or not<jupyter_code>Ebuss_df["reviews_didPurchase"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Whether a particular user has recommended the product or not<jupyter_code>Ebuss_df["reviews_doRecommend"].value_counts().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Top 5 reviews_rating<jupyter_code>Ebuss_df["reviews_rating"].value_counts().sort_values(ascending=False).head(5)
ratings = Ebuss_df["reviews_rating"].value_counts()
fig = plt.figure(figsize=(50, 20))

plt.bar(ratings.index, ratings.values, color ='g',
        width = 0.4)
 
plt.xlabel("Ratings")
plt.ylabel("Frequencies")
plt.title("Ratings distribution")
plt.show()
ratings.values<jupyter_output><empty_output><jupyter_text>Now trying to find the length of review comments for different review ratings . So first we are trying to derive reviews length<jupyter_code>Ebuss_df['reviews_length']=Ebuss_df['reviews_text'].apply(len)
Ebuss_df['reviews_length']
gk = Ebuss_df.groupby(['reviews_rating','reviews_length'])

gk.first()

reviews = Ebuss_df.groupby('reviews_rating')['reviews_length'].mean().sort_values()

fig = plt.figure(figsize=(50, 20))

plt.bar(reviews.index, reviews.values, color ='g',
        width = 0.4)
 
plt.xlabel("Review Rating")
plt.ylabel("Avg Review Length")
plt.title("Ratings Length Sentiment Analysis")
plt.show()<jupyter_output><empty_output><jupyter_text>Observations : We have observed that most users have given 5 ratings but they have written more in detail while giving 1 rating  User Sentiments Stat<jupyter_code>user_sentiment = Ebuss_df["user_sentiment"].value_counts()

fig = plt.figure(figsize=(50, 20))

plt.bar(user_sentiment.index, user_sentiment.values, color ='g',
        width = 0.4)
 
plt.xlabel("User Sentiment")
plt.ylabel("Frequencies")
plt.title("user_sentiment Distribution")
plt.show()<jupyter_output><empty_output><jupyter_text>Observation - Postive response are more<jupyter_code>sns.heatmap(Ebuss_df.corr(),cmap='coolwarm',annot=True,linewidths=.5)<jupyter_output><empty_output><jupyter_text>Observation - review_ratings are negatively correlated to review comment size<jupyter_code>Ebuss_df.shape<jupyter_output><empty_output><jupyter_text>Observation - 30,000 rows * 16 columns<jupyter_code>Ebuss_df.describe()
Ebuss_df.dtypes<jupyter_output><empty_output><jupyter_text>3. Data cleaning<jupyter_code>Ebuss_df_subset = Ebuss_df.dropna(subset=['reviews_text'])<jupyter_output><empty_output><jupyter_text>Observation - dropping rows where review_text is NULL and storing it into a new dataframe<jupyter_code>X = Ebuss_df_subset['reviews_text']
y = Ebuss_df_subset['reviews_rating']<jupyter_output><empty_output><jupyter_text>4. Feature extraction: In order to extract features from the text data, you may choose from any of the methods, including bag-of-words, TF-IDF vectorization or word embedding.<jupyter_code>from sklearn.model_selection import cross_val_score
from scipy.sparse import hstack
from sklearn.feature_extraction.text import TfidfVectorizer



review_text=Ebuss_df_subset['reviews_text']
train_text=Ebuss_df_subset['reviews_text']
type(train_text)<jupyter_output><empty_output><jupyter_text>TF-IDF<jupyter_code>word_vectorizer = TfidfVectorizer(
    sublinear_tf=True,
    strip_accents='unicode',
    analyzer='word',
    token_pattern=r'\w{1,}',
    stop_words='english',
    ngram_range=(1, 1),
    max_features=10000)
word_vectorizer.fit(review_text)
train_word_features = word_vectorizer.transform(train_text)
char_vectorizer = TfidfVectorizer(
    sublinear_tf=True,
    strip_accents='unicode',
    analyzer='char',
    stop_words='english',
    ngram_range=(2, 6),
    max_features=50000)
char_vectorizer.fit(review_text)
train_char_features = char_vectorizer.transform(train_text)

train_features = hstack([train_char_features, train_word_features])<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/feature_extraction/text.py:520: UserWarning: The parameter 'stop_words' will not be used since 'analyzer' != 'word'
  warnings.warn("The parameter 'stop_words' will not be used"<jupyter_text>5. Training a text classification model: You need to build at least three ML models. You then need to analyse the performance of each of these models and choose the best model.   a. XGBOOST<jupyter_code>!pip install xgboost
import xgboost as xgb
xgb_model=xgb.XGBClassifier()<jupyter_output><empty_output><jupyter_text>Training & Test Data<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train_features, y,test_size=0.3,random_state=101)
xgb_model.fit(X_train,y_train)

xgb_model_preds =xgb_model.predict(X_test)

from sklearn.metrics import accuracy_score


xgb_accuracy=accuracy_score(xgb_model_preds,y_test)

print("XGBoost Model accuracy",xgb_accuracy)<jupyter_output>XGBoost Model accuracy 0.7428888888888889<jupyter_text>b. Random Forest Classifier Model<jupyter_code>from sklearn.ensemble import RandomForestClassifier

rfc_model = RandomForestClassifier()
rfc_model.fit(X_train,y_train)
rfc_preds=rfc_model.predict(X_test) 
rfc_accuracy=accuracy_score(rfc_preds,y_test)

print("RFC Model accuracy",rfc_accuracy)<jupyter_output>RFC Model accuracy 0.7573333333333333<jupyter_text>c. Logistics Regression Model<jupyter_code>from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression(max_iter = 1000)
lr_model.fit(X_train,y_train)
lr_preds=lr_model.predict(X_test) 
lr_accuracy=accuracy_score(lr_preds,y_test)

print("LR Model accuracy",lr_accuracy)<jupyter_output><empty_output><jupyter_text>Random forest is best among these 3 model. So it would be good to go with it  Saving the model<jupyter_code>import pickle
# Save the Modle to file in the current working directory

Pkl_Filename = "Pickle_RFC1_Model.pkl"  

with open(Pkl_Filename, 'wb') as file:  
    pickle.dump(rfc_model, file)
# Load the Model back from file
Pkl_Filename = "Pickle_RFC1_Model.pkl"  

with open(Pkl_Filename, 'rb') as file:  
    Pickled_RFC_Model = pickle.load(file)
    
Pickled_RFC_Model


from google.colab import files
files.download(Pkl_Filename)
# Save the Modle to file in the current working directory

Pkl_Filename = "Pickle_XGB1_Model.pkl"  

with open(Pkl_Filename, 'wb') as file:  
    pickle.dump(xgb_model, file)
# Load the Model back from file
Pkl_Filename = "Pickle_XGB1_Model.pkl"  

with open(Pkl_Filename, 'rb') as file:  
    Pickled_XGB_Model = pickle.load(file)
    
Pickled_XGB_Model
from google.colab import files
files.download(Pkl_Filename)
# Save the Modle to file in the current working directory

Pkl_Filename = "Pickle_LR1_Model.pkl"  

with open(Pkl_Filename, 'wb') as file:  
    pickle.dump(lr_model, file)
# Load the Model back from file
Pkl_Filename = "Pickle_LR1_Model.pkl"  

with open(Pkl_Filename, 'rb') as file:  
    Pickled_LR_Model = pickle.load(file)
    
Pickled_LR_Model
from google.colab import files
files.download(Pkl_Filename)
# Use the Reloaded Model to 
# Calculate the accuracy score and predict target values

# Calculate the Score 
score = Pickled_RFC_Model.score(X_test, y_test)  
# Print the Score
print("Test score: {0:.2f} %".format(100 * score))  

# Predict the Labels using the reloaded Model
Ypredict = Pickled_RFC_Model.predict(X_test)  

Ypredict
X_test[0]<jupyter_output><empty_output><jupyter_text>I have noticed that item-item based is providing a better result. Please refer notebook - "Final Capstone -Recommendation System by Rakesh Sept 9th.ipynb" for more details<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10610831234256927, 'fraction_numerical': 0.01658270361041142, 'mean_word_length': 7.678506375227687, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '464821', 'license': 'MIT', 'n_tokens_mistral': 3558, 'n_tokens_neox': 3377, 'n_words': 657, 'path': 'Final_Capstone_Sentiment_Model_by_Rakesh_Sept_9th.ipynb', 'repo_name': 'rr1989/capstone-project-recommendation-system-by-rr', 'size': '9586'}