<jupyter_start><jupyter_text>Natural Language Processing Micro ProjectIn this NLP project we will attempt to classify Yelp Reviews into 1 star or 5 star categories based off the text content in the reviews. We will use the [Yelp Review Data Set from Kaggle](https://www.kaggle.com/c/yelp-recsys-2013).<jupyter_code>import pandas as pd
import numpy as np

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>DataEach observation in this dataset is a review of a particular business by a particular user.The "stars" column is the number of stars (1 through 5) assigned by the reviewer to the business. (Higher stars is better.) In other words, it is the rating of the business by the person who wrote the review.The "cool" column is the number of "cool" votes this review received from other Yelp users. All reviews start with 0 "cool" votes, and there is no limit to how many "cool" votes a review can receive. In other words, it is a rating of the review itself, not a rating of the business.The "useful" and "funny" columns are similar to the "cool" column.<jupyter_code>yelp = pd.read_csv('data/yelp.csv')
yelp.head(3)
yelp.info()
yelp.describe()
#Creating a new column to denote the length of the text
yelp['text length'] = yelp['text'].apply(len)<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>sns.set_style('whitegrid')<jupyter_output><empty_output><jupyter_text>Let's create a grid of 5 histograms of text length based off of the star ratings.<jupyter_code>g = sns.FacetGrid(yelp,col='stars')
g.map(plt.hist,'text length')<jupyter_output><empty_output><jupyter_text>Visualizing the counts for each rating type.<jupyter_code>sns.countplot(x='stars',data=yelp)<jupyter_output><empty_output><jupyter_text>Grouping the reviews on the basis of the stars.<jupyter_code>stars =  yelp.groupby('stars').mean()
stars<jupyter_output><empty_output><jupyter_text>Let's check out the correlation between different features for the reviews.<jupyter_code>stars.corr()<jupyter_output><empty_output><jupyter_text>And visualize this correlation:<jupyter_code>sns.heatmap(stars.corr(),annot=True)<jupyter_output><empty_output><jupyter_text>NLP ClassificationMoving on to the actual task: we'll just grab the reviews that were either 1 star or 5 stars, to make life easier in this micro-project.<jupyter_code>yelp_class = yelp[(yelp['stars']==1) | (yelp['stars']==5) ]
yelp_class.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4086 entries, 0 to 9999
Data columns (total 11 columns):
business_id    4086 non-null object
date           4086 non-null object
review_id      4086 non-null object
stars          4086 non-null int64
text           4086 non-null object
type           4086 non-null object
user_id        4086 non-null object
cool           4086 non-null int64
useful         4086 non-null int64
funny          4086 non-null int64
text length    4086 non-null int64
dtypes: int64(5), object(6)
memory usage: 383.1+ KB<jupyter_text>Our data will be the text content of the reviews, and we'll try to predict if it's a 1 star or a 5 star review. So let's assign those to our X and y variables.<jupyter_code>X = yelp_class['text']
y = yelp_class['stars']
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
X = cv.fit_transform(X)<jupyter_output><empty_output><jupyter_text>Train Test Split<jupyter_code>from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3,random_state=101)
from sklearn.naive_bayes import MultinomialNB

nb = MultinomialNB()
nb.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Predictions and Evaluations<jupyter_code>preds = nb.predict(X_test)<jupyter_output><empty_output><jupyter_text>** Create a confusion matrix and classification report using these predictions and y_test **<jupyter_code>from sklearn.metrics import confusion_matrix,classification_report
print(confusion_matrix(y_test,preds))
print(classification_report(y_test,preds))<jupyter_output>[[159  69]
 [ 22 976]]
             precision    recall  f1-score   support

          1       0.88      0.70      0.78       228
          5       0.93      0.98      0.96       998

avg / total       0.92      0.93      0.92      1226
{'fraction_non_alphanumeric': 0.08779244845957841, 'fraction_numerical': 0.034051424600416956, 'mean_word_length': 4.9476584022038566, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '557236', 'license': 'MIT', 'n_tokens_mistral': 1546, 'n_tokens_neox': 1389, 'n_words': 434, 'path': '.ipynb_checkpoints/NLP Project -checkpoint.ipynb', 'repo_name': 'hritwizsri/data-science-portfolio', 'size': '4317'}