<jupyter_start><jupyter_text>EDA Let us check the rating distribution<jupyter_code>x = business['stars'].value_counts().index
y = business['stars'].value_counts().values
plt.figure(figsize=(9,6))
ax= sns.barplot(x, y,data= business ,alpha=0.8 )
plt.title("Ratings Distribution")
plt.xlabel('Ratings ', fontsize=12)<jupyter_output><empty_output><jupyter_text>Let us check the review count based on the rating<jupyter_code>plt.figure(figsize=(9,6))
ax= sns.barplot(x = 'stars', y='review_count',data= business ,alpha=0.8 )
plt.title("Ratings Distribution")
plt.xlabel('Ratings ', fontsize=12)
business['categories'].head()
business_cat=' '.join(business['categories'])
categry=pd.DataFrame(business_cat.split(';'),columns=['category'])
x = categry.category.value_counts()
x=x.sort_values(ascending=False)
x=x.iloc[0:20]<jupyter_output><empty_output><jupyter_text>Types of Business<jupyter_code>plt.figure(figsize=(16,4))
ax = sns.barplot(x.index, x.values, alpha=0.8)
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.show()
x = business['city'].value_counts().sort_values(ascending = False)
x=x.iloc[0:25]<jupyter_output><empty_output><jupyter_text>Cities with most business<jupyter_code>plt.figure(figsize=(16,4))
ax = sns.barplot(x.index, x.values, alpha=0.8)
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.show()
x = business['name'].value_counts().sort_values(ascending = False)

x=x.iloc[0:25]<jupyter_output><empty_output><jupyter_text>Most reviwed business<jupyter_code>plt.figure(figsize=(16,4))
ax = sns.barplot(x.index, x.values, alpha=0.8)
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.show()
busi_attr = pd.read_csv('yelp_review.csv') 
busi_attr = busi_attr[:100000]<jupyter_output><empty_output><jupyter_text>As the data is huge it is not possible for my system to perform analysis on the entire dataset, but if it is possible with your system try using more reviews.  I have selected data from 100,000 reviews.  EDA on the reviews<jupyter_code>from collections import Counter
from nltk.tokenize import RegexpTokenizer
from stop_words import get_stop_words
import nltk
from nltk.corpus import stopwords
from nltk import sent_tokenize, word_tokenize
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\Akshay\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>Let us create a bag of words consisting of all the reviews!<jupyter_code>a = busi_attr['text'].str.lower().str.cat(sep=' ')
import re
b = re.sub('[^A-Za-z]+', ' ', a)
b[:1000]
stop_words = list(get_stop_words('en'))         
nltk_words = list(stopwords.words('english'))   
stop_words.extend(nltk_words)
word_tokens = word_tokenize(b)
len(word_tokens)
filtered_sentence = []
for w in word_tokens:
    if w not in stop_words:
        filtered_sentence.append(w)
len(filtered_sentence)<jupyter_output><empty_output><jupyter_text>After removing the stop words, we can see reduction of size by 50 percent<jupyter_code># Remove characters which have length less than 2  
without_single_chr = [word for word in filtered_sentence if len(word) > 2]

# Remove numbers
cleaned_data_title = [word for word in without_single_chr if not word.isnumeric()]<jupyter_output><empty_output><jupyter_text>Let us find the most frequently used words in the reviews!<jupyter_code>top_N = 100
word_dist = nltk.FreqDist(cleaned_data_title)
rslt = pd.DataFrame(word_dist.most_common(top_N),
                    columns=['Word', 'Frequency'])

plt.figure(figsize=(10,10))
sns.set_style("whitegrid")
ax = sns.barplot(x="Word",y="Frequency", data=rslt.head(7))<jupyter_output><empty_output><jupyter_text>Let us create a wordcloud<jupyter_code>from wordcloud import WordCloud, STOPWORDS
def wc(data,bgcolor,title):
    plt.figure(figsize = (100,100))
    wc = WordCloud(background_color = bgcolor, max_words = 1000,  max_font_size = 50)
    wc.generate(' '.join(data))
    plt.imshow(wc)
    plt.axis('off')
wc(cleaned_data_title,'black','Most Used Words')<jupyter_output><empty_output><jupyter_text>Let us try to perform analysis on the entire review rather than all the words. For this we make use of the TextBlob<jupyter_code>from textblob import TextBlob

bloblist_desc = list()

df_review_str=busi_attr['text'].astype(str)
for row in df_review_str:
    blob = TextBlob(row)
    bloblist_desc.append((row,blob.sentiment.polarity, blob.sentiment.subjectivity))
    df_polarity_desc = pd.DataFrame(bloblist_desc, columns = ['Review','sentiment','polarity'])
df_polarity_desc.head()<jupyter_output><empty_output><jupyter_text>Based on my analysis, I have selected these values however you could use based on your insights<jupyter_code>def f(df_polarity_desc):
    if df_polarity_desc['sentiment'] > 0:
        val = "Positive Review"
    elif df_polarity_desc['sentiment'] == 0:
        val = "Neutral Review"
    else:
        val = "Negative Review"
    return val
df_polarity_desc['Sentiment_Type'] = df_polarity_desc.apply(f, axis=1)

plt.figure(figsize=(10,10))
sns.set_style("whitegrid")
ax = sns.countplot(x="Sentiment_Type", data=df_polarity_desc)
positive_reviews=df_polarity_desc[df_polarity_desc['Sentiment_Type']=='Positive Review']
negative_reviews=df_polarity_desc[df_polarity_desc['Sentiment_Type']=='Negative Review']
negative_reviews.head()<jupyter_output><empty_output><jupyter_text>Let us look at the wordcloud of the most used words in a positive review<jupyter_code>wc(positive_reviews['Review'],'black','Most Used Words')<jupyter_output><empty_output><jupyter_text>Let us look at the wordcloud of the most used words in a negative review<jupyter_code>wc(negative_reviews['Review'],'black','Most Used Words')<jupyter_output><empty_output><jupyter_text>Using machine learning to predict whether a review has 1 star rating or 5 star rating Let us now train a model We are taking only review with ratings 1 and 5 to perform the analysis to make the analysis more simple.<jupyter_code>busi_attr=busi_attr.dropna(axis=0,how='any')
rating_class = busi_attr[(busi_attr['stars'] == 1) | (busi_attr['stars'] == 5)]
X_review=rating_class['text']
y=rating_class['stars']
import string
def text_process(review):
    nopunc=[word for word in review if word not in string.punctuation]
    nopunc=''.join(nopunc)
    return [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]
from sklearn.feature_extraction.text import CountVectorizer
bow_transformer=CountVectorizer(analyzer=text_process).fit(X_review)
X_review = bow_transformer.transform(X_review)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_review, y, test_size=0.3, random_state=101)
X_train<jupyter_output><empty_output><jupyter_text>Using SVM<jupyter_code>from sklearn.svm import SVC
sv_model = SVC()
sv_model.fit(X_train, y_train)
Y_pred = sv_model.predict(X_test)
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
print(confusion_matrix(y_test, Y_pred))
print('\n Accuracy:')
print(accuracy_score(y_test, Y_pred))
print(classification_report(y_test, Y_pred))<jupyter_output>[[   64  3887]
 [    0 12845]]

 Accuracy:
0.7685758513931888
             precision    recall  f1-score   support

          1       1.00      0.02      0.03      3951
          5       0.77      1.00      0.87     12845

avg / total       0.82      0.77      0.67     16796<jupyter_text>Using Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
lg_model = LogisticRegression()
lg_model.fit(X_train, y_train)
Y_pred = lg_model.predict(X_test)
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
print(confusion_matrix(y_test, Y_pred))
print('\n Accuracy:')
print(accuracy_score(y_test, Y_pred))
print(classification_report(y_test, Y_pred))<jupyter_output>[[ 3566   385]
 [  218 12627]]

 Accuracy:
0.9640985949035484
             precision    recall  f1-score   support

          1       0.94      0.90      0.92      3951
          5       0.97      0.98      0.98     12845

avg / total       0.96      0.96      0.96     16796<jupyter_text>Using Naive Bayes<jupyter_code>from sklearn.naive_bayes import MultinomialNB
nb = MultinomialNB()
nb.fit(X_train, y_train)
predict=nb.predict(X_test)
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
print(confusion_matrix(y_test, predict))
print('\n Accuracy:')
print(accuracy_score(y_test, predict))
print(classification_report(y_test, predict))<jupyter_output>[[ 3474   477]
 [  550 12295]]

 Accuracy:
0.9388544891640866
             precision    recall  f1-score   support

          1       0.86      0.88      0.87      3951
          5       0.96      0.96      0.96     12845

avg / total       0.94      0.94      0.94     16796
{'fraction_non_alphanumeric': 0.11574178935447339, 'fraction_numerical': 0.036919592298980744, 'mean_word_length': 5.4039158810732415, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '29577', 'license': 'MIT', 'n_tokens_mistral': 3333, 'n_tokens_neox': 3096, 'n_words': 665, 'path': 'Yelp.ipynb', 'repo_name': 'akshaykapoor347/Yelp-reviews', 'size': '8830'}