<jupyter_start><jupyter_text>Bag of Words**Step 1 : Preprocess raw reviews to cleaned reviews****Step 2 : Create BoW using CountVectorizer / Tfidfvectorizer in sklearn****Step 3 : Transform review text to numerical representations (feature vectors)****Step 4 : Fit feature vectors to supervised learning algorithm (eg. Naive Bayes, Logistic regression, etc.)****Step 5 : Improve the model performance by GridSearch** Text Preprocessing**Step 1 : remove html tags using BeautifulSoup****Step 2 : remove non-character such as digits and symbols****Step 3 : convert to lower case****Step 4 : remove stop words such as "the" and "and" if needed****Step 5 : convert to root words by stemming if needed**<jupyter_code>def cleanText(raw_text, remove_stopwords=False, stemming=False, split_text=False, \
             ):
    '''
    Convert a raw review to a cleaned review
    '''
    text = BeautifulSoup(raw_text, 'html.parser').get_text()
    letters_only = re.sub("[^a-zA-Z]", " ", text)
    words = letters_only.lower().split() 
    
    if remove_stopwords:
        stops = set(stopwords.words("english"))
        words = [w for w in words if not w in stops]
        
    if stemming==True:

        stemmer = SnowballStemmer('english') 
        words = [stemmer.stem(w) for w in words]
        
    if split_text==True:
        return (words)
    
    return( " ".join(words))
import re
import nltk
from nltk.corpus import stopwords 
from nltk.stem.porter import PorterStemmer
from nltk.stem import SnowballStemmer, WordNetLemmatizer
from nltk import sent_tokenize, word_tokenize, pos_tag
from bs4 import BeautifulSoup 
import logging
from wordcloud import WordCloud
from gensim.models import word2vec
from gensim.models import Word2Vec
from gensim.models.keyedvectors import KeyedVectors

X_train_cleaned = []
X_test_cleaned = []

for d in X_train:
    X_train_cleaned.append(cleanText(d))
print('Show a cleaned review in the training set : \n',  X_train_cleaned[10])
    
for d in X_test:
    X_test_cleaned.append(cleanText(d))<jupyter_output>Show a cleaned review in the training set : 
 panting through a glaze of clean sweat mike and lorraine smile and look into each other s eyes the smile slowly disappears will they kiss they re close really close lips almost touching mike tries to muster up the courage but it s been so long he can t do it he lets her up the floor clears exhausted dancers push past them forget it the moment s gone what the hell they had a great time what s the hurry soft cut to ext la brea avenue outside the derby night mike is walking lorraine to her car they come upon a parked escort lorraine well this is it mike listen i had a great time lorraine me too mike i would love to see you again sometime lorraine i ll be around mike that s not good enough i want to make plans to see you lorraine let me get a pen out of my car opens the door do you have something to write on mike hands her a business card lorraine looking at it you re a comedian mike yeah and an actor lorraine i ll have to come see you sometime[...]<jupyter_text>CountVectorizer with Mulinomial Naive Bayes (Benchmark Model)<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.naive_bayes import BernoulliNB, MultinomialNB
countVect = CountVectorizer() 
X_train_countVect = countVect.fit_transform(X_train_cleaned)
print("Number of features : %d \n" %len(countVect.get_feature_names())) #6378 
print("Show some feature names : \n", countVect.get_feature_names()[::1000])
X_train_countVect.shape,y_train.shape
# from imblearn.over_sampling import SMOTE
# smote = SMOTE()
# x_train_smote, y_train_smote = smote.fit_sample(X_train_countVect,y_train)
# # from collections import Counter
# # print("Before smote:",Counter(X_train_countVect))
# # print("After smote:",Counter(x_train_smote))
# Train MultinomialNB classifier
mnb = MultinomialNB()
mnb.fit(X_train_countVect, y_train)
from sklearn import metrics
from sklearn.metrics import accuracy_score,roc_auc_score
def modelEvaluation(predictions):
    '''
    Print model evaluation to predicted result 
    '''
    print ("\nAccuracy on validation set: {:.4f}".format(accuracy_score(y_test, predictions)))
    #print("\nAUC score : {:.4f}".format(roc_auc_score(y_test, predictions)))
    print("\nClassification report : \n", metrics.classification_report(y_test, predictions))
    print("\nConfusion Matrix : \n", metrics.confusion_matrix(y_test, predictions))
predictions = mnb.predict(countVect.transform(X_test_cleaned))
modelEvaluation(predictions)
import pickle
pickle.dump(mnb,open('Naive_Bayes_model.pkl','wb'))
import pickle
pickle.dump(countVect,open('countVect_Genre.pkl','wb'))<jupyter_output><empty_output><jupyter_text>TfidfVectorizer with Logistic Regression<jupyter_code>from sklearn.naive_bayes import BernoulliNB, MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
tfidf = TfidfVectorizer(min_df=5) #minimum document frequency of 5
X_train_tfidf = tfidf.fit_transform(X_train)
print("Number of features : %d \n" %len(tfidf.get_feature_names())) #1722
print("Show some feature names : \n", tfidf.get_feature_names()[::1000])
lr = LogisticRegression()
lr.fit(X_train_tfidf, y_train)
predictions = lr.predict(tfidf.transform(X_test_cleaned))
modelEvaluation(predictions)<jupyter_output>Accuracy on validation set: 0.6858

Classification report : 
               precision    recall  f1-score   support

           1       1.00      0.48      0.65        23
           2       0.86      0.23      0.36        26
           3       0.68      0.69      0.68        80
           4       0.65      0.86      0.74        97

    accuracy                           0.69       226
   macro avg       0.80      0.56      0.61       226
weighted avg       0.72      0.69      0.67       226


Confusion Matrix : 
 [[11  0  7  5]
 [ 0  6  6 14]
 [ 0  0 55 25]
 [ 0  1 13 83]]
{'fraction_non_alphanumeric': 0.08357059119075531, 'fraction_numerical': 0.023614134985764527, 'mean_word_length': 3.9314616019818334, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '275766', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1970, 'n_tokens_neox': 1840, 'n_words': 619, 'path': 'Movie Genre Prediction.ipynb', 'repo_name': 'DheerajKumar97/Movie-Genre-Prediction-Flask--Deployment', 'size': '5971'}