<jupyter_start><jupyter_text>Assignment:-  Apply SVM to Amazon Reviews data set  Amazon Fine Food Reviews AnalysisData Source: https://www.kaggle.com/snap/amazon-fine-food-reviewsThe Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.Number of reviews: 568,454Number of users: 256,059Number of products: 74,258Timespan: Oct 1999 - Oct 2012Number of Attributes/Columns in data: 10 Attribute Information:1. Id2. ProductId - unique identifier for the product3. UserId - unqiue identifier for the user4. ProfileName5. HelpfulnessNumerator - number of users who found the review helpful6. HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not7. Score - rating between 1 and 58. Time - timestamp for the review9. Summary - brief summary of the review10. Text - text of the review  1. Objective:Given a review, determine whether the review is positive (Rating of 4 or 5) or negative (rating of 1 or 2). Use BoW, TF-IDF, Avg-Word2Vec,TF-IDF-Word2Vec to vectorise the reviews. Apply RBFSVM Algorithm for Amazon fine food Reviewsfind right C, gamma(ɣ)  where ɣ=1/c  by Gridsearch cv,Random search cv<jupyter_code># loading required libraries 
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd 
import matplotlib 
import sqlite3
import string
import gensim
import scipy 
import nltk
import time
import seaborn as sns 
from scipy import stats
from matplotlib import pyplot as plt 

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer

from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.linear_model import LogisticRegression

from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_curve, roc_auc_score, auc
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_recall_fscore_support as prf1

from sklearn.model_selection import KFold
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>1.1 Connecting SQL file<jupyter_code>#Loading the data
con = sqlite3.connect('./final.sqlite')

data = pd.read_sql_query("""
SELECT *
FROM Reviews
""", con)
print(data.shape)
data.head()<jupyter_output>(364171, 12)<jupyter_text>1.2 Data Preprocessing<jupyter_code>data.Score.value_counts()
#i had done data preprocessing i had stored in final.sqlite now loaded this file no need to do again data preprocessing<jupyter_output><empty_output><jupyter_text>1.3 Sorting the data<jupyter_code># Sorting the data according to the time-stamp
sorted_data = data.sort_values('Time', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
sorted_data.head()<jupyter_output><empty_output><jupyter_text>1.4 Mapping<jupyter_code>def partition(x):
    if x == 'positive':
        return 1
    return 0

#Preparing the filtered data
actualScore = sorted_data['Score']
positiveNegative = actualScore.map(partition) 
sorted_data['Score'] = positiveNegative
sorted_data.head()<jupyter_output><empty_output><jupyter_text>1.5 Taking first 140k rows<jupyter_code># We will collect different 140000 rows without repetition from time_sorted_data dataframe
my_final = sorted_data[:140000]
print(my_final.shape)
my_final.head()<jupyter_output>(140000, 12)<jupyter_text>1.6 Spliting data into train and test based on time (70:30)<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_validate


x=my_final['CleanedText'].values
y=my_final['Score']

#Splitting data into train test and cross validation 
x_train,x_test,y_train,y_test =train_test_split(x,y,test_size =0.3,random_state = 42)

print(x_train.shape)
print(x_test.shape)
print(y_train.shape)
print(y_test.shape)<jupyter_output>(98000,)
(42000,)
(98000,)
(42000,)<jupyter_text>2.  Techniques For Vectorization  Why we have to convert text to vectorBy converting text to vector we can use whole power of linear algebra.we can find a plane to seperate  2.1 BOW<jupyter_code>#Bow

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer() 
final_counts_Bow_tr= count_vect.fit_transform(x_train)# computing Bow
print("the type of count vectorizer ",type(final_counts_Bow_tr))
print("the shape of out text BOW vectorizer ",final_counts_Bow_tr.get_shape())
print("the number of unique words ", final_counts_Bow_tr.get_shape()[1])
final_counts_Bow_test= count_vect.transform(x_test)# computing Bow
print("the type of count vectorizer ",type(final_counts_Bow_test))
print("the shape of out text BOW vectorizer ",final_counts_Bow_test.get_shape())<jupyter_output>the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>
the shape of out text BOW vectorizer  (98000, 36949)
the number of unique words  36949
the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>
the shape of out text BOW vectorizer  (42000, 36949)<jupyter_text>2.1.1 Normalizing Data<jupyter_code># Data-preprocessing: Normalizing Data

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(final_counts_Bow_tr)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(final_counts_Bow_test)
print(standardized_data_test.shape)<jupyter_output>(98000, 36949)
(42000, 36949)<jupyter_text>2.2 Applying Support vector Machine Algorithm  2.2.1 Applying SGD Classifier by GridSearchCV<jupyter_code>from sklearn import linear_model
clf=linear_model.SGDClassifier(loss='hinge',max_iter=1000,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = GridSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')
model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>2.3 Confusion Matrix<jupyter_code>cm_bow=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_bow, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_bow.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 0 
 false positives are 5613 
 false negatives are 0
 true positives are 36387<jupyter_text>2.4 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score

# evaluating accuracy
acc_bow = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha = %.3f is %f%%' % (optimal_alpha, acc_bow))

# Error on test data
test_error_bow = 100-acc_bow
print("\nTest Error SVM for alpha  is  %f%%" % (test_error_bow))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM for alpha  = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM for alpha  = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM for alpha  \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha = 10.000 is 86.635714%

Test Error SVM for alpha  is  13.364286%

The Test Precision of the  SVM for alpha  = 10.000 is 0.866357

The Test Recall of the  SVM for alpha  = 10.000 is 1.000000

The Test classification report of the  SVM for alpha  

                precision    recall  f1-score   support

           0       0.00      0.00      0.00      5613
           1       0.87      1.00      0.93     36387

   micro avg       0.87      0.87      0.87     42000
   macro avg       0.43      0.50      0.46     42000
weighted avg       0.75      0.87      0.80     42000<jupyter_text>2.5 Plotting roc_auc curve<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="bow, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>2.6 Top 25 words<jupyter_code>top_n=25
# sorting the weight indices 
pos_class_weight_sorted = -clf.coef_.argsort()
neg_class_weight_sorted = clf.coef_.argsort()
            
# top positive features 
print('\nTop '+str(top_n)+' "POSITIVE" Features are:\n')
pos = np.take(count_vect.get_feature_names(), pos_class_weight_sorted[:top_n])
for i in range(top_n):
    print(pos[0][i])

# top negative features
print('\n Top '+str(top_n)+' "NEGATIVE" Features are:\n')
neg = np.take(count_vect.get_feature_names(), neg_class_weight_sorted[:top_n])
for i in range(top_n):
    print(neg[0][i])
pos=''.join(str(e) for e in pos)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(pos)

plt.title("Positive Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
neg=''.join(str(e) for e in neg)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(neg)

plt.title("Negative Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>2.7 Applying SGD Classifier by RandomizedSearchCV<jupyter_code>from sklearn import linear_model
from sklearn.model_selection import RandomizedSearchCV
clf=linear_model.SGDClassifier(loss='hinge',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = RandomizedSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')
model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>2.8 Confusion Matrix<jupyter_code>cm_bow=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_bow, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_bow.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 1109 
 false positives are 4504 
 false negatives are 570
 true positives are 35817<jupyter_text>2.9 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score

# evaluating accuracy
acc_bow = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha = %.3f is %f%%' % (optimal_alpha, acc_bow))

# Error on test data
test_error_bow = 100-acc_bow
print("\nTest Error SVM for alpha  is  %f%%" % (test_error_bow))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM for alpha  = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM for alpha  = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM for alpha  \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha = 0.100 is 87.919048%

Test Error SVM for alpha  is  12.080952%

The Test Precision of the  SVM for alpha  = 0.100 is 0.888296

The Test Recall of the  SVM for alpha  = 0.100 is 0.984335

The Test classification report of the  SVM for alpha  

                precision    recall  f1-score   support

           0       0.66      0.20      0.30      5613
           1       0.89      0.98      0.93     36387

   micro avg       0.88      0.88      0.88     42000
   macro avg       0.77      0.59      0.62     42000
weighted avg       0.86      0.88      0.85     42000<jupyter_text>2.10 Plotting roc_auc curve<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="BOW, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>2.11 Top 25 words<jupyter_code>top_n=25
# sorting the weight indices 
pos_class_weight_sorted = -clf.coef_.argsort()
neg_class_weight_sorted = clf.coef_.argsort()
            
# top positive features 
print('\nTop '+str(top_n)+' "POSITIVE" Features are:\n')
pos = np.take(count_vect.get_feature_names(), pos_class_weight_sorted[:top_n])
for i in range(top_n):
    print(pos[0][i])

# top negative features
print('\nTop '+str(top_n)+' "NEGATIVE" Features are:\n')
neg = np.take(count_vect.get_feature_names(), neg_class_weight_sorted[:top_n])
for i in range(top_n):
    print(neg[0][i])
pos=''.join(str(e) for e in pos)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(pos)

plt.title("Positive Frequent words")
plt.figure(1, figsize=(25, 25))
plt.tight_layout(pad=0)
plt.imshow(wordcloud)
plt.axis('off')
plt.show()
neg=''.join(str(e) for e in neg)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(neg)

plt.title("Negative Frequent words")
plt.figure(1, figsize=(25, 25))
plt.tight_layout(pad=0)
plt.imshow(wordcloud)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>3. Applying RBF Kernel for Grid Search<jupyter_code># PCA for dimensionality redcution (non-visualization)
# initializing the pca
from scipy.sparse import csr_matrix
from sklearn import decomposition
pca = decomposition.PCA()

standardized_data_train = csr_matrix(standardized_data_train)
standardized_data_train=standardized_data_train.todense()
print(standardized_data_train.shape)

pca.n_components = 12
pca_data = pca.fit_transform(standardized_data_train)

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);

cum_var_explained = np.cumsum(percentage_var_explained)

# Plot the PCA spectrum
plt.figure(1, figsize=(6, 4))

plt.clf()
plt.plot(cum_var_explained, linewidth=2)
plt.axis('tight')
plt.grid()
plt.xlabel('n_components')
plt.ylabel('Cumulative_explained_variance')
plt.show()


# If we take 9-dimensions, approx. 90% of variance is expalined.
from sklearn.decomposition import TruncatedSVD
final_counts_Bow_tr_svd = TruncatedSVD(n_components=9).fit_transform(final_counts_Bow_tr)
final_counts_Bow_tr_svd1 = TruncatedSVD(n_components=9).fit_transform(final_counts_Bow_test)<jupyter_output><empty_output><jupyter_text>3.1 Normalizing data<jupyter_code># Data-preprocessing: Normalizing the data

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
standardized_data_train = scaler.fit_transform(final_counts_Bow_tr_svd)
print(standardized_data_train.shape)
standardized_data_test = scaler.fit_transform(final_counts_Bow_tr_svd1)
print(standardized_data_test.shape)
#in RBF SVC hyper parameters are C,gamma

from sklearn.svm import SVC
clf=SVC(kernel='rbf',max_iter=1000,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = GridSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')
model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>3.2 Confusion Matrix<jupyter_code>cm_bow=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_bow, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_bow.ravel()
( tp, fp, fn, tp)
print(" true negatives are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negatives are 10 
 false positives are 5603 
 false negatives are 139
 true positives are 36248<jupyter_text>3.3 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score

# evaluating accuracy
acc_bow = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_bow))

# Error on test data
test_error_bow = 100-acc_bow
print("\nTest Error SVM using RBF for alpha  is  %f%%" % (test_error_bow))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM using RBF kernel  for C and gamma  \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM using RBF kernel  for C and gamma = 100.000 ,100.000 is 86.328571%

Test Error SVM using RBF for alpha  is  13.671429%

The Test Precision of the  SVM using RBF kernel  for C and gamma = 100.000 ,100.000 is 0.866120

The Test Recall of the  SVM using RBF kernel  for C and gamma = 100.000 ,100.000 is 0.996180

The Test classification report of the  SVM using RBF kernel  for C and gamma  

                precision    recall  f1-score   support

           0       0.07      0.00      0.00      5613
           1       0.87      1.00      0.93     36387

   micro avg       0.86      0.86      0.86     42000
   macro avg       0.47      0.50      0.47     42000
weighted avg       0.76      0.86      0.80     42000<jupyter_text>3.5 Applying RBF Kernel for Random Search<jupyter_code>#in RBF SVC hyper parameters are C,gamma
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = RandomizedSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')
model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>3.6 Confusion Matrix<jupyter_code>cm_bow=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_bow, annot=True, fmt='d')
plt.show()

#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_bow.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>Confusion Matrix:<jupyter_text>3.7 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score

# evaluating accuracy
acc_bow = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_bow))

# Error on test data
test_error_bow = 100-acc_bow
print("\nTest Error SVM using RBF  for C and gamma is  %f%%" % (test_error_bow))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM using RBF kernel  for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM using RBF kernel  for C and gamma \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM using RBF kernel  for C and gamma = 100.000 ,0.010 is 81.661905%

Test Error SVM using RBF  for C and gamma is  18.338095%

The Test Precision of the  SVM using RBF kernel  for C and gamma = 100.000 ,0.010 is 0.868181

The Test Recall of the  SVM using RBF kernel  for C and gamma = 100.000 ,0.010 is 0.929453

The Test classification report of the  SVM using RBF kernel  for C and gamma 

                precision    recall  f1-score   support

           0       0.16      0.09      0.11      5613
           1       0.87      0.93      0.90     36387

   micro avg       0.82      0.82      0.82     42000
   macro avg       0.51      0.51      0.50     42000
weighted avg       0.77      0.82      0.79     42000<jupyter_text>4. TF-IDF<jupyter_code>#tf-idf
from sklearn.feature_extraction.text import TfidfVectorizer
tf_idf_vect = TfidfVectorizer()

final_counts_tfidf_tr= tf_idf_vect.fit_transform(x_train)
print("the type of count vectorizer ",type(final_counts_tfidf_tr))
print("the shape of out text tfidf vectorizer ",final_counts_tfidf_tr.get_shape())
print("the number of unique words ", final_counts_tfidf_tr.get_shape()[1])
final_counts_tfidf_test= tf_idf_vect.transform(x_test)
print("the type of count vectorizer ",type(final_counts_tfidf_test))
print("the shape of out text tfidf vectorizer ",final_counts_tfidf_test.get_shape())
print("the number of unique words ", final_counts_tfidf_test.get_shape()[1])<jupyter_output>the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>
the shape of out text tfidf vectorizer  (98000, 36949)
the number of unique words  36949
the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>
the shape of out text tfidf vectorizer  (42000, 36949)
the number of unique words  36949<jupyter_text>4.1 Normalizing Data<jupyter_code># Data-preprocessing: Normalizing Data
from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(final_counts_tfidf_tr)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(final_counts_tfidf_test)
print(standardized_data_test.shape)<jupyter_output>(98000, 36949)
(42000, 36949)<jupyter_text>4.2 Applying SGD Classifier by GridSearchCV<jupyter_code>from sklearn import linear_model
clf=linear_model.SGDClassifier(loss='hinge',penalty='l2',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = GridSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')

model.best_params_
results = model.cv_results_
results['mean_test_score']

C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>4.3 Confusion Matrix<jupyter_code>cm_tfidf=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidf, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidf.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 5613 
 false positives are 0 
 false negatives are 36387
 true positives are 0<jupyter_text>4.4 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidf = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha = %.3f is %f%%' % (optimal_alpha, acc_tfidf))

# Error on test data
test_error_tfidf = 100-acc_tfidf
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_tfidf))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report oof the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha = 10.000 is 13.364286%

Test Error of the SVM for alpha is  86.635714%

The Test Precision of the SVM for alpha is = 10.000 is 0.000000

The Test Recall of the SVM for alpha is = 10.000 is 0.000000

The Test classification report oof the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.13      1.00      0.24      5613
           1       0.00      0.00      0.00     36387

   micro avg       0.13      0.13      0.13     42000
   macro avg       0.07      0.50      0.12     42000
weighted avg       0.02      0.13      0.03     42000<jupyter_text>4.5 Plotting roc_auc curve<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="tfidf, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>4.6 Top 25 words<jupyter_code>top_n=25
# sorting the weight indices 
pos_class_weight_sorted = -clf.coef_.argsort()
neg_class_weight_sorted = clf.coef_.argsort()
            
# top positive features 
print('\nTop '+str(top_n)+' "POSITIVE" Features are:\n')
pos = np.take(tf_idf_vect.get_feature_names(), pos_class_weight_sorted[:top_n])
for i in range(top_n):
    print(pos[0][i])

# top negative features
print('Top '+str(top_n)+' "NEGATIVE" Features are:\n')
neg = np.take(tf_idf_vect.get_feature_names(), neg_class_weight_sorted[:top_n])
for i in range(top_n):
    print(neg[0][i])
pos=''.join(str(e) for e in pos)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(pos)

plt.title("Positive Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
plt.show()
neg=''.join(str(e) for e in neg)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(neg)

plt.title("Negative Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>4.7 Applying SGD Classifier by RandomizedSearchCV<jupyter_code>from sklearn import linear_model
from sklearn.model_selection import RandomizedSearchCV
clf=linear_model.SGDClassifier(loss='hinge',penalty='l2',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = RandomizedSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')
model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>4.8 Confusion Matrix<jupyter_code>cm_tfidf=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidf, annot=True, fmt='d')
plt.show()

#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidf.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>Confusion Matrix:<jupyter_text>4.9 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidf = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha = %.3f is %f%%' % (optimal_alpha, acc_tfidf))

# Error on test data
test_error_tfidf = 100-acc_tfidf
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_tfidf))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report oof the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha = 0.010 is 76.680952%

Test Error of the SVM for alpha is  23.319048%

The Test Precision of the SVM for alpha is = 0.010 is 0.975265

The Test Recall of the SVM for alpha is = 0.010 is 0.749856

The Test classification report oof the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.35      0.88      0.50      5613
           1       0.98      0.75      0.85     36387

   micro avg       0.77      0.77      0.77     42000
   macro avg       0.66      0.81      0.67     42000
weighted avg       0.89      0.77      0.80     42000<jupyter_text>4.10 Plotting Roc_auc score<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="tfidf, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>4.11 Calculating Top25 features<jupyter_code>top_n=25
# sorting the weight indices 
pos_class_weight_sorted = -clf.coef_.argsort()
neg_class_weight_sorted = clf.coef_.argsort()
            
# top positive features 
print('\nTop '+str(top_n)+' "POSITIVE" Features are:\n')
pos = np.take(tf_idf_vect.get_feature_names(), pos_class_weight_sorted[:top_n])
for i in range(top_n):
    print(pos[0][i])

# top negative features
print('Top '+str(top_n)+' "NEGATIVE" Features are:\n')
neg = np.take(tf_idf_vect.get_feature_names(), neg_class_weight_sorted[:top_n])
for i in range(top_n):
    print(neg[0][i])
pos=''.join(str(e) for e in pos)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(pos)

plt.title("Positive Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
plt.show()
neg=''.join(str(e) for e in neg)

from wordcloud import WordCloud

wordcloud = WordCloud(background_color='black',
                      width=3000,
                      height=2500
                     ).generate(neg)

plt.title("Negative Frequent words")
plt.figure(1, figsize=(25, 25))
plt.imshow(wordcloud)
plt.tight_layout(pad=0)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>5. Applying RBF Kernel for GridSearch<jupyter_code># PCA for dimensionality redcution (non-visualization)
# initializing the pca
from scipy.sparse import csr_matrix
from sklearn import decomposition
pca = decomposition.PCA()

standardized_data_train = csr_matrix(standardized_data_train)
standardized_data_train=standardized_data_train.todense()
print(standardized_data_train.shape)

pca.n_components = 12
pca_data = pca.fit_transform(standardized_data_train)

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);

cum_var_explained = np.cumsum(percentage_var_explained)

# Plot the PCA spectrum
plt.figure(1, figsize=(6, 4))

plt.clf()
plt.plot(cum_var_explained, linewidth=2)
plt.axis('tight')
plt.grid()
plt.xlabel('n_components')
plt.ylabel('Cumulative_explained_variance')
plt.show()


# If we take 9-dimensions, approx. 90% of variance is expalined.
from sklearn.decomposition import TruncatedSVD
final_counts_tfidf_tr_svd = TruncatedSVD(n_components=9).fit_transform(final_counts_tfidf_tr)
final_counts_tfidf_tr_svd1 = TruncatedSVD(n_components=9).fit_transform(final_counts_tfidf_test)<jupyter_output><empty_output><jupyter_text>5.1 Normalizing data<jupyter_code># Data-preprocessing: Normalizing data 

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(final_counts_tfidf_tr_svd)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(final_counts_tfidf_tr_svd1)
print(standardized_data_test.shape)

#In RBF SVC hyper parameters are C,gamma
from sklearn.svm import SVC
clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = GridSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')

model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>5.2 Confusion Matrix<jupyter_code>cm_tfidf=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidf, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidf.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 0 
 false positives are 5613 
 false negatives are 6
 true positives are 36381<jupyter_text>5.3 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidf = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_tfidf))

# Error on test data
test_error_tfidf = 100-acc_tfidf
print("\nTest Error of the SVM for alpha  %f%%" % (test_error_tfidf))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for C and gamma \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for C and gamma = 10.000 ,1000.000 is 86.621429%

Test Error of the SVM for alpha  13.378571%

The Test Precision of the SVM for C and gamma = 10.000 ,1000.000 is 0.866338

The Test Recall of the SVM for C and gamma = 10.000 ,1000.000 is 0.999835

The Test classification report of the SVM for C and gamma 

                precision    recall  f1-score   support

           0       0.00      0.00      0.00      5613
           1       0.87      1.00      0.93     36387

   micro avg       0.87      0.87      0.87     42000
   macro avg       0.43      0.50      0.46     42000
weighted avg       0.75      0.87      0.80     42000<jupyter_text>5.5 Applying RBF Kernel for RandomizedSearchCV<jupyter_code>#In RBF SVC hyper parameters are C,gamma
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
clf=SVC(kernel='rbf',probability=True,max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = RandomizedSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')
model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>5.6 Confusion Matrix<jupyter_code>cm_tfidf=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidf, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidf.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 0 
 false positives are 5613 
 false negatives are 6
 true positives are 36381<jupyter_text>5.7 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidf = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_tfidf))

# Error on test data
test_error_tfidf = 100-acc_tfidf
print("\nTest Error of the SVM for C and gamma %f%%" % (test_error_tfidf))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for alpha \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for C and gamma = 10.000 ,1000.000 is 86.621429%

Test Error of the SVM for C and gamma 13.378571%

The Test Precision of the SVM for C and gamma = 10.000 ,1000.000 is 0.866338

The Test Recall of the SVM for C and gamma = 10.000 ,1000.000 is 0.999835

The Test classification report of the SVM for alpha 

                precision    recall  f1-score   support

           0       0.00      0.00      0.00      5613
           1       0.87      1.00      0.93     36387

   micro avg       0.87      0.87      0.87     42000
   macro avg       0.43      0.50      0.46     42000
weighted avg       0.75      0.87      0.80     42000<jupyter_text>6. WORD2VEC<jupyter_code>from gensim.models import Word2Vec
# List of sentence in X_train text
sent_of_train=[]
for sent in x_train:
    sent_of_train.append(sent.split())

# List of sentence in X_est text
sent_of_test=[]
for sent in x_test:
    sent_of_test.append(sent.split())   
    
# Train your own Word2Vec model using your own train text corpus 
# min_count = 5 considers only words that occured atleast 5 times
w2v_model=Word2Vec(sent_of_train,min_count=5,size=50, workers=4)

w2v_words = list(w2v_model.wv.vocab)
print("number of words that occured minimum 5 times ",len(w2v_words))<jupyter_output>number of words that occured minimum 5 times  12448<jupyter_text>7. Avg Word2Vec<jupyter_code># compute average word2vec for each review for X_train .
train_vectors = []; 
for sent in sent_of_train:
    sent_vec = np.zeros(50) 
    cnt_words =0; 
    for word in sent: # 
        if word in w2v_words:
            vec = w2v_model.wv[word]
            sent_vec += vec
            cnt_words += 1
    if cnt_words != 0:
        sent_vec /= cnt_words
    train_vectors.append(sent_vec)
    
# compute average word2vec for each review for X_test .
test_vectors = []; 
for sent in sent_of_test:
    sent_vec = np.zeros(50) 
    cnt_words =0; 
    for word in sent: # 
        if word in w2v_words:
            vec = w2v_model.wv[word]
            sent_vec += vec
            cnt_words += 1
    if cnt_words != 0:
        sent_vec /= cnt_words
    test_vectors.append(sent_vec)<jupyter_output><empty_output><jupyter_text>7.1 Normalizing data<jupyter_code># Data-preprocessing: Normalizing the data

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(train_vectors)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(test_vectors)
print(standardized_data_test.shape)<jupyter_output>(98000, 50)
(42000, 50)<jupyter_text>7.2 Applying SGD Classifier by GridSearchCV<jupyter_code>from sklearn import linear_model
clf=linear_model.SGDClassifier(loss='hinge',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = GridSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
optimal_alpha = model.best_estimator_.alpha
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))

a = model.best_params_
optimal_alpha = a.get('alpha')


model.best_params_
results = model.cv_results_
results['mean_test_score']

C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>7.3 Confusion Matrix<jupyter_code>cm_avgw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_avgw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_avgw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 2822 
 false positives are 2791 
 false negatives are 2838
 true positives are 33549<jupyter_text>7.4 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_avgw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha is = %.3f is %f%%' % (optimal_alpha, acc_avgw2v))

# Error on test data
test_error_avgw2v = 100-acc_avgw2v
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_avgw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha is = 0.100 is 86.597619%

Test Error of the SVM for alpha is  13.402381%

The Test Precision of the SVM for alpha is = 0.100 is 0.923198

The Test Recall of the SVM for alpha is = 0.100 is 0.922005

The Test classification report of the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.50      0.50      0.50      5613
           1       0.92      0.92      0.92     36387

   micro avg       0.87      0.87      0.87     42000
   macro avg       0.71      0.71      0.71     42000
weighted avg       0.87      0.87      0.87     42000<jupyter_text>7.5 Evaluating roc_auc_score<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="Avgword2vec, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>7.6 Applying SGD Classifier by RandomizedSearchCV<jupyter_code>from sklearn import linear_model
from sklearn.model_selection import RandomizedSearchCV
clf=linear_model.SGDClassifier(loss='hinge',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = RandomizedSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
optimal_alpha = model.best_estimator_.alpha
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))

a = model.best_params_
optimal_alpha = a.get('alpha')
model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>7.7 Confusion Matrix<jupyter_code>cm_avgw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_avgw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_avgw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 4666 
 false positives are 947 
 false negatives are 7120
 true positives are 29267<jupyter_text>7.8 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_avgw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha is = %.3f is %f%%' % (optimal_alpha, acc_avgw2v))

# Error on test data
test_error_avgw2v = 100-acc_avgw2v
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_avgw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha is = 0.001 is 80.792857%

Test Error of the SVM for alpha is  19.207143%

The Test Precision of the SVM for alpha is = 0.001 is 0.968657

The Test Recall of the SVM for alpha is = 0.001 is 0.804326

The Test classification report of the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.40      0.83      0.54      5613
           1       0.97      0.80      0.88     36387

   micro avg       0.81      0.81      0.81     42000
   macro avg       0.68      0.82      0.71     42000
weighted avg       0.89      0.81      0.83     42000<jupyter_text>7.9 Plotting ROC CURVE<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="Avgw2vec, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>8. APPLYING RBF KERNEL BY GRIDSEARCH<jupyter_code># PCA for dimensionality redcution (non-visualization)
# initializing the pca
from scipy.sparse import csr_matrix
from sklearn import decomposition
pca = decomposition.PCA()

standardized_data_train = csr_matrix(standardized_data_train)
standardized_data_train=standardized_data_train.todense()
print(standardized_data_train.shape)

pca.n_components = 12
pca_data = pca.fit_transform(standardized_data_train)

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);

cum_var_explained = np.cumsum(percentage_var_explained)

# Plot the PCA spectrum
plt.figure(1, figsize=(6, 4))

plt.clf()
plt.plot(cum_var_explained, linewidth=2)
plt.axis('tight')
plt.grid()
plt.xlabel('n_components')
plt.ylabel('Cumulative_explained_variance')
plt.show()


# If we take 9-dimensions, approx. 90% of variance is expalined.
from sklearn.decomposition import TruncatedSVD
final_counts_avg_w2v_svd = TruncatedSVD(n_components=9).fit_transform(train_vectors)
final_counts_avg_w2v_svd1 = TruncatedSVD(n_components=9).fit_transform(test_vectors)<jupyter_output><empty_output><jupyter_text>8.2 Normalizing data<jupyter_code># Data-preprocessing:Normalizing data

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(final_counts_avg_w2v_svd)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(final_counts_avg_w2v_svd1)
print(standardized_data_test.shape)
#In RBF SVC hyper parameters are C,gamma
from sklearn.svm import SVC
clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = GridSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')

model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>8.3 Confusion Matrix<jupyter_code>cm_avgw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_avgw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_avgw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 340 
 false positives are 5273 
 false negatives are 1045
 true positives are 35342<jupyter_text>8.4 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_avgw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_avgw2v))

# Error on test data
test_error_avgw2v = 100-acc_avgw2v
print("\nTest Error of the SVM for C and gamma is  %f%%" % (test_error_avgw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for C and gamma  is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for C and gamma = 10.000 ,10000.000 is 84.957143%

Test Error of the SVM for C and gamma is  15.042857%

The Test Precision of the SVM for C and gamma = 10.000 ,10000.000 is 0.870171

The Test Recall of the SVM for C and gamma = 10.000 ,10000.000 is 0.971281

The Test classification report of the SVM for C and gamma  is 

                precision    recall  f1-score   support

           0       0.25      0.06      0.10      5613
           1       0.87      0.97      0.92     36387

   micro avg       0.85      0.85      0.85     42000
   macro avg       0.56      0.52      0.51     42000
weighted avg       0.79      0.85      0.81     42000<jupyter_text>8.5 APPLYING RBF KERNEL BY RandomizedSearchCV<jupyter_code>#in RBF SVC hyper parameters are C,gamma
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV

clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = RandomizedSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')
model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>8.6 Confusion Matrix<jupyter_code>cm_avgw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_avgw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_avgw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 5462 
 false positives are 151 
 false negatives are 34825
 true positives are 1562<jupyter_text>8.7 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_avgw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_avgw2v))

# Error on test data
test_error_avgw2v = 100-acc_avgw2v
print("\nTest Error of the SVM for C and gamma is  %f%%" % (test_error_avgw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for C and gamma \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for C and gamma = 10.000 ,10.000 is 16.723810%

Test Error of the SVM for C and gamma is  83.276190%

The Test Precision of the SVM for C and gamma = 10.000 ,10.000 is 0.911851

The Test Recall of the SVM for C and gamma = 10.000 ,10.000 is 0.042927

The Test classification report of the SVM for C and gamma 

                precision    recall  f1-score   support

           0       0.14      0.97      0.24      5613
           1       0.91      0.04      0.08     36387

   micro avg       0.17      0.17      0.17     42000
   macro avg       0.52      0.51      0.16     42000
weighted avg       0.81      0.17      0.10     42000<jupyter_text>9. TFIDF-Word2Vec<jupyter_code>#tf-idf weighted w2v

from sklearn.feature_extraction.text import TfidfVectorizer

tfidfw2v_vect = TfidfVectorizer()
final_counts_tfidfw2v_train= tfidfw2v_vect.fit_transform(x_train) 
print(type(final_counts_tfidfw2v_train))
print(final_counts_tfidfw2v_train.shape)

final_counts_tfidfw2v_test= tfidfw2v_vect.transform(x_test) 
print(type(final_counts_tfidfw2v_test))
print(final_counts_tfidfw2v_test.shape)
# we are converting a dictionary with word as a key, and the idf as a value
dictionary = dict(zip(tfidfw2v_vect.get_feature_names(), list(tfidfw2v_vect.idf_)))


# TF-IDF weighted Word2Vec
tfidf_feat = tfidfw2v_vect.get_feature_names() # tfidf words/col-names
# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf

tfidf_sent_vectors = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in sent_of_train: # for each review/sentence 
    sent_vec = np.zeros(50) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        if word in w2v_words:
            vec = w2v_model.wv[word]
#             tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]
            # to reduce the computation we are 
            # dictionary[word] = idf value of word in whole courpus
            # sent.count(word) = tf valeus of word in this review
            tf_idf = dictionary[word]*(sent.count(word)/len(sent))
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
    if weight_sum != 0:
        sent_vec /= weight_sum
    tfidf_sent_vectors.append(sent_vec)
    row += 1

#Test case

tfidf_sent_vectors1 = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in sent_of_test: # for each review/sentence 
    sent_vec = np.zeros(50) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        if word in w2v_words:
            vec = w2v_model.wv[word]
#             tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]
            # to reduce the computation we are 
            # dictionary[word] = idf value of word in whole courpus
            # sent.count(word) = tf valeus of word in this review
            tf_idf = dictionary[word]*(sent.count(word)/len(sent))
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
    if weight_sum != 0:
        sent_vec /= weight_sum
    tfidf_sent_vectors1.append(sent_vec)
    row += 1
print(len(tfidf_sent_vectors))
print(len(tfidf_sent_vectors1))<jupyter_output>98000
42000<jupyter_text>9.1 Normalizing the data<jupyter_code># Data-preprocessing: Normalizing the data

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(tfidf_sent_vectors)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(tfidf_sent_vectors1)
print(standardized_data_test.shape)<jupyter_output>(98000, 50)
(42000, 50)<jupyter_text>9.2 Applying SGD Classifier Using GridSearch<jupyter_code>from sklearn import linear_model
clf=linear_model.SGDClassifier(loss='hinge',penalty='l2',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = GridSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
optimal_alpha = model.best_estimator_.alpha
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')

model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>9.3 Confusion Matrix<jupyter_code>cm_tfidfw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidfw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidfw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 0 
 false positives are 5613 
 false negatives are 0
 true positives are 36387<jupyter_text>9.4 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidfw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM for alpha = %.3f is %f%%' % (optimal_alpha, acc_tfidfw2v))

# Error on test data
test_error_tfidfw2v = 100-acc_tfidfw2v
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_tfidfw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM for alpha = 10.000 is 86.635714%

Test Error of the SVM for alpha is  13.364286%

The Test Precision of the SVM for alpha is = 10.000 is 0.866357

The Test Recall of the SVM for alpha is = 10.000 is 1.000000

The Test classification report of the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.00      0.00      0.00      5613
           1       0.87      1.00      0.93     36387

   micro avg       0.87      0.87      0.87     42000
   macro avg       0.43      0.50      0.46     42000
weighted avg       0.75      0.87      0.80     42000<jupyter_text>Evaluating roc_auc_score<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="tfidfw2vec, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>9.5 Applying SGD Classifier Using RandomizedSearchCV<jupyter_code>from sklearn import linear_model
from sklearn.model_selection import RandomizedSearchCV

clf=linear_model.SGDClassifier(loss='hinge',penalty='l2',max_iter=100,class_weight='balanced')
tuned_parameters = {'alpha': [0.001, 0.01, 0.1, 1, 10]}

model = RandomizedSearchCV(clf, tuned_parameters,cv=3,scoring='f1', n_jobs=-1,pre_dispatch=2)
model.fit(standardized_data_train, y_train)
print("best alpha value is",model.best_estimator_)
optimal_alpha = model.best_estimator_.alpha
print("Accuracy of the model : {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_alpha = a.get('alpha')
model.best_params_
results = model.cv_results_
results['mean_test_score']
C=0.001, 0.01, 0.1, 1, 10
plt.plot(C,results['mean_test_score'],marker='o')
plt.xlabel('C')
plt.ylabel('f1score')
plt.title("F1score vs hyperparameter C")
plt.grid()
plt.show()
clf = linear_model.SGDClassifier(loss='hinge',alpha=optimal_alpha,class_weight='balanced')
clf.fit(standardized_data_train,y_train)
y_pred = clf.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>9.6 Confusion Matrix<jupyter_code>cm_tfidfw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidfw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidfw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 5154 
 false positives are 459 
 false negatives are 18905
 true positives are 17482<jupyter_text>9.7 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidfw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the SVM classifier for alpha = %.3f is %f%%' % (optimal_alpha, acc_tfidfw2v))

# Error on test data
test_error_tfidfw2v = 100-acc_tfidfw2v
print("\nTest Error of the SVM for alpha is  %f%%" % (test_error_tfidfw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the SVM for alpha is = %.3f is %f' % (optimal_alpha, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the SVM for alpha is = %.3f is %f' % (optimal_alpha, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the SVM for alpha is \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the SVM classifier for alpha = 0.100 is 53.895238%

Test Error of the SVM for alpha is  46.104762%

The Test Precision of the SVM for alpha is = 0.100 is 0.974416

The Test Recall of the SVM for alpha is = 0.100 is 0.480446

The Test classification report of the SVM for alpha is 

                precision    recall  f1-score   support

           0       0.21      0.92      0.35      5613
           1       0.97      0.48      0.64     36387

   micro avg       0.54      0.54      0.54     42000
   macro avg       0.59      0.70      0.50     42000
weighted avg       0.87      0.54      0.60     42000<jupyter_text>Evaluating Roc_auc_score<jupyter_code>from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=3)
calibrated.fit(standardized_data_train, y_train)
y_pred_proba = calibrated.predict_proba(standardized_data_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)
auc = metrics.roc_auc_score(y_test, y_pred_proba)
plt.plot(fpr,tpr,label="tfidfw2vec, AUC="+str(auc))
plt.plot([0,1],[0,1],'r--')
plt.title('ROC curve: SVM')
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>10. APPLYING RBF KERNEL BY GRID SEARCH  10.1 Gridsearch Cross Validation<jupyter_code># PCA for dimensionality redcution (non-visualization)
# initializing the pca
from scipy.sparse import csr_matrix
from sklearn import decomposition
pca = decomposition.PCA()

standardized_data_train = csr_matrix(standardized_data_train)
standardized_data_train=standardized_data_train.todense()
print(standardized_data_train.shape)

pca.n_components = 12
pca_data = pca.fit_transform(standardized_data_train)

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);

cum_var_explained = np.cumsum(percentage_var_explained)

# Plot the PCA spectrum
plt.figure(1, figsize=(6, 4))

plt.clf()
plt.plot(cum_var_explained, linewidth=2)
plt.axis('tight')
plt.grid()
plt.xlabel('n_components')
plt.ylabel('Cumulative_explained_variance')
plt.show()


# If we take 9-dimensions, approx. 90% of variance is expalined.
from sklearn.decomposition import TruncatedSVD
final_counts_tfidf_w2v_svd = TruncatedSVD(n_components=9).fit_transform(tfidf_sent_vectors)
final_counts_tfidf_w2v_svd1 = TruncatedSVD(n_components=9).fit_transform(tfidf_sent_vectors1)<jupyter_output><empty_output><jupyter_text>10.2 Normalizing the data<jupyter_code># Data-preprocessing: Normalizing the data

from sklearn import preprocessing
standardized_data_train = preprocessing.normalize(final_counts_tfidf_w2v_svd)
print(standardized_data_train.shape)
standardized_data_test = preprocessing.normalize(final_counts_tfidf_w2v_svd1)
print(standardized_data_test.shape)
#in RBF SVC hyper parameters are C,gamma 
from sklearn.svm import SVC
clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = GridSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')

model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>10.3 Confusion Matrix<jupyter_code>cm_tfidfw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidfw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidfw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 36 
 false positives are 5577 
 false negatives are 456
 true positives are 35931<jupyter_text>10.4 Calculating Accuracy, Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidfw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the  SVM using RBF kernel for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_tfidfw2v))

# Error on test data
test_error_tfidfw2v = 100-acc_tfidfw2v
print("\nTest Error  of the  SVM using RBF kernel for C and gamma =   %f%%" % (test_error_tfidfw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM using RBF kernel for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM using RBF kernel for C and gamma = %.3f ,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM using RBF kernel for alpha \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the  SVM using RBF kernel for C and gamma = 10.000 ,10000.000 is 85.635714%

Test Error  of the  SVM using RBF kernel for C and gamma =   14.364286%

The Test Precision of the  SVM using RBF kernel for C and gamma = 10.000 ,10000.000 is 0.865640

The Test Recall of the  SVM using RBF kernel for C and gamma = 10.000 ,10000.000 is 0.987468

The Test classification report of the  SVM using RBF kernel for alpha 

                precision    recall  f1-score   support

           0       0.07      0.01      0.01      5613
           1       0.87      0.99      0.92     36387

   micro avg       0.86      0.86      0.86     42000
   macro avg       0.47      0.50      0.47     42000
weighted avg       0.76      0.86      0.80     42000<jupyter_text>10.5 APPLYING RBF KERNEL BY RandomizedSearchCV<jupyter_code>#in RBF SVC hyper parameters are C,gamma 
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV


clf=SVC(kernel='rbf',max_iter=100,class_weight='balanced')
tuned_parameters = {'C': [10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4],'gamma':[10**-4,10**-3, 10**-2,10**-1, 1, 10**1,10**2,10**3, 10**4]}
model = RandomizedSearchCV(clf, tuned_parameters,cv=3,n_jobs=-1)
model.fit(standardized_data_train, y_train)
print("tunded parameters",model.best_estimator_)
print("Accuracy of model using SVC classifier by rbf kernel {:.2f}".format(model.score(standardized_data_test, y_test)))
a = model.best_params_
optimal_C = a.get('C')
optimal_gamma=a.get('gamma')
model.best_params_
results = model.cv_results_
results['mean_test_score']
y_pred = model.predict(standardized_data_test)<jupyter_output><empty_output><jupyter_text>10.6 Confusion Matrix<jupyter_code>cm_tfidfw2v=confusion_matrix(y_test,y_pred)
print("Confusion Matrix:")
sns.heatmap(cm_tfidfw2v, annot=True, fmt='d')
plt.show()
#finding out  true negative , false positive , false negative and true positve 
tn, fp, fn, tp = cm_tfidfw2v.ravel()
( tp, fp, fn, tp)
print(" true negitves are {} \n false positives are {} \n false negatives are {}\n true positives are {} \n ".format(tn,fp,fn,tp))<jupyter_output>true negitves are 772 
 false positives are 4841 
 false negatives are 6011
 true positives are 30376<jupyter_text>10.7 Calculating Accuracy,Error on test data,Precision,Recall,Classification Report<jupyter_code>from sklearn.metrics import recall_score
from sklearn.metrics import precision_score
from sklearn.metrics import classification_report


# evaluating accuracy
acc_tfidfw2v = accuracy_score(y_test, y_pred) * 100
print('\nThe Test Accuracy of the  SVM using RBF kernel for C and gamma = %.3f ,%.3f is %f%%' % (optimal_C,optimal_gamma, acc_tfidfw2v))

# Error on test data
test_error_tfidfw2v = 100-acc_tfidfw2v
print("\nTest Error  of the  SVM using RBF kernel for C and gamma  %f%%" % (test_error_tfidfw2v))

# evaluating precision
precision_score = precision_score(y_test, y_pred) 
print('\nThe Test Precision of the  SVM using RBF kernel for C and gamma = %.3f,%.3f is %f' % (optimal_C,optimal_gamma, precision_score))

# evaluating recall
recall_score = recall_score(y_test, y_pred)
print('\nThe Test Recall of the  SVM using RBF kernel for C and gamma = %.3f,%.3f is %f' % (optimal_C,optimal_gamma, recall_score))

# evaluating Classification report
classification_report = classification_report(y_test, y_pred)
print('\nThe Test classification report of the  SVM using RBF kernel for C and gamma \n\n ',(classification_report))<jupyter_output>The Test Accuracy of the  SVM using RBF kernel for C and gamma = 100.000 ,10.000 is 74.161905%

Test Error  of the  SVM using RBF kernel for C and gamma  25.838095%

The Test Precision of the  SVM using RBF kernel for C and gamma = 100.000,10.000 is 0.862538

The Test Recall of the  SVM using RBF kernel for C and gamma = 100.000,10.000 is 0.834804

The Test classification report of the  SVM using RBF kernel for C and gamma 

                precision    recall  f1-score   support

           0       0.11      0.14      0.12      5613
           1       0.86      0.83      0.85     36387

   micro avg       0.74      0.74      0.74     42000
   macro avg       0.49      0.49      0.49     42000
weighted avg       0.76      0.74      0.75     42000
{'fraction_non_alphanumeric': 0.10667495399315077, 'fraction_numerical': 0.04763994003875544, 'mean_word_length': 4.855562691786199, 'pattern_counts': {'":': 0, '<': 282, '<?xml version=': 0, '>': 282, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 32, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '80', 'dir': 'jupyter-structured-clean-dedup', 'id': '308395', 'license': 'MIT', 'n_tokens_mistral': 31918, 'n_tokens_neox': 28618, 'n_words': 7302, 'path': 'svm updated.ipynb', 'repo_name': 'saikrishna6680/Amazon-Fine-Food-Reviews-', 'size': '82053'}