<jupyter_start><jupyter_text>Logistic Regression on Amazon Food Reviews Data Source: https://www.kaggle.com/snap/amazon-fine-food-reviewsThe Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.Number of reviews: 568,454 Number of users: 256,059 Number of products: 74,258 Timespan: Oct 1999 - Oct 2012 Number of Attributes/Columns in data: 10Attribute Information:1.  index2.  Id3.  ProductId - unique identifier for the product4.  UserId - unqiue identifier for the user5.  ProfileName6.  HelpfulnessNumerator - number of users who found the review helpful7.  HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not8.  Score - rating between 1 and 59.  Time - timestamp for the review10. Summary - brief summary of the review11. Text - text of the review12. ProcessedText - Cleaned & Preprocessed Text of the review**Objective: Given Amazon Food reviews, convert all the reviews into a vector using three techniques: ****1. Average W2V. ****2. Average TFIDF-W2V. ****3. GloVe. ****Then perform following tasks under each technique:****Task 1. Split train and test data in a ratio of 80:20.** **Task 2. Perform GridSearch Cross Validation and Random Search Cross Validation to find optimal Value of λ.****Task 3. Apply Logistic Regression using both L1 and L2 regularizations and report accuracy.****Task 4. Use L1 regularization for different values of λ and report error and sparsity for each value of λ.****Task 5. Check for multi-collinearity of features and find top-10 most important features**[Q] How to determine if a review is positive or negative?[Ans] We could use the Score/Rating. A rating of 4 or 5 could be cosnidered a positive review. A review of 1 or 2 could be considered negative. A review of 3 is nuetral and ignored. This is an approximate and proxy way of determining the polarity (positivity/negativity) of a review.Loading the dataSQLite DatabaseIn order to load the data, We have used the SQLITE dataset as it easier to query the data and visualise the data efficiently. Here as we only want to get the global sentiment of the recommendations (positive or negative), we will purposefully ignore all Scores equal to 3. If the score id above 3, then the recommendation wil be set to "positive". Otherwise, it will be set to "negative".<jupyter_code>import sqlite3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plot

from gensim.models import Word2Vec
import gensim
import csv
import re
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import StandardScaler

from sklearn.cross_validation import train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.grid_search import RandomizedSearchCV
from sklearn.linear_model import LogisticRegression
connection = sqlite3.connect('FinalAmazonFoodReviewsDataset.sqlite')
data = pd.read_sql_query("SELECT * FROM Reviews", connection)
data.head()
data.shape
data["Score"].value_counts()
def changingScores(score):
    if score == "Positive":
        return 1
    else:
        return -1
# changing score
# Positive = 1
# Negative = -1
actualScore = list(data["Score"])                                                                                         
positiveNegative = list(map(changingScores, actualScore)) #map(function, list of numbers)
data['Score'] = positiveNegative
data.head()
allPositiveReviews = data[(data["Score"] == 1)]
allPositiveReviews.shape
positiveReviews_2500 = allPositiveReviews[:2500]
positiveReviews_2500.shape
positiveReviews_2500.head()
allNegativeReviews = data[(data["Score"] == -1)]
allNegativeReviews.shape
negativeReviews_2500 = allNegativeReviews[:2500]
negativeReviews_2500.shape
negativeReviews_2500.head()
frames_5000 = [positiveReviews_2500, negativeReviews_2500]
FinalPositiveNegative = pd.concat(frames_5000)
FinalPositiveNegative.shape
#Sorting FinalDataframe by "Time"
FinalSortedPositiveNegative_5000 = FinalPositiveNegative.sort_values('Time', axis=0, ascending=True, inplace=False)
FinalSortedPositiveNegativeScore_5000 = FinalSortedPositiveNegative_5000["Score"]
FinalSortedPositiveNegative_5000.shape
FinalSortedPositiveNegativeScore_5000.shape
Data = FinalSortedPositiveNegative_5000
Data_Labels = FinalSortedPositiveNegativeScore_5000
print(Data.shape)
print(Data_Labels.shape)
Data.head()<jupyter_output><empty_output><jupyter_text>Average W2V<jupyter_code>i = 0
listOfSentences = []
for sentence in Data["ProcessedText"].values:
    subSentence = []
    for word in sentence.split():
        subSentence.append(word)
        
    listOfSentences.append(subSentence)
print(Data['ProcessedText'].values[0])
print("\n")
print(listOfSentences[0:2])
print("\n")
print(type(listOfSentences))
w2vModel = gensim.models.Word2Vec(listOfSentences, size=400, min_count=5, workers=4)
# compute average word2vec for each review.
sentenceAsW2V = []
for sentence in listOfSentences:
    sentenceVector = np.zeros(400)
    TotalWordsPerSentence = 0
    for word in sentence:
        try:
            vect = w2vModel.wv[word]
            sentenceVector += vect
            TotalWordsPerSentence += 1
        except:
            pass
    sentenceVector /= TotalWordsPerSentence
    sentenceAsW2V.append(sentenceVector)

print(type(sentenceAsW2V))
print(len(sentenceAsW2V))
print(len(sentenceAsW2V[0]))
standardized_Avg_w2v = StandardScaler().fit_transform(sentenceAsW2V)
print(standardized_Avg_w2v.shape)
print(type(standardized_Avg_w2v))<jupyter_output>(5000, 400)
<class 'numpy.ndarray'><jupyter_text>Task 1. Split train and test data in a ratio of 80:20.<jupyter_code>train_AvgW2V, test_AvgW2V, train_labels_AvgW2V, test_labels_AvgW2V = train_test_split(standardized_Avg_w2v, FinalSortedPositiveNegativeScore_5000, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Task 2. Perform GridSearch Cross Validation and Random Search Cross Validation to find optimal Value of λ.  Grid Search<jupyter_code>clf = LogisticRegression()

hyper_parameters = [{'C': [10**-3, 10**-2, 10**-1, 10**0, 2, 4, 6, 8, 10**1, 50, 10**2, 500, 10**3]}]
bestCV = GridSearchCV(clf, hyper_parameters, scoring = "accuracy", cv = 5)
bestCV.fit(train_AvgW2V, train_labels_AvgW2V)

print(bestCV.best_estimator_)
best_parameter = bestCV.best_params_ 
best_parameter["C"]
scoreData = bestCV.grid_scores_
scoreData
error = []
parameter = []
for i in range(len(scoreData)):
    error.append(1 - scoreData[i][1])
    parameter.append(scoreData[i][0]["C"])
    
plot.plot(parameter, np.round(error, 4))
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error1 = []
for e in error:
    error1.append(np.round(e,3))
    
parameter1 = []
for p in parameter:
    parameter1.append(np.round(p,3))

for xy in zip(parameter1, error1):
    plot.annotate(xy,xy)

plot.show()<jupyter_output><empty_output><jupyter_text>Random Search<jupyter_code>n = list(np.random.normal(loc=1000, scale=300, size = 500))  #taking 500 numbers which are distributed normally with mean = 1000
                                                            #and std-dev = 300
clf = LogisticRegression()

hyper_parameters2 = {'C': n}

bestCV_random = RandomizedSearchCV(clf, hyper_parameters2, scoring = "accuracy", cv = 3)

bestCV_random.fit(train_AvgW2V, train_labels_AvgW2V)
print(bestCV_random.best_estimator_)
best_random_parameter = bestCV_random.best_params_ 
best_random_parameter["C"]
scoreRandomData = bestCV_random.grid_scores_
scoreRandomData
error2 = []
parameter2 = []
for i in range(len(scoreRandomData)):
    error2.append(1 - scoreRandomData[i][1])
    parameter2.append(scoreRandomData[i][0]["C"])  

plot.plot(parameter2, error2)
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error3 = []
for e in error2:
    error3.append(np.round(e,3))
    
parameter3 = []
for p in parameter2:
    parameter3.append(np.round(p,3))

for xy in zip(parameter3, error3):
        plot.annotate(xy,xy)

plot.show()
# We are taking our hyper-parameter λ as the average of best λ computed from gridSearchCV and RandomSearchCV
FinalHP = (best_parameter["C"] + best_random_parameter["C"]) / 2
FinalHP<jupyter_output><empty_output><jupyter_text>Task 3. Apply Logistic Regression using both L1 and L2 regularizations and report accuracy.  Implementing L2 Regularization<jupyter_code>model = LogisticRegression(penalty="l2", C = FinalHP)

model.fit(train_AvgW2V, train_labels_AvgW2V)

prediction = model.predict(test_AvgW2V)

AccuracyScore = accuracy_score(test_labels_AvgW2V, prediction) * 100

print("Accuracy score on L2 regularization = "+str(AccuracyScore)+"%")
Precision = precision_score(test_labels_AvgW2V, prediction)
print("Precision Score on L2 regularization = "+str(Precision))
Recall = recall_score(test_labels_AvgW2V, prediction)
print("Recall Score on L2 regularization = "+str(Recall))
Confusion_Matrix = confusion_matrix(test_labels_AvgW2V, prediction)
print("Confusion Matrix on L2 regularization  \n"+str(Confusion_Matrix))
tn, fp, fn, tp = confusion_matrix(test_labels_AvgW2V, prediction).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Implementing L1 Regularization<jupyter_code>model2 = LogisticRegression(penalty="l1", C = FinalHP)

model2.fit(train_AvgW2V, train_labels_AvgW2V)

prediction2 = model2.predict(test_AvgW2V)

AccuracyScore2 = accuracy_score(test_labels_AvgW2V, prediction2) * 100

print("Accuracy score on L1 regularization = "+str(AccuracyScore2)+"%")
Precision2 = precision_score(test_labels_AvgW2V, prediction2)
print("Precision Score on L1 regularization = "+str(Precision2))
Recall2 = recall_score(test_labels_AvgW2V, prediction2)
print("Recall Score on L1 regularization = "+str(Recall2))
Confusion_Matrix2 = confusion_matrix(test_labels_AvgW2V, prediction2)
print("Confusion Matrix on L1 regularization \n"+str(Confusion_Matrix2))
tn, fp, fn, tp = confusion_matrix(test_labels_AvgW2V, prediction2).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Task 4. Use L1 regularization for different values of λ and report error and sparsity for each value of λ.<jupyter_code>model3 = LogisticRegression(penalty="l1", C = 10000)

model3.fit(train_AvgW2V, train_labels_AvgW2V)

accuracyScore_10000 = model3.score(test_AvgW2V, test_labels_AvgW2V)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.0001 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.0001 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 1000)

model3.fit(train_AvgW2V, train_labels_AvgW2V)

accuracyScore_1000 = model3.score(test_AvgW2V, test_labels_AvgW2V)

error = 1 - accuracyScore_1000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.001 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.001 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10)

model3.fit(train_AvgW2V, train_labels_AvgW2V)

accuracyScore_10 = model3.score(test_AvgW2V, test_labels_AvgW2V)

error = 1 - accuracyScore_10

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.1 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.1 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10**-3)

model3.fit(train_AvgW2V, train_labels_AvgW2V)

accuracyScore_0001 = model3.score(test_AvgW2V, test_labels_AvgW2V)

error = 1 - accuracyScore_0001

weightVector = model3.coef_

print("Number of non-zero for λ value of 1000 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 1000 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10**-4)

model3.fit(train_AvgW2V, train_labels_AvgW2V)

accuracyScore_00001 = model3.score(test_AvgW2V, test_labels_AvgW2V)

error = 1 - accuracyScore_00001

weightVector = model3.coef_

print("Number of non-zero for λ value of 10000 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 10000 = "+str(error))<jupyter_output>Number of non-zero for λ value of 10000 = 0
Error for λ value of 10000 = 0.498<jupyter_text>Task 5. Check for multi-collinearity of features and find top-10 most important features<jupyter_code># computing weight vector prior to adding noise in out data
model4 = LogisticRegression(penalty="l2", C = FinalHP)

model4.fit(train_AvgW2V, train_labels_AvgW2V)

weightVector1 = model4.coef_
noise = np.random.normal(loc=0.1, scale=0.1) # generating random positive number from normal distribution
noise_matrix = np.full((4000, 400), noise) # generating matrix of size 4000 * 400 where every number in matrix is 'n' 
dataWithNoise_AvgW2V = noise_matrix + train_AvgW2V # adding noise to training data fro pertubation test

print(noise)
print(dataWithNoise_AvgW2V.shape)
# computing weight vector after adding noise in out data
model5 = LogisticRegression(penalty="l2", C = FinalHP)

model5.fit(dataWithNoise_AvgW2V, train_labels_AvgW2V)

weightVector2 = model5.coef_
differenceInWeights = weightVector1 - weightVector2
differenceInWeights.ravel()<jupyter_output><empty_output><jupyter_text>**Here, as you can see that the differnce in the weight vectors prior and after adding noise is high. It means that features are collinear, hence we cannot use weight vectors as feature importance.****Since word to vec generates vectors for words which are dependent on each other owing to their similarity of preserving sementic meaning between similar words. Therefore, features in word 2 vec are collinear**  TFIDF-W2V<jupyter_code>tfidf_vect = TfidfVectorizer(ngram_range = (1,2))
tfidf = tfidf_vect.fit_transform(Data["ProcessedText"].values)
w2v_Model = gensim.models.Word2Vec(listOfSentences, size=300, min_count=5, workers=4)
print(tfidf.shape)
print(type(tfidf))
# TF-IDF weighted Word2Vec
tfidf_features = tfidf_vect.get_feature_names()

tfidf_w2v = []
reviews = 0

for sentence in listOfSentences:
    sentenceVector = np.zeros(300)
    weightTfidfSum = 0
    for word in sentence:
        try:
            W2V_Vector = w2v_Model.wv[word]
            tfidfVector = tfidf[reviews, tfidf_features.index(word)]
            sentenceVector += (W2V_Vector * tfidfVector)
            weightTfidfSum += tfidfVector
        except:
            pass
    sentenceVector /= weightTfidfSum
    tfidf_w2v.append(sentenceVector)
    reviews += 1
standardized_tfidf_w2v = StandardScaler().fit_transform(tfidf_w2v)
print(standardized_tfidf_w2v.shape)
print(type(standardized_tfidf_w2v))<jupyter_output>(5000, 300)
<class 'numpy.ndarray'><jupyter_text>Task 1. Split train and test data in a ratio of 80:20.<jupyter_code>train_tfidf_w2v, test_tfidf_w2v, train_labels_tfidf_w2v, test_labels_tfidf_w2v = train_test_split(standardized_tfidf_w2v, FinalSortedPositiveNegativeScore_5000, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Task 2. Perform GridSearch Cross Validation and Random Search Cross Validation to find optimal Value of λ.  Grid Search<jupyter_code>clf = LogisticRegression()

hyper_parameters = [{'C': [10**-3, 10**-2, 10**-1, 10**0, 2, 4, 6, 8, 10**1, 50, 10**2, 500, 10**3]}]
bestCV = GridSearchCV(clf, hyper_parameters, scoring = "accuracy", cv = 5)
bestCV.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

print(bestCV.best_estimator_)
best_parameter = bestCV.best_params_ 
best_parameter["C"]
scoreData = bestCV.grid_scores_
scoreData
error = []
parameter = []
for i in range(len(scoreData)):
    error.append(1 - scoreData[i][1])
    parameter.append(scoreData[i][0]["C"])
    
plot.plot(parameter, np.round(error, 4))
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error1 = []
for e in error:
    error1.append(np.round(e,3))
    
parameter1 = []
for p in parameter:
    parameter1.append(np.round(p,3))

for xy in zip(parameter1, error1):
    plot.annotate(xy,xy)

plot.show()<jupyter_output><empty_output><jupyter_text>Random Search<jupyter_code>n = list(np.random.normal(loc=100, scale=40, size = 500))  #taking 500 numbers which are distributed normally with mean = 100
                                                            #and std-dev = 40
clf = LogisticRegression()

hyper_parameters2 = {'C': n}

bestCV_random = RandomizedSearchCV(clf, hyper_parameters2, scoring = "accuracy", cv = 3)

bestCV_random.fit(train_tfidf_w2v, train_labels_tfidf_w2v)
print(bestCV_random.best_estimator_)
best_random_parameter = bestCV_random.best_params_ 
best_random_parameter["C"]
scoreRandomData = bestCV_random.grid_scores_
scoreRandomData
error2 = []
parameter2 = []
for i in range(len(scoreRandomData)):
    error2.append(1 - scoreRandomData[i][1])
    parameter2.append(scoreRandomData[i][0]["C"])  

plot.plot(parameter2, error2)
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error3 = []
for e in error2:
    error3.append(np.round(e,3))
    
parameter3 = []
for p in parameter2:
    parameter3.append(np.round(p,3))

for xy in zip(parameter3, error3):
        plot.annotate(xy,xy)

plot.show()
# We are taking our hyper-parameter λ as the average of best λ computed from gridSearchCV and RandomSearchCV
FinalHP = (best_parameter["C"] + best_random_parameter["C"]) / 2
FinalHP<jupyter_output><empty_output><jupyter_text>Task 3. Apply Logistic Regression using both L1 and L2 regularizations and report accuracy.  Implementing L2 Regularization<jupyter_code>model = LogisticRegression(penalty="l2", C = FinalHP)

model.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

prediction = model.predict(test_tfidf_w2v)

AccuracyScore = accuracy_score(test_labels_tfidf_w2v, prediction) * 100

print("Accuracy score on L2 regularization = "+str(AccuracyScore)+"%")
Precision = precision_score(test_labels_tfidf_w2v, prediction)
print("Precision Score on L2 regularization = "+str(Precision))
Recall = recall_score(test_labels_tfidf_w2v, prediction)
print("Recall Score on L2 regularization = "+str(Recall))
Confusion_Matrix = confusion_matrix(test_labels_tfidf_w2v, prediction)
print("Confusion Matrix on L2 regularization \n"+str(Confusion_Matrix))
tn, fp, fn, tp = confusion_matrix(test_labels_tfidf_w2v, prediction).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Implementing L1 Regularization<jupyter_code>model2 = LogisticRegression(penalty="l1", C = FinalHP)

model2.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

prediction2 = model2.predict(test_tfidf_w2v)

AccuracyScore2 = accuracy_score(test_labels_tfidf_w2v, prediction2) * 100

print("Accuracy score on L1 regularization = "+str(AccuracyScore2)+"%")
Precision2 = precision_score(test_labels_tfidf_w2v, prediction2)
print("Precision Score on L1 regularization = "+str(Precision2))
Recall2 = recall_score(test_labels_tfidf_w2v, prediction2)
print("Precision Score on L1 regularization = "+str(Precision2))
Confusion_Matrix2 = confusion_matrix(test_labels_tfidf_w2v, prediction2)
print("Confusion Matrix on L1 regularization \n"+str(Confusion_Matrix2))
tn, fp, fn, tp = confusion_matrix(test_labels_tfidf_w2v, prediction2).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Task 4. Use L1 regularization for different values of λ and report error and sparsity for each value of λ.<jupyter_code>model3 = LogisticRegression(penalty="l1", C = 10000)

model3.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

accuracyScore_10000 = model3.score(test_tfidf_w2v, test_labels_tfidf_w2v)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.0001 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.0001 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 100)

model3.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

accuracyScore_10000 = model3.score(test_tfidf_w2v, test_labels_tfidf_w2v)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.01 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.01 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 1)

model3.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

accuracyScore_10000 = model3.score(test_tfidf_w2v, test_labels_tfidf_w2v)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 1 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 1 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10**-3)

model3.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

accuracyScore_10000 = model3.score(test_tfidf_w2v, test_labels_tfidf_w2v)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 1000 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 1000 = "+str(error))<jupyter_output>Number of non-zero for λ value of 0.0001 = 0
Error for λ value of 1000 = 0.498<jupyter_text>Task 5. Check for multi-collinearity of features and find top-10 most important features<jupyter_code># computing weight vector prior to adding noise in out data
model4 = LogisticRegression(penalty="l2", C = FinalHP)

model4.fit(train_tfidf_w2v, train_labels_tfidf_w2v)

weightVector1 = model4.coef_
noise = np.random.normal(loc=0.1, scale=0.1) # generating random positive number from normal distribution
noise_matrix = np.full((4000, 300), noise) # generating matrix of size 4000 * 300 where every number in matrix is 'noise' 
dataWithNoise_tfidf_AvgW2V = noise_matrix + train_tfidf_w2v # adding noise to training data fro pertubation test

print(noise)
print(dataWithNoise_tfidf_AvgW2V.shape)
# computing weight vector after adding noise in out data
model5 = LogisticRegression(penalty="l2", C = FinalHP)

model5.fit(dataWithNoise_tfidf_AvgW2V, train_labels_tfidf_w2v)

weightVector2 = model5.coef_
differenceWeights = weightVector1 - weightVector2
differenceWeights.ravel()<jupyter_output><empty_output><jupyter_text>**Here, as you can see that the differnce in the weight vectors prior and after adding noise is high. It means that features are collinear, hence we cannot use weight vectors as feature importance. Since word to vec generates vectors for words which are dependent on each other owing to their similarity of preserving sementic meaning between similar words. Therefore, features in word 2 vec are collinear**  GloVe<jupyter_code>def cleanhtml(sentence): #function to clean htmltags
    cleanr = re.compile("<.*?>")
    cleantext = re.sub(cleanr, " ", sentence)
    return cleantext

def cleanpunc(sentence): #function to clean the word of any punctuation or special characters
    cleaned = re.sub(r'[?|!|\'|"|#]',r'',sentence)
    cleaned = re.sub(r'[.|,|)|(|\|/]',r' ',cleaned)
    return  cleaned
#removing HTML tags and punctuation from our text

i = 0
final_string = []
s = ""
for sentence in data["Text"].values:
    filteredSentence = []
    EachReviewText = ""
    sentenceHTMLCleaned = cleanhtml(sentence)
    for eachWord in sentenceHTMLCleaned.split():
        for sentencePunctCleaned in cleanpunc(eachWord).split():
            if((sentencePunctCleaned.isalpha()) & (len(sentencePunctCleaned)>2)):
                sentenceLower = sentencePunctCleaned.lower()
                filteredSentence.append(sentenceLower)
                     
    EachReviewText = ' '.join(filteredSentence)
    final_string.append(EachReviewText)
data["ProcessedText2"] = final_string
data.head()
allPositiveReviews2 = data[(data["Score"] == 1)]
allPositiveReviews2.shape
positiveReviews2_500 = allPositiveReviews2[:500]
positiveReviews2_500.shape
positiveReviews2_500.head()
allNegativeReviews2 = data[(data["Score"] == -1)]
allNegativeReviews2.shape
negativeReviews2_500 = allNegativeReviews2[:500]
negativeReviews2_500.shape
negativeReviews2_500.head()
frames2_1000 = [positiveReviews2_500, negativeReviews2_500]
FinalPositiveNegative2 = pd.concat(frames2_1000)
FinalPositiveNegative2.shape
#Sorting FinalDataframe by "Time"
FinalSortedPositiveNegative2_1000 = FinalPositiveNegative2.sort_values('Time', axis=0, ascending=True, inplace=False)
FinalSortedPositiveNegativeScore2_1000 = FinalSortedPositiveNegative2_1000["Score"]
FinalSortedPositiveNegative2_1000.shape
FinalSortedPositiveNegativeScore2_1000.shape
Data2 = FinalSortedPositiveNegative2_1000
Data2_Labels = FinalSortedPositiveNegativeScore2_1000
print(Data2.shape)
print(Data2_Labels.shape)
Data2.head()
i = 0
listOfSentences2 = []
for sentence in Data2["ProcessedText2"].values:
    subSentence = []
    for word in sentence.split():
        subSentence.append(word)
        
    listOfSentences2.append(subSentence)
print(Data2['ProcessedText2'].values[0])
print("\n")
print(listOfSentences2[0:2])
print("\n")
print(type(listOfSentences2))
print("\n")
print(len(listOfSentences2))
#loading pre-trained GloVe vectors
words = pd.read_table("glove.6B.100d.txt", sep=" ", index_col=0, header=None, quoting=csv.QUOTE_NONE) 

# Here, We have downloaded pre-trained Glove vectors. You just have to type "Glove word vectors" on google then click on 
# "https://nlp.stanford.edu/projects/glove/" link. Then you can download pre-trained word-vectors. Zip file will be 
# downloaded, you just have to extract it then load the txt file from extracted folder into ipython notebook using pandas,
# just like we have done above.
def check(word):
    if (words.index == word).any():
        return 1
    else:
        return 0
# compute average GloVe for each review.
sentenceAsGlove = []
for sentence in listOfSentences2:
    sentenceVector = np.zeros(100)
    TotalWordsPerSentence = 0
    for word in sentence:
        if check(word) == 1:
            vect = words.loc[word]
            sentenceVector += vect
            TotalWordsPerSentence += 1
            
    sentenceVector /= TotalWordsPerSentence
    sentenceAsGlove.append(sentenceVector)

print(type(sentenceAsGlove))
print(len(sentenceAsGlove))
print(len(sentenceAsGlove[0]))
standardized_Avg_Glove = StandardScaler().fit_transform(sentenceAsGlove)
print(standardized_Avg_Glove.shape)
print(type(standardized_Avg_Glove))<jupyter_output>(1000, 100)
<class 'numpy.ndarray'><jupyter_text>Task 1. Split train and test data in a ratio of 80:20.<jupyter_code>train_glove, test_glove, train_labels_glove, test_labels_glove = train_test_split(standardized_Avg_Glove, Data2_Labels, test_size = 0.2, random_state = 0)
train_glove.shape, test_glove.shape, train_labels_glove.shape, test_labels_glove.shape<jupyter_output><empty_output><jupyter_text>Task 2. Perform GridSearch Cross Validation and Random Search Cross Validation to find optimal Value of λ.  Grid Search<jupyter_code>clf = LogisticRegression()

hyper_parameters = [{'C': [10**-3, 10**-2, 10**-1, 10**0, 2, 4, 6, 8, 10**1, 50, 10**2, 500, 10**3]}]
bestCV = GridSearchCV(clf, hyper_parameters, scoring = "accuracy", cv = 5)
bestCV.fit(train_glove, train_labels_glove)

print(bestCV.best_estimator_)
best_parameter = bestCV.best_params_ 
best_parameter["C"]
scoreData = bestCV.grid_scores_
scoreData
error = []
parameter = []
for i in range(len(scoreData)):
    error.append(1 - scoreData[i][1])
    parameter.append(scoreData[i][0]["C"])
    
plot.plot(parameter, np.round(error, 4))
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error1 = []
for e in error:
    error1.append(np.round(e,3))
    
parameter1 = []
for p in parameter:
    parameter1.append(np.round(p,3))

for xy in zip(parameter1, error1):
    plot.annotate(xy,xy)

plot.show()<jupyter_output><empty_output><jupyter_text>Random Search<jupyter_code>n = list(np.random.normal(loc=0.01, scale=0.001, size = 500))  #taking 500 numbers which are distributed normally with mean = 0.01
                                                            #and std-dev = 0.001
clf = LogisticRegression()

hyper_parameters2 = {'C': n}

bestCV_random = RandomizedSearchCV(clf, hyper_parameters2, scoring = "accuracy", cv = 3)

bestCV_random.fit(train_glove, train_labels_glove)
print(bestCV_random.best_estimator_)
best_random_parameter = bestCV_random.best_params_ 
best_random_parameter["C"]
scoreRandomData = bestCV_random.grid_scores_
scoreRandomData
error2 = []
parameter2 = []
for i in range(len(scoreRandomData)):
    error2.append(1 - scoreRandomData[i][1])
    parameter2.append(scoreRandomData[i][0]["C"])  

plot.plot(parameter2, error2)
plot.xlabel("Parameter 'c'")
plot.ylabel("Error")

error3 = []
for e in error2:
    error3.append(np.round(e,3))
    
parameter3 = []
for p in parameter2:
    parameter3.append(np.round(p,3))

for xy in zip(parameter3, error3):
        plot.annotate(xy,xy)

plot.show()
# We are taking our hyper-parameter λ as the average of best λ computed from gridSearchCV and RandomSearchCV
FinalHP = (best_parameter["C"] + best_random_parameter["C"]) / 2
FinalHP<jupyter_output><empty_output><jupyter_text>Task 3. Apply Logistic Regression using both L1 and L2 regularizations and report accuracy.  Implementing L2 Regularization<jupyter_code>model = LogisticRegression(penalty="l2", C = FinalHP)

model.fit(train_glove, train_labels_glove)

prediction = model.predict(test_glove)

AccuracyScore = accuracy_score(test_labels_glove, prediction) * 100

print("Accuracy score on L2 regularization = "+str(AccuracyScore)+"%")
Precision = precision_score(test_labels_glove, prediction)
print("Precision Score on L2 regularization = "+str(Precision))
Recall = recall_score(test_labels_glove, prediction)
print("Recall Score on L2 regularization = "+str(Recall))
ConfusionMatrix = confusion_matrix(test_labels_glove, prediction)
print("Confusion Matrix on L2 regularization \n= "+str(ConfusionMatrix))
tn, fp, fn, tp = confusion_matrix(test_labels_glove, prediction).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Implementing L1 Regularization<jupyter_code>model2 = LogisticRegression(penalty="l1", C = FinalHP)

model2.fit(train_glove, train_labels_glove)

prediction2 = model2.predict(test_glove)

AccuracyScore2 = accuracy_score(test_labels_glove, prediction2) * 100

print("Accuracy score on L1 regularization = "+str(AccuracyScore2)+"%")
Precision = precision_score(test_labels_glove, prediction2)
print("Precision Score on L1 regularization = "+str(Precision))
Recall = recall_score(test_labels_glove, prediction2)
print("Recall Score on L1 regularization = "+str(Recall))
ConfusionMatrix = confusion_matrix(test_labels_glove, prediction2)
print("Confusion Matrix on L1 regularization \n= "+str(ConfusionMatrix))
tn, fp, fn, tp = confusion_matrix(test_labels_glove, prediction2).ravel()
tn, fp, fn, tp<jupyter_output><empty_output><jupyter_text>Task 4. Use L1 regularization for different values of λ and report error and sparsity for each value of λ.<jupyter_code>model3 = LogisticRegression(penalty="l1", C = 1000)

model3.fit(train_glove, train_labels_glove)

accuracyScore_10000 = model3.score(test_glove, test_labels_glove)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.001 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.001 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 100)

model3.fit(train_glove, train_labels_glove)

accuracyScore_10000 = model3.score(test_glove, test_labels_glove)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 0.01 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 0.01 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 1)

model3.fit(train_glove, train_labels_glove)

accuracyScore_10000 = model3.score(test_glove, test_labels_glove)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 1 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 1 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10**-2)

model3.fit(train_glove, train_labels_glove)

accuracyScore_10000 = model3.score(test_glove, test_labels_glove)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 100 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 100 = "+str(error))
model3 = LogisticRegression(penalty="l1", C = 10**-4)

model3.fit(train_glove, train_labels_glove)

accuracyScore_10000 = model3.score(test_glove, test_labels_glove)

error = 1 - accuracyScore_10000

weightVector = model3.coef_

print("Number of non-zero for λ value of 10000 = "+str(np.count_nonzero(weightVector)))

print("Error for λ value of 10000 = "+str(error))<jupyter_output>Number of non-zero for λ value of 10000 = 0
Error for λ value of 10000 = 0.525<jupyter_text>Task 5. Check for multi-collinearity of features and find top-10 most important features<jupyter_code># computing weight vector prior to adding noise in out data
model4 = LogisticRegression(penalty="l2", C = FinalHP)

model4.fit(train_glove, train_labels_glove)

weightVector1 = model4.coef_
noise = np.random.normal(loc=0.1, scale=0.1) # generating random positive number from normal distribution
noise_matrix = np.full((800, 100), noise) # generating matrix of size 4000 * 300 where every number in matrix is 'n' 
dataWithNoise_glove = noise_matrix + train_glove # adding noise to training data fro pertubation test

print(noise)
print(dataWithNoise_glove.shape)
# computing weight vector after adding noise in out data
model5 = LogisticRegression(penalty="l2", C = FinalHP)

model5.fit(dataWithNoise_glove, train_labels_glove)

weightVector2 = model5.coef_
differenceWeights = weightVector1 - weightVector2
differenceWeights.ravel()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10264017501273637, 'fraction_numerical': 0.0409361982678534, 'mean_word_length': 5.756428426807046, 'pattern_counts': {'":': 1, '<': 99, '<?xml version=': 0, '>': 100, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '437361', 'license': 'MIT', 'n_tokens_mistral': 12154, 'n_tokens_neox': 11120, 'n_words': 2758, 'path': 'sentimentaal classification/Logistic_Regression_Amazon_Food_Reviews/LogisticRegression_AmazonFoodReviews.ipynb', 'repo_name': 'rahul263-stack/PROJECT-Dump', 'size': '33369'}