<jupyter_start><jupyter_text>Natural Language Processing: Sentiment Analysis with Python ------------------------------------ Authors :- <NAME>- <NAME>- <NAME>-----------------------------------  I. Introduction Dans ce projet, nous fessons face à un probleme de classification supervisé sur des commentaires recoltés sur amazon. Le but ici est de traiter puis d'implémenter différent modeles afin d'en déterminer le meilleur pour faire de la prédiction sur de nouveaux commentaires.les principaux modèles utilisés dans ce TP sont : - **Naïve Bayes Classifier**- **Support Vector Machine Classification**- **K-Nearest Neighbor Classification**- **Artificial Neural Networks**Pour la suite, on effectuera notre analyse suivant le plan ci dessous :![plan](https://user-images.githubusercontent.com/45575893/79023555-e36fb280-7b80-11ea-9e7f-060aee10b0e5.png)<jupyter_code>!pip install jyquickhelper
!pip install langdetect
!pip install scikit-plot
### Ignore warning
import warnings
warnings.filterwarnings("ignore")

### Print table of content
from jyquickhelper import add_notebook_menu
add_notebook_menu()<jupyter_output><empty_output><jupyter_text>II. Importation des données<jupyter_code>### Packages de base
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
get_ipython().magic('matplotlib inline')
import time
from itertools import chain 
from multiprocessing import Pool
import gc
import sys
from scipy import sparse
from random import sample 

### Packages pour NLP
from sklearn.model_selection import train_test_split
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
import nltk 
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('wordnet')
stop = set(stopwords.words('english'))
from nltk.stem.snowball import SnowballStemmer
from nltk.stem import WordNetLemmatizer
stemmer = SnowballStemmer('english')
import nltk.tokenize as tokenize
import gensim
from wordcloud import WordCloud
from langdetect import detect

### Packages pour les plots
import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.tools as tls
from nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA
nltk.download('vader_lexicon')
sia = SIA()

### Packages ML
from umap import UMAP
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.externals import joblib
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from keras.models import Sequential
from keras.layers import Dense, BatchNormalization, Dropout
from keras.optimizers import Adam
from sklearn.preprocessing import Normalizer
import scikitplot as skplt
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
import os
os.chdir("/content/drive/My Drive")
! ls 
df_train = pd.read_csv('train.txt', sep="\t",header =None)
df_train = pd.DataFrame(df_train[0].apply(lambda x: x.split(None, 1)).tolist(), columns=['label', 'text'])
print('Shape of data :',df_train.shape)<jupyter_output>Shape of data : (3600000, 2)<jupyter_text>Notre analyse portera sur exactement 3.600.000 commentaires correspondant aux observations dans la base d'apprentisage et de 2 colones texte & label. II.1 Description des données<jupyter_code>df_train.describe(include='all')
df_train.info()
df_train.head(20)
df_train = df_train.drop_duplicates()
df_train['label'].unique()<jupyter_output><empty_output><jupyter_text>Il y a donc 2 classes à prévoir On obseve maintenant quelques pharses de manière complète<jupyter_code>print(df_train['label'][0])
df_train['text'][0]
print(df_train['label'][1])
df_train['text'][1]
print(df_train['label'][6])
df_train['text'][6]
print(df_train['label'][14])
df_train['text'][14]<jupyter_output>__label__1<jupyter_text>Remarque 1:Les textes étudiés sont  appriorie redigés en anglais. En outre, ils comportent pas mal de ponctuations et quelsues emoji.On peut apriori apres lecture dire que le label 1  correspond aux commentaires plutot négatifs alors que ceux du label 2 sont positif.  Rename label<jupyter_code>### Rename label
df_train['label_new'] = 'good'
df_train['label_new'][df_train['label']=='__label__1']='bad'
df_train.drop('label',axis = 1,inplace=True)
sns.barplot(np.unique(df_train['label_new']),df_train.groupby('label_new').count().values[:,0])<jupyter_output><empty_output><jupyter_text>La figure ci-dessus donne un appercu de la distribution de nos labels dans le data train. ( Distrubution équilibrée )  II.2 Création de nouvelles variablesUne fois cette premiere visualisation réalisé, on se demande maintenant si la création de nouvelles variables par simple transformation de nos données peut nous apporter des informations suplémentaires. C'est pour quoi, on s'intéresse maintenant à la distribution de la taille des textes par rapport aux différents labels.<jupyter_code>df_train['len'] = df_train['text'].apply(lambda x: len(x.split()))
plt.figure(figsize = (12, 7))
sns.distplot(df_train['len'][df_train['label_new'] =='good'], hist = True, label = "good",)
sns.distplot(df_train['len'][df_train['label_new'] =='bad'], hist = True, label = "bad")
plt.legend(fontsize = 10)
plt.title("Length Distribution by Class", fontsize = 12)
plt.show()<jupyter_output><empty_output><jupyter_text>On n'observe qu'une légère différence entre les deux distributions. Néamoins, pour la suite on considère que cette différence différence serait suceptible de nous apporter des informations. On decide donc de conserver cette nouvelle variable pour l'élaboration de notre modèle. II.2.A Certaines Poinctuations apportent elles des informations sur la distribution  des labels ?<jupyter_code>sf_train_exclamation_mark = df_train[df_train['text'].apply(lambda x: '!' in x)]
sns.barplot(np.unique(sf_train_exclamation_mark['label_new']),sf_train_exclamation_mark.groupby('label_new').count().values[:,0])
plt.title('Exclamation_mark (n =' +str(sf_train_exclamation_mark.shape[0])+')')
sf_train_question_mark = df_train[df_train['text'].apply(lambda x: '?' in x)]
sns.barplot(np.unique(sf_train_question_mark['label_new']),sf_train_question_mark.groupby('label_new').count().values[:,0])
plt.title('Question_mark (n =' +str(sf_train_question_mark.shape[0])+')')
sf_train_emotion_unhappy = df_train[df_train['text'].apply(lambda x: ':(' in x)]
sns.barplot(np.unique(sf_train_emotion_unhappy['label_new']),sf_train_emotion_unhappy.groupby('label_new').count().values[:,0])
plt.title('Emotion_unhappy (n =' +str(sf_train_emotion_unhappy.shape[0])+')')
sf_train_emotion_happy = df_train[df_train['text'].apply(lambda x: ':)' in x)]
sns.barplot(np.unique(sf_train_emotion_happy['label_new']),sf_train_emotion_happy.groupby('label_new').count().values[:,0])
plt.title('Emotion_happy (n =' +str(sf_train_emotion_happy.shape[0])+')')<jupyter_output><empty_output><jupyter_text>Remarque2:On observe qu'on a beaucoup plus de point d'exclamation dans les commentaires classé positivement et plus de point d'intérogation dans les messages classé négativement. Deplus, il y a une forte corélation entre l'apparution des emojis **happy** et le fait que le texte soit classé positivement et inversement pour **unhappy**.  II.2.B Sentiment Intensity Analysis (SIA)Dans cette parti une analyserons l'intensité des sentiments des différents commentaires par **VADER** (Valence Aware Dictionary and sEntiment Reasoner). Cette analyse ne sera éffectué que sur 10000 observations par souci de temps de calcul. Notons que la variable "label" dans la table ci-dessous est la prediction de la SIA<jupyter_code>train_sentiment = pd.DataFrame.from_records(df_train['text'][0:10000].apply(lambda x: sia.polarity_scores(x)))
train_sentiment['len'] = df_train['text'][0:10000].apply(len)
train_sentiment['label'] = 'good'
train_sentiment['label'][train_sentiment['neg']>train_sentiment['pos']] ='bad'
train_sentiment['real_label'] = df_train['label_new'][0:10000]
train_sentiment.head()
print(classification_report(df_train['label_new'][0:10000], train_sentiment['label']))
accuracy_score(df_train['label_new'][0:10000], train_sentiment['label'])<jupyter_output><empty_output><jupyter_text>L'accuracy trouvé ce dessus n'est pas tres performante. Cependant on conserve la variable car on estime que cette statistique nous apportera des informations dans notre model.  II.2.C Language Detection<jupyter_code>lang_detection = df_train.iloc[0:10000]
lang_detection['lang'] = df_train['text'][0:10000].map(detect,) 
lang_detection[lang_detection['lang'] != 'en']
lang_detection['lang'].value_counts()<jupyter_output><empty_output><jupyter_text>On vient ici d'identifier plusieurs commentaires qui ne sont pas en anglais et on décide de les retirer de notre jeu de données. Cependant, en terme de proportion cela ne représente que 0,31% des commentaires.On a décidé pour la suite on décide de ne pas retenir cette variable car la détection de langue demande enormement de temps de calcul pour une proportionalité relativement faible.<jupyter_code>### Clean memory
n_obs = 10000
df_train = df_train.iloc[0:n_obs]
del sf_train_emotion_happy, sf_train_emotion_unhappy, sf_train_question_mark, sf_train_exclamation_mark, lang_detection
gc.collect()<jupyter_output><empty_output><jupyter_text>III. PreprocessingPour cette étape, nous faisons les travaux comme suivant : * Tokenizer les text -- https://www.debuggex.com/cheatsheet/regex/python * Enlever les punctuation * Enlever les stopwords * Faire une lematisation avec part of speach -- https://www.machinelearningplus.com/nlp/lemmatization-examples-python/<jupyter_code>from textblob import TextBlob
def lemmatize_with_postag(sentence):
    sent = TextBlob(sentence)
    tag_dict = {"J": 'a', 
                "N": 'n', 
                "V": 'v', 
                "R": 'r'}
    words_and_tags = [(w, tag_dict.get(pos[0], 'n')) for w, pos in sent.tags]    
    lemmatized_list = [wd.lemmatize(tag) for wd, tag in words_and_tags]
    return " ".join(lemmatized_list)

### Function which define if a word is a number or not

def is_number(word):
    try: 
        x = float(word)
        return (x == x) and (x - 1 != x)
    except Exception:
        return False
    
### Tokenization using regular expression pattern which keep numbers, ?_mark, !_mark, :), :(     
tokenizer= tokenize.RegexpTokenizer(r'[0-9]*\.?[0-9]+|[a-zA-Z]+|[!]+|[:)]+|[:(]+|[?]+|[^[a-zA-Z]\s]+|[^[a-zA-Z][0-9]]')

### dictionary number to word
number2word = {'0':'zero','1':'one','2':'two','3':'three','4':'four','5':'five','6':'six','7':'seven','8':'eight','9':'nine','.':'point'}

### English Stemming 
stemmer = SnowballStemmer('english')

def text_process(mess):
    """
    Takes in a string of text, then performs the following:
    1. Remove all punctuation
    2. Remove all stopwords
    3. Returns a list of the cleaned text
    4. Keep ?, keep !
    5. Keep . for decimal numbers
    """
    mess = tokenizer.tokenize(mess.lower())
    
    word_list = []
        
    for word in mess:
        if (word ==':') or (word.startswith(')')) or (word.startswith('(')): 
            next
        elif word.startswith(':)') :
            word_list.append('emotion_happy')
        elif word.startswith(':(') :
            word_list.append('emotion_unhappy')
        elif word.startswith('?') : 
            word_list.append('question_mark')
        elif word.startswith('!') :
            word_list.append('exclamation_mark')
        elif (word not in stopwords.words('english')) and (word not in ':.%...') and (is_number(word)==False):
            word_list.append(lemmatize_with_postag(word))
        elif is_number(word)==True:
            b = [number2word.get(i) for i in list(word)]
            word_list += b
    return word_list


print('Pharse originale : \n')
print(df_train['text'][32],'\n')
print('Pharse transformée: \n')
print(' '.join(text_process(df_train['text'][32])),'\n')
print('Liste de mots (Tokenization): \n')
text_process(df_train['text'][32])
print('Pharse originale : \n')
print(df_train['text'][979],'\n')
print('Pharse transformée: \n')
print(' '.join(text_process(df_train['text'][979])),'\n')
print('Liste de mots (Tokenization): \n')
text_process(df_train['text'][32])<jupyter_output>Pharse originale : 

Great desk and a great buy! Thanks Amazon:): We bought this desk for my seven year old daughter to keep her lap top on and to do homework. It was not to disappoint. The desk is perfect size for her to sit and work. Very high quality and easy to put together. I would highly recommend this product to others? 

Pharse transformée: 

great desk great buy exclamation_mark thanks amazon emotion_happy bought desk seven year old daughter keep lap top homework disappoint desk perfect size sit work high quality easy put together would highly recommend product others question_mark 

Liste de mots (Tokenization):<jupyter_text>Pour la suite, nous ferons la modélisation sur seulement 10000 observations. On choisira le meuilleur modèle et l'addapte sur tout le jeu d'apprentisage, puis on sortira le résultat prévu sur la base de test.<jupyter_code>### These functions allow us to work in parallel
num_partitions = 10 #number of partitions to split dataframe
num_cores = 4 #number of cores on your machine

def vec_text_process(array):
    return array.apply(text_process)

def parallelize_dataframe(df):
    df_split = np.array_split(df, num_partitions)
    pool = Pool(num_cores)
    df = pd.Series(list(chain.from_iterable(pool.map(vec_text_process, df_split))))
    pool.close()
    pool.join()
    return df

df_train_tokens = parallelize_dataframe(df_train['text'][0:n_obs])<jupyter_output><empty_output><jupyter_text>IV. Création de la matrice des featuresNous allons transformer notre texte en matrice numérique pouvant etre implémenté dans différents algorithmes de machine learning pour faire de la classifacation.Il existe plusieurs méthode de transformation du texte. Le plus simple d'entre eux est le Bag-of-words ou en ligne on aura les différents commentaire et en colone chaque unique mot présent dans le texte analisé. IV.1 Train test splitting<jupyter_code>train_index = df_train.index.isin(sample(range(int(n_obs)), int(n_obs*0.7)))
X_train = df_train_tokens[train_index]
Y_train = df_train['label_new'][train_index].reset_index(drop=True)

X_test = df_train_tokens[~train_index].reset_index(drop=True)
Y_test = df_train['label_new'][~train_index].reset_index(drop=True)

### Clear memory
#del df_train, df_train_tokens
gc.collect()<jupyter_output><empty_output><jupyter_text>IV.2  Frequence d'apparution des mots<jupyter_code>### Create a dictionary 
df_train_dict = gensim.corpora.Dictionary(X_train)
### we draw word cloud on first 10000 reviews 
text = ' '.join(df_train['text'].str.lower().values[0:10000])
wordcloud = WordCloud(max_font_size=None, stopwords=stop, background_color='white',
                      width=1200, height=1000).generate(text)
plt.figure(figsize=(12, 8))
plt.imshow(wordcloud)
plt.title('Top words in review')
plt.axis("off")
plt.show()
### Clear memory
del df_train, df_train_tokens
gc.collect()
# fonction qui sort la dataframe de la fréquence
def most_frequent_words(dictionary) : 
    df_tokens = pd.DataFrame.from_dict(dictionary.dfs, orient='index',columns=['counts'])
    get_words = np.vectorize(dictionary.get)
    df_tokens['word'] = get_words(df_tokens.index)
    df_tokens.sort_values('counts',ascending=False,inplace = True)
    return df_tokens
df_tokens = most_frequent_words(df_train_dict)
print('The length of dictionary =', df_tokens.shape[0])
df_tokens.reset_index(drop = True, inplace=True)
df_tokens.head()
fig, ax1 = plt.subplots(figsize=(14,8))
g = sns.barplot(df_tokens['word'][0:50],y=df_tokens['counts'][0:50], ax=ax1)
plt.xticks(rotation=90,fontsize=10)
plt.title('The most 50 frequent words')

ax2 = ax1.twinx()
ax2.set_ylim(0,ax1.get_ylim()[1]/X_train.shape[0])
ax2.set_ylabel('counts / documents')
fig.tight_layout() 
plt.show()
df_tokens['counts'].describe()
print('Quantile 95% :', np.quantile(df_tokens['counts'], 0.95))
print('Quantile 97.5% :', np.quantile(df_tokens['counts'], 0.975))
print('Number of words appearing only 1 time =', sum(df_tokens['counts'] ==1 ))
print('Number of frequencies of the 10000th word =', df_tokens['counts'][9999])<jupyter_output>Quantile 95% : 40.0
Quantile 97.5% : 83.84999999999854
Number of words appearing only 1 time = 10583
Number of frequencies of the 10000th word = 2<jupyter_text>En ne prenant en compte que l'intensité d'apparution des mots sur le graphe cidessus, on se rend compte que la représentation n'est pas tres net et difficile à interpréter.Pour la suite, on va multiplier cette frequence par une variable permettant de définir le poid de chaque mots<jupyter_code>df_train_dict_1 = df_train_dict
df_train_dict_1.filter_extremes(keep_n=1000)
X_train_corpus  = [df_train_dict_1.doc2bow(doc) for doc in X_train]
X_train_visu = gensim.matutils.corpus2dense(X_train_corpus, num_terms=len(df_train_dict_1)).T
a = X_train_visu/ X_train_visu.sum(axis =0)[None,:]
plt.figure(figsize=(15,8))

plt.imshow(a[np.argsort(Y_train),:].T,)<jupyter_output><empty_output><jupyter_text>Le gaphique ci-dessus illustre la densité de mots dans la matrice de Bage of WordsNous decidons de garder 10000 mots les plus présentants dans la dictionnaire<jupyter_code>df_train_dict.filter_extremes(keep_n=10000)
X_train_corpus  = [df_train_dict.doc2bow(doc) for doc in X_train]
X_train_tfidf = gensim.models.TfidfModel(X_train_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_train_tfidf_full = gensim.matutils.corpus2csc(X_train_tfidf[X_train_corpus], num_terms=len(df_train_dict)).T
X_test_corpus  = [df_train_dict.doc2bow(doc) for doc in X_test]
X_test_tfidf = gensim.models.TfidfModel(X_test_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_test_tfidf_full = gensim.matutils.corpus2csc(X_test_tfidf[X_test_corpus], num_terms=len(df_train_dict)).T
df_train_dict.filter_extremes(keep_n=1000)
X_train_corpus_visu  = [df_train_dict.doc2bow(doc) for doc in X_train]
X_train_tfidf_visu = gensim.models.TfidfModel(X_train_corpus_visu, df_train_dict)
X_train_tfidf_visu = gensim.matutils.corpus2dense(X_train_tfidf_visu[X_train_corpus_visu], num_terms=len(df_train_dict)).T
b = X_train_tfidf_visu/ X_train_tfidf_visu.sum(axis =0)[None,:]
plt.figure(figsize=(15,8))
plt.imshow(b[np.argsort(Y_train),:].T)<jupyter_output><empty_output><jupyter_text>Le gaphique ci-dessus illustre le poid des mots dans la matrice de TF-IDF Pour la suite, on utilisera la matrice TF-IDF parce qu'on considère qu'elle est la version "scaling" du Bag of Words, et est plus addaptée pour la plupart des algorithme de ML  IV.3 Réduction de dimentionA cette étape, on souhaite réduire la taille de notre matrice. En effet, le bag of words selectione tout les mots uniques dans l'enssemble de commentaires ce qui peut entrainer avec un trop grand nombre de zero car certaint mots n'apparaissent que dans une poigné de commentaires.Deplus, une matrie de taille pleine peut entrainer des erreurs dans notre algorhytme c'est pour quoiil a été normale de se questioner sur la viabilité de conserver ces mots dans notre matricePour effectuer cette réduction, nous testerons différentes méthodes qui se prete bien au traitement de texte. Les quelques methodes implémenté sont : 1. Principal component analysis (PCA)2. Random projection (RP)3. Latent Dirichet Allocation (LDA) http://cs229.stanford.edu/proj2017/final-reports/5163902.pdf4. Uniform Manifold Approximation and Projection (UMAP) IV.3.1 PCA<jupyter_code>### For PCA, we try first with 200 components
p = time.time()
pca = PCA(n_components = 200)
X_train_tfidf_pca = pca.fit_transform(X_train_tfidf_full.A)

pca_analysis = pd.DataFrame(columns = ['Number of components','Cumulative of variance ratio'])
pca_analysis['Number of components'] = np.arange(1,201)
pca_analysis['Cumulative of variance ratio'] = np.cumsum(pca.explained_variance_ratio_)
pca_analysis<jupyter_output><empty_output><jupyter_text>Dans notre ACP on fait le choix  de 200  composante principale qui ne contribue que de 20% de la totalité de nos variable. à premiere vue, ce n'est pas très significatif mais pour des soucis de temps de calcul on accepte et regardera plus tard si on doit en récupérer plus.<jupyter_code>X_test_tfidf_pca = pca.transform(X_test_tfidf_full.A)
time_pca=time.time() - p<jupyter_output><empty_output><jupyter_text>IV.3.2 LDA<jupyter_code>p = time.time()
lda_model_tfidf = gensim.models.LdaMulticore(X_train_tfidf[X_train_corpus], num_topics= 20, id2word=df_train_dict, passes=2, workers=4)
X_train_lda_topic = lda_model_tfidf[X_train_tfidf[X_train_corpus]]
X_train_tfidf_lda = sparse.lil_matrix((len(X_train_tfidf[X_train_corpus]),20), dtype=np.float64)

for i in range(len(X_train_tfidf[X_train_corpus])):
    for j in X_train_lda_topic[i]:
        X_train_tfidf_lda[i,j[0]] = j[1]

X_test_lda_topic = lda_model_tfidf[X_test_tfidf[X_test_corpus]]
X_test_tfidf_lda = sparse.lil_matrix((len(X_test_tfidf[X_test_corpus]),20), dtype=np.float64)

for i in range(len(X_test_tfidf[X_test_corpus])):
    for j in X_test_lda_topic[i]:
        X_test_tfidf_lda[i,j[0]] = j[1]
        
del lda_model_tfidf, X_train_lda_topic, X_test_lda_topic
time_lda=time.time() - p<jupyter_output><empty_output><jupyter_text>IV.3.3 RP<jupyter_code>p = time.time()
rp_model_tfidf = gensim.models.RpModel(X_train_tfidf[X_train_corpus], id2word=df_train_dict, num_topics= 100)
X_train_rp_topic = rp_model_tfidf[X_train_tfidf[X_train_corpus]]
X_train_tfidf_rp = sparse.lil_matrix((len(X_train_tfidf[X_train_corpus]),100), dtype=np.float64)

for i in range(len(X_train_tfidf[X_train_corpus])):
    for j in X_train_rp_topic[i]:
        X_train_tfidf_rp[i,j[0]] = j[1]
        

X_test_rp_topic = rp_model_tfidf[X_test_tfidf[X_test_corpus]]
X_test_tfidf_rp = sparse.lil_matrix((len(X_test_tfidf[X_test_corpus]),100), dtype=np.float64)

for i in range(len(X_test_tfidf[X_test_corpus])):
    for j in X_test_rp_topic[i]:
        X_test_tfidf_rp[i,j[0]] = j[1]
        
del rp_model_tfidf, X_train_rp_topic, X_test_rp_topic

time_rp=time.time() - p<jupyter_output><empty_output><jupyter_text>IV.3.4 UMAP<jupyter_code>### n_components can be set between 1 to 100, but this technique consume a lot of time, so we choose only 20
p = time.time()
umap = UMAP(n_components=20, metric = 'cosine')
X_train_tfidf_umap = umap.fit_transform(X_train_tfidf_full.A)
X_test_tfidf_umap = umap.transform(X_test_tfidf_full.A)
del umap
time_umap = time.time()-p<jupyter_output><empty_output><jupyter_text>Par ailleurs, on rattache a cette matrice  l'analyse des sentiment et les variable supplémentaire crée précédemment.<jupyter_code>train_sentiment['len'] = Normalizer().fit_transform(np.array(train_sentiment['len']).reshape(1,-1)).reshape(-1,1)
sentiment = train_sentiment[['neg','neu','pos','compound','len']]
del train_sentiment

X_train_tfidf_full = sparse.hstack((sentiment[0:int(n_obs*0.7)], X_train_tfidf_full))
X_test_tfidf_full = sparse.hstack((sentiment[int(n_obs*0.7):n_obs], X_test_tfidf_full))

X_train_tfidf_pca = np.concatenate([sentiment[0:int(n_obs*0.7)],X_train_tfidf_pca],axis =1)
X_test_tfidf_pca = np.concatenate([sentiment[int(n_obs*0.7):n_obs],X_test_tfidf_pca],axis =1)

X_train_tfidf_lda = sparse.hstack((sentiment[0:int(n_obs*0.7)], X_train_tfidf_lda))
X_test_tfidf_lda = sparse.hstack((sentiment[int(n_obs*0.7):n_obs], X_test_tfidf_lda))

X_train_tfidf_rp = sparse.hstack((sentiment[0:int(n_obs*0.7)], X_train_tfidf_rp))
X_test_tfidf_rp = sparse.hstack((sentiment[int(n_obs*0.7):n_obs], X_test_tfidf_rp))

X_train_tfidf_umap = np.concatenate([sentiment[0:int(n_obs*0.7)],X_train_tfidf_umap],axis =1)
X_test_tfidf_umap = np.concatenate([sentiment[int(n_obs*0.7):n_obs],X_test_tfidf_umap],axis =1)

del sentiment
### Clean memory
del X_train, X_test
gc.collect()<jupyter_output><empty_output><jupyter_text>IV.3.5 Comparaison des méthodes de la reduction de dimension<jupyter_code>dimension_reduction_comparison = pd.DataFrame(columns = ['Method','Computing time','Number of new features'])
dimension_reduction_comparison['Method']=['Principal component analysis (PCA)','Random projection (RP)','Latent Dirichet Allocation (LDA)','Uniform Manifold Approximation and Projection (UMAP)']
dimension_reduction_comparison['Computing time'] = [time_pca, time_lda, time_rp, time_umap]
dimension_reduction_comparison['Number of new features'] =[200,20,100,20]
dimension_reduction_comparison<jupyter_output><empty_output><jupyter_text>V. Machine Learning model  V.1 Logistic Regression<jupyter_code>### On full data
p = time.time()
model_LR_full = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')
model_LR_full.fit(X_train_tfidf_full,Y_train)
Y_predict_LR_full = model_LR_full.predict(X_test_tfidf_full)
time_LR_full = time.time()-p
### On PCA data
p = time.time()
model_LR_pca = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')
model_LR_pca.fit(X_train_tfidf_pca,Y_train)
Y_predict_LR_pca = model_LR_pca.predict(X_test_tfidf_pca)
time_LR_pca = time.time()-p
### On LDA data
p = time.time()
model_LR_lda = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')
model_LR_lda.fit(X_train_tfidf_lda,Y_train)
Y_predict_LR_lda = model_LR_lda.predict(X_test_tfidf_lda)
time_LR_lda = time.time()-p
### On RP data
p = time.time()
model_LR_rp = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')
model_LR_rp.fit(X_train_tfidf_rp,Y_train)
Y_predict_LR_rp = model_LR_rp.predict(X_test_tfidf_rp)
time_LR_rp = time.time()-p
### On UMAP data
p = time.time()
model_LR_umap = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')
model_LR_umap.fit(X_train_tfidf_umap,Y_train)
Y_predict_LR_umap = model_LR_umap.predict(X_test_tfidf_umap)
time_LR_umap = time.time()-p<jupyter_output><empty_output><jupyter_text>V.2 Support Vector Machines Classifier<jupyter_code>### On full data
p = time.time()
model_SVM_full = SVC(kernel='linear', probability = True)
model_SVM_full.fit(X_train_tfidf_full,Y_train)
Y_predict_SVM_full = model_SVM_full.predict(X_test_tfidf_full)
Y_predict_proba_SVM_full = model_SVM_full.predict_proba(X_test_tfidf_full)
time_SVM_full = time.time()-p
### On PCA data
p = time.time()
model_SVM_pca = SVC(kernel='linear', probability = True)
model_SVM_pca.fit(X_train_tfidf_pca,Y_train)
Y_predict_SVM_pca = model_SVM_pca.predict(X_test_tfidf_pca)
Y_predict_proba_SVM_pca = model_SVM_pca.predict_proba(X_test_tfidf_pca)
time_SVM_pca = time.time()-p
### On LDA data
p = time.time()
model_SVM_lda = SVC(kernel='linear')
model_SVM_lda.fit(X_train_tfidf_lda,Y_train)
Y_predict_SVM_lda = model_SVM_lda.predict(X_test_tfidf_lda)
time_SVM_lda = time.time()-p
### On RP data
p = time.time()
model_SVM_rp = SVC(kernel='linear')
model_SVM_rp.fit(X_train_tfidf_rp,Y_train)
Y_predict_SVM_rp = model_SVM_rp.predict(X_test_tfidf_rp)
time_SVM_rp = time.time()-p
### On UMAP data
p = time.time()
model_SVM_umap = SVC(kernel='linear')
model_SVM_umap.fit(X_train_tfidf_umap,Y_train)
Y_predict_SVM_umap = model_SVM_umap.predict(X_test_tfidf_umap)
time_SVM_umap = time.time()-p<jupyter_output><empty_output><jupyter_text>V.3 K-Neighbors Classifier<jupyter_code>### On full data
p = time.time()
model_KNN_full = KNeighborsClassifier()
model_KNN_full.fit(X_train_tfidf_full,Y_train)
Y_predict_KNN_full = model_KNN_full.predict(X_test_tfidf_full)
time_KNN_full = time.time()-p
### On PCA data
p = time.time()
model_KNN_pca = KNeighborsClassifier()
model_KNN_pca.fit(X_train_tfidf_pca,Y_train)
Y_predict_KNN_pca = model_KNN_pca.predict(X_test_tfidf_pca)
time_KNN_pca = time.time()-p
### On LDA data
p = time.time()
model_KNN_lda = KNeighborsClassifier()
model_KNN_lda.fit(X_train_tfidf_lda,Y_train)
Y_predict_KNN_lda = model_KNN_lda.predict(X_test_tfidf_lda)
time_KNN_lda = time.time()-p
### On RP data
p = time.time()
model_KNN_rp = KNeighborsClassifier()
model_KNN_rp.fit(X_train_tfidf_rp,Y_train)
Y_predict_KNN_rp = model_KNN_rp.predict(X_test_tfidf_rp)
time_KNN_rp = time.time()-p
### On UMAP data
p = time.time()
model_KNN_umap = KNeighborsClassifier()
model_KNN_umap.fit(X_train_tfidf_umap,Y_train)
Y_predict_KNN_umap = model_KNN_umap.predict(X_test_tfidf_umap)
time_KNN_umap = time.time()-p<jupyter_output><empty_output><jupyter_text>V.4 Artificial neural network<jupyter_code>Y_train_NN = (Y_train=='good')
Y_test_NN = (Y_test=='good')
### On full data
p = time.time()
model_NN = Sequential()
model_NN.add(Dense(128, activation = 'relu',input_shape=(X_train_tfidf_full.shape[1],)))
model_NN.add(BatchNormalization())
model_NN.add(Dropout(0.2)) # to avoid overfitting
model_NN.add(Dense(1, activation = 'sigmoid'))
model_NN.compile(loss = 'binary_crossentropy', optimizer = Adam(lr = 0.001, decay = 0.01), metrics = ['accuracy'])
model_NN.fit(X_train_tfidf_full.A,Y_train_NN, validation_data=(X_test_tfidf_full.A, Y_test_NN), epochs = 5, batch_size = 128)
Y_predict_NN_full = np.where(model_NN.predict(X_test_tfidf_full.A)<0.5,'bad', 'good')
time_NN_full = time.time()
### On PCA data
p = time.time()
model_NN = Sequential()
model_NN.add(Dense(128, activation = 'relu',input_shape=(X_train_tfidf_pca.shape[1],)))
model_NN.add(BatchNormalization())
model_NN.add(Dropout(0.2)) # to avoid overfitting
model_NN.add(Dense(1, activation = 'sigmoid'))
model_NN.compile(loss = 'binary_crossentropy', optimizer = Adam(lr = 0.001, decay = 0.01), metrics = ['accuracy'])
model_NN.fit(X_train_tfidf_pca,Y_train_NN, validation_data=(X_test_tfidf_pca, Y_test_NN), epochs = 5, batch_size = 128)
Y_predict_NN_pca = np.where(model_NN.predict(X_test_tfidf_pca)<0.5,'bad', 'good')
time_NN_pca = time.time()
### On LDA data
p = time.time()
model_NN = Sequential()
model_NN.add(Dense(128, activation = 'relu',input_shape=(X_train_tfidf_lda.shape[1],)))
model_NN.add(BatchNormalization())
model_NN.add(Dropout(0.2)) # to avoid overfitting
model_NN.add(Dense(1, activation = 'sigmoid'))
model_NN.compile(loss = 'binary_crossentropy', optimizer = Adam(lr = 0.001, decay = 0.01), metrics = ['accuracy'])
model_NN.fit(X_train_tfidf_lda.A,Y_train_NN, validation_data=(X_test_tfidf_lda.A, Y_test_NN), epochs = 5, batch_size = 128)
Y_predict_NN_lda = np.where(model_NN.predict(X_test_tfidf_lda.A)<0.5,'bad', 'good')
time_NN_lda = time.time()
### On RP data
p = time.time()
model_NN = Sequential()
model_NN.add(Dense(128, activation = 'relu',input_shape=(X_train_tfidf_rp.shape[1],)))
model_NN.add(BatchNormalization())
model_NN.add(Dropout(0.2)) # to avoid overfitting
model_NN.add(Dense(1, activation = 'sigmoid'))
model_NN.compile(loss = 'binary_crossentropy', optimizer = Adam(lr = 0.001, decay = 0.01), metrics = ['accuracy'])
model_NN.fit(X_train_tfidf_rp.A,Y_train_NN, validation_data=(X_test_tfidf_rp.A, Y_test_NN), epochs = 5, batch_size = 128)
Y_predict_NN_rp = np.where(model_NN.predict(X_test_tfidf_rp.A)<0.5,'bad', 'good')
time_NN_rp = time.time()
### On UMAP data
p = time.time()
model_NN = Sequential()
model_NN.add(Dense(128, activation = 'relu',input_shape=(X_train_tfidf_umap.shape[1],)))
model_NN.add(BatchNormalization())
model_NN.add(Dropout(0.2)) # to avoid overfitting
model_NN.add(Dense(1, activation = 'sigmoid'))
model_NN.compile(loss = 'binary_crossentropy', optimizer = Adam(lr = 0.001, decay = 0.01), metrics = ['accuracy'])
model_NN.fit(X_train_tfidf_umap,Y_train_NN, validation_data=(X_test_tfidf_umap, Y_test_NN), epochs = 5, batch_size = 128)
Y_predict_NN_umap = np.where(model_NN.predict(X_test_tfidf_umap)<0.5,'bad', 'good')
time_NN_umap = time.time()<jupyter_output>Train on 7000 samples, validate on 3000 samples
Epoch 1/5
7000/7000 [==============================] - 1s 130us/step - loss: 0.7406 - acc: 0.5547 - val_loss: 0.6679 - val_acc: 0.6203
Epoch 2/5
7000/7000 [==============================] - 0s 43us/step - loss: 0.6741 - acc: 0.6049 - val_loss: 0.6482 - val_acc: 0.6343
Epoch 3/5
7000/7000 [==============================] - 0s 44us/step - loss: 0.6678 - acc: 0.6180 - val_loss: 0.6472 - val_acc: 0.6397
Epoch 4/5
7000/7000 [==============================] - 0s 46us/step - loss: 0.6622 - acc: 0.6221 - val_loss: 0.6475 - val_acc: 0.6363
Epoch 5/5
7000/7000 [==============================] - 0s 42us/step - loss: 0.6606 - acc: 0.6239 - val_loss: 0.6449 - val_acc: 0.6380<jupyter_text>VI. Comparaison des Resultats Nous allons comparer les temps de calculs des différentes méthodes et sélectionner celle qui nous donnera un meilleur rapport temps d'analyse et meilleur accuracy.<jupyter_code>def accuracy_score_vector(x) :
    return [accuracy_score(Y_test, predictor) for predictor in x]

perfomance_comparison =pd.DataFrame(columns = ['Machine Learning model','Reduction method','Accuracy score','Computing time'])
perfomance_comparison['Machine Learning model'] = ['Logistic Regression']*5 + ['Support vector machine']*5+['K-nearest neighbors']*5+['Neural network']*5
perfomance_comparison['Reduction method'] = ['Full matrix', 'PCA', 'LDA', 'RP', 'UMAP']*4
perfomance_comparison['Accuracy score'] = accuracy_score_vector([
    Y_predict_LR_full,Y_predict_LR_pca, Y_predict_LR_lda, Y_predict_LR_rp, Y_predict_LR_umap,
    Y_predict_SVM_full,Y_predict_SVM_pca, Y_predict_SVM_lda, Y_predict_SVM_rp, Y_predict_SVM_umap,
    Y_predict_KNN_full,Y_predict_KNN_pca, Y_predict_KNN_lda, Y_predict_KNN_rp, Y_predict_KNN_umap,
    Y_predict_NN_full,Y_predict_NN_pca, Y_predict_NN_lda, Y_predict_NN_rp, Y_predict_NN_umap])
perfomance_comparison['Computing time'] = [time_LR_full, time_LR_pca, time_LR_lda, time_LR_rp, time_LR_umap,
                                           time_SVM_full, time_SVM_pca, time_SVM_lda, time_SVM_rp, time_SVM_umap,
                                           time_KNN_full, time_KNN_pca, time_KNN_lda, time_KNN_rp, time_KNN_umap,
                                           time_NN_full, time_NN_pca, time_NN_lda, time_NN_rp, time_NN_umap]
perfomance_comparison['Computing time added dimension reduction time'] = perfomance_comparison['Computing time'] + np.array([0,time_pca, time_lda, time_rp, time_umap]*4)
perfomance_comparison['Estimated computing time on all training set'] = perfomance_comparison['Computing time added dimension reduction time']/n_obs*3600000
perfomance_comparison<jupyter_output><empty_output><jupyter_text>Apres analyse du tableau ci dessus,  nous sélectionnons deux model avec un fort accuracy qui sont :* La logistique regression* Le SVMces deux modèles ont été implémentés sur  la full matrice et la matrice réduite par PCA. Afin de déterminer lequel des deux modèle on choisira finalement  on observera la matrice de confusion et la courbe ROC  VI.1 Resultat de Logistic Regression VI.1.1 Tout le dictionnaire<jupyter_code>print(classification_report(Y_test, Y_predict_LR_full))
cm = confusion_matrix(Y_test,Y_predict_LR_full)
cm = cm.astype('float')/cm.sum(axis=1)[:, np.newaxis]
#Visualization of the Confusion matrix
ax=plt.subplot(title='Confusion matrix of the Logistic Model on full data set');
sns.heatmap(cm,annot=True, fmt='.4f', cmap='Blues', xticklabels=['good', 'bad'], yticklabels=['good', 'bad'])
ax.set_xlabel('LR Prediction');
ax.set_ylabel('True labels');
Y_predict_proba_LR_full = model_LR_full.predict_proba(X_test_tfidf_full)
skplt.metrics.plot_roc_curve(Y_test, Y_predict_proba_LR_full)
plt.show()<jupyter_output><empty_output><jupyter_text>VI.1.2 PCA data set<jupyter_code>print(classification_report(Y_test, Y_predict_LR_pca))
cm = confusion_matrix(Y_test,Y_predict_LR_pca)
cm = cm.astype('float')/cm.sum(axis=1)[:, np.newaxis]
#Visualization of the Confusion matrix
ax=plt.subplot(title='Confusion matrix of the Logistic Model on PCA data set');
sns.heatmap(cm,annot=True, fmt='.4f', cmap='Blues', xticklabels=['good', 'bad'], yticklabels=['good', 'bad'])
ax.set_xlabel('LR Prediction');
ax.set_ylabel('True labels');
Y_predict_proba_LR_pca = model_LR_pca.predict_proba(X_test_tfidf_pca)
skplt.metrics.plot_roc_curve(Y_test, Y_predict_proba_LR_pca)
plt.show()<jupyter_output><empty_output><jupyter_text>VI.2 Resultat de SVM VI.2.1 Tout le dictionnaire<jupyter_code>print(classification_report(Y_test, Y_predict_SVM_full))
cm = confusion_matrix(Y_test,Y_predict_SVM_full)
cm = cm.astype('float')/cm.sum(axis=1)[:, np.newaxis]
#Visualization of the Confusion matrix
ax=plt.subplot(title='Confusion matrix of the Support Vector Machines on full data set');
sns.heatmap(cm,annot=True, fmt='.4f', cmap='Blues', xticklabels=['good', 'bad'], yticklabels=['good', 'bad'])
ax.set_xlabel('SVM Prediction');
ax.set_ylabel('True labels');
Y_predict_proba_SVM_full = model_SVM_full.predict_proba(X_test_tfidf_full)
skplt.metrics.plot_roc_curve(Y_test, Y_predict_proba_SVM_full)
plt.show()<jupyter_output><empty_output><jupyter_text>VI.2.2  PCA data set<jupyter_code>print(classification_report(Y_test, Y_predict_SVM_pca))
cm = confusion_matrix(Y_test,Y_predict_SVM_pca)
cm = cm.astype('float')/cm.sum(axis=1)[:, np.newaxis]
#Visualization of the Confusion matrix
ax=plt.subplot(title='Confusion matrix of the Support Vector Machines on PCA data set');
sns.heatmap(cm,annot=True, fmt='.4f', cmap='Blues', xticklabels=['good', 'bad'], yticklabels=['good', 'bad'])
ax.set_xlabel('SVM Prediction');
ax.set_ylabel('True labels');
Y_predict_proba_SVM_pca = model_SVM_pca.predict_proba(X_test_tfidf_pca)
skplt.metrics.plot_roc_curve(Y_test, Y_predict_proba_SVM_pca)
plt.show()<jupyter_output><empty_output><jupyter_text>VII. ConclusionApres avoir choisi de ne conserver que deux modèles qui avaient l'accuracy la plus élevé, nous venons d'effectuer deux critères de comparaison afin de définir le model le plus performant et prenant en compte bien évidemment le temps de calcul nous permettant de l'appliquer sur toute la base de donnée.à Au vue des informations que nous apportent l'accuracy, la courbe ROC, et la matrice de confusion, nous choisissons pour la suite d'effectuer la prédiction du avec le modèle de **regression logistique**. VIII. Les codes pour modéliser tout le jeu d'apprentisage<jupyter_code>### Delete all variables 
del X_train_tfidf_full, X_train_tfidf_pca, X_train_tfidf_lda, X_train_tfidf_rp, X_train_tfidf_umap
del X_test_tfidf_full, X_test_tfidf_pca, X_test_tfidf_lda, X_test_tfidf_rp, X_test_tfidf_umap
del Y_train, Y_test
gc.collect()
### Reimport full train set
df_train = pd.read_csv('train.txt', sep="\t",header =None)
df_train = pd.DataFrame(df_train[0].apply(lambda x: x.split(None, 1)).tolist(), columns=['label', 'text'])
### Rename label
df_train['label_new'] = 'good'
df_train['label_new'][df_train['label']=='__label__1']='bad'
df_train.drop('label',axis = 1,inplace=True)
### SIA on all train set
train_sentiment = pd.DataFrame.from_records(df_train['text'].apply(lambda x: sia.polarity_scores(x)))
train_sentiment['Y_train'] = df_train['label_new']
train_sentiment['len'] = df_train['text'].apply(len)
train_sentiment['len'] = Normalizer().fit_transform(np.array(train_sentiment['len']).reshape(1,-1)).reshape(-1,1)
### Divide train test in 4 parties to avoid the memory error
df_train_tokens_p1 =  parallelize_dataframe(df_train['text'][0:1000000])
df_train_tokens_p2 =  parallelize_dataframe(df_train['text'][1000000:2000000])
df_train_tokens_p3 =  parallelize_dataframe(df_train['text'][2000000:3000000])
df_train_tokens_p4 =  parallelize_dataframe(df_train['text'][3000000:])
### Delete the full train set
del df_train
gc.collect()
### Join all sub train sets to one
df_train_tokens = pd.concat([df_train_tokens_p1, df_train_tokens_p2, df_train_tokens_p3, df_train_tokens_p4], axis = 0)
### Clear memory
del df_train_tokens_p1, df_train_tokens_p2, df_train_tokens_p3, df_train_tokens_p4
gc.collect()
### Create a dictionary on train test
df_train_dict = gensim.corpora.Dictionary(df_train_tokens)
df_train_dict.filter_extremes(keep_n=10000)
### Divide train test in 4 again to tokenize
df_train_tokens_p1 = df_train_tokens[0:900000]
df_train_tokens_p2 = df_train_tokens[900000:1800000]
df_train_tokens_p3 = df_train_tokens[1800000:2700000]
df_train_tokens_p4 = df_train_tokens[2700000:]
del df_train_tokens
gc.collect()
X_train_corpus  = [df_train_dict.doc2bow(doc) for doc in df_train_tokens_p1]
X_train_tfidf = gensim.models.TfidfModel(X_train_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_train_tfidf_full_p1 = gensim.matutils.corpus2csc(X_train_tfidf[X_train_corpus], num_terms=len(df_train_dict)).T
del df_train_tokens_p1
gc.collect()
X_train_corpus  = [df_train_dict.doc2bow(doc) for doc in df_train_tokens_p2]
X_train_tfidf = gensim.models.TfidfModel(X_train_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_train_tfidf_full_p2 = gensim.matutils.corpus2csc(X_train_tfidf[X_train_corpus], num_terms=len(df_train_dict)).T
del df_train_tokens_p2
gc.collect()
X_train_corpus  = [df_train_dict.doc2bow(doc) for doc in df_train_tokens_p3]
X_train_tfidf = gensim.models.TfidfModel(X_train_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_train_tfidf_full_p3 = gensim.matutils.corpus2csc(X_train_tfidf[X_train_corpus], num_terms=len(df_train_dict)).T
del df_train_tokens_p3
gc.collect()
X_train_corpus  = [df_train_dict.doc2bow(doc) for doc in df_train_tokens_p4]
X_train_tfidf = gensim.models.TfidfModel(X_train_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_train_tfidf_full_p4 = gensim.matutils.corpus2csc(X_train_tfidf[X_train_corpus], num_terms=len(df_train_dict)).T
del df_train_tokens_p4
gc.collect()
del df_train, df_train_tokens
gc.collect()
### Join all sub tfidf matrices
X_train_tfidf_full = sparse.vstack([X_train_tfidf_full_p1,X_train_tfidf_full_p2,X_train_tfidf_full_p3,X_train_tfidf_full_p4])
### add others statistic variables (SIA and length scaled)
X_train_tfidf_full = sparse.hstack([train_sentiment[['neg','neu','pos','compound','len']], X_train_tfidf_full])
### assign Y_train
Y_train = train_sentiment['Y_train']
### Import test set
df_test = pd.read_csv('test.txt', sep="\t",header =None)
df_test = pd.DataFrame(df_test[0].apply(lambda x: x.split(None, 1)).tolist(), columns=['label', 'text'])
df_test['label_new'] = 'good'
df_test['label_new'][df_test['label']=='__label__1']='bad'
df_test.drop('label',axis = 1,inplace=True)
### SIA on test set
test_sentiment = pd.DataFrame.from_records(df_test['text'].apply(lambda x: sia.polarity_scores(x)))
test_sentiment['Y_test'] = df_test['label_new']
test_sentiment['len'] = df_test['text'].apply(len)
test_sentiment['len'] = Normalizer().fit_transform(np.array(test_sentiment['len']).reshape(1,-1)).reshape(-1,1)
### tokenize test set
df_test_tokens =  parallelize_dataframe(df_test['text'])
del df_test
gc.collect()
### transform test set to TFIDF matrix
X_test_corpus  = [df_train_dict.doc2bow(doc) for doc in df_test_tokens]
X_test_tfidf = gensim.models.TfidfModel(X_test_corpus, df_train_dict)

### transform data to spare matrix in order to reduce memory
X_test_tfidf_full = gensim.matutils.corpus2csc(X_test_tfidf[X_test_corpus], num_terms=len(df_train_dict)).T
del df_test_tokens_p1, X_test_tfidf, X_test_corpus
gc.collect()
### Add others statistic
X_test_tfidf_full = sparse.hstack([test_sentiment[['neg','neu','pos','compound','len']], X_test_tfidf_full])
### Train Logistic regression model on all train set
p = time.time()
model_LR = LogisticRegression(random_state=0, solver='newton-cg', multi_class='multinomial')
model_LR.fit(X_train_tfidf_full,Y_train)
time_LR_full = time.time()-p
### Predict Y_test
Y_predict_full = model_LR.predict(X_test_tfidf_full)

Y_test = test_sentiment['Y_test']
accuracy_score(Y_test, Y_predict_full)
print(classification_report(Y_test, Y_predict_full))
cm = confusion_matrix(Y_test,Y_predict_full)
cm = cm.astype('float')/cm.sum(axis=1)[:, np.newaxis]
#Visualization of the Confusion matrix
ax=plt.subplot(title='Confusion matrix of the Logistic Regression on all train set');
sns.heatmap(cm,annot=True, fmt='.4f', cmap='Blues', xticklabels=['good', 'bad'], yticklabels=['good', 'bad'])
ax.set_xlabel('LR Prediction');
ax.set_ylabel('True labels');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12480791828617915, 'fraction_numerical': 0.021377564855825724, 'mean_word_length': 7.0052460202604925, 'pattern_counts': {'":': 4, '<': 136, '<?xml version=': 0, '>': 132, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '211370', 'license': 'MIT', 'n_tokens_mistral': 17338, 'n_tokens_neox': 16530, 'n_words': 3396, 'path': 'project/sentiment_analysis_project.ipynb', 'repo_name': 'Niangmohamed/SentimentAnalysis', 'size': '44275'}