<jupyter_start><jupyter_text>Sentiment analysis through Convolutional Neural Network **Context**In this project I try to developed sentiment analysis algorithims based on the structure proposed by <NAME> in ["Convolutional Neural Networks for Sentence Classification"](https://arxiv.org/abs/1408.5882) Here has been used a [Dataset](https://www.kaggle.com/cosmos98/twitter-and-reddit-sentimental-analysis-dataset) that was created as a part of the university Project On NLP and Sentimental AnalysisThese  Comments Were Made on Narendra Modi and Other Leaders as well as Peoples Opinion Towards the Next Prime Minister of The Nation ( In Context with General Elections Held In India - 2019).All the Comments From Reddit were Cleaned using Pythons re and also NLP with a Sentimental Label to each ranging from -1 to 1.- 0 Indicating it is a Neutral Tweet/Comment- 1 Indicating a Postive Sentiment- -1 Indicating a Negative Tweet/Comment<jupyter_code>import numpy as np
#from emo_utils import *
#import emoji
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split


from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import confusion_matrix, accuracy_score

from tensorflow.keras.preprocessing.text import Tokenizer
#from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.initializers import Constant
from keras.models import Model
from keras.layers import *
from keras.utils.np_utils import to_categorical
import re

import matplotlib.pyplot as plt
%matplotlib inline

from keras import regularizers

import os
from google.colab import drive
drive.mount('/content/gdrive')
home = "/content/gdrive/MyDrive/Corsi/cycle1/Text Mining and NLP - Prof. Marinai/Project"
url = "https://raw.githubusercontent.com/gobrac/datasets/main/Reddit_Data.csv" 
df = pd.read_csv(url)
#df.index = df['date']
df.head(5)
ax = sns.countplot(x="category", data=df)
plt.title('Countplot of Target')
plt.xlabel('target')
plt.ylabel('observations')
plt.show()<jupyter_output><empty_output><jupyter_text>CNN<jupyter_code>df.category.value_counts()<jupyter_output><empty_output><jupyter_text>The sentences present in the dataset seems to be already cleaned. However, we apply a standardization to remove anything that's not alphanumeric or a space. Then we lowercase what's left.<jupyter_code>def clean_str(in_str):
    in_str = str(in_str)

    # clean text
    in_str = re.sub(r'([^\s\w]|_)+', '', in_str)
    
    return in_str.strip().lower()


df['text'] = df['clean_comment'].apply(clean_str)
df<jupyter_output><empty_output><jupyter_text>Since we have a different proportion of observation for each category, we randomly down-sample each target category to the amount of the smallest one<jupyter_code>df_0 = df[df['category'] == -1].sample(frac=1)
df_1 = df[df['category'] == 0].sample(frac=1)
df_2 = df[df['category'] == 1].sample(frac=1)


# we want a balanced set for training against - there are 8277 `-1` examples
sample_size = min(df.category.value_counts())

data = pd.concat([df_0.head(sample_size), df_1.head(sample_size), df_2.head(sample_size)]).sample(frac=1)
data=df
data['l'] = data['clean_comment'].apply(lambda x: len(str(x).split(' ')))
print("mean length of sentence: " + str(data.l.mean()))
print("max length of sentence: " + str(data.l.max()))
print("std dev length of sentence: " + str(data.l.std()))
data.category.value_counts()
sequence_length = 1500
max_words = 20000 # this is the max number of words considered


tokenizer = Tokenizer(num_words=max_words, split=' ', oov_token='<unw>')
tokenizer.fit_on_texts(data['text'].values)



# this takes our sentences and replaces each word with an integer
X = tokenizer.texts_to_sequences(data['text'].values)

# we then pad the sequences so they're all the same length (sequence_length)
X   = pad_sequences(X, sequence_length)

y = pd.get_dummies(data['category']).values

# where there isn't a test set, Kim keeps back 10% of the data for testing, I'm going to do the same since we have an ok amount to play with
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

  


print("train set size " + str(len(X_train)))
print("test set size " + str(len(X_test)))<jupyter_output>train set size 29799
test set size 7450<jupyter_text>Model 1: Random embeddings ![SCHEMA.jpg](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/4gIoSUNDX1BST0ZJTEUAAQEAAAIYAAAAAAIQAABtbnRyUkdCIFhZWiAAAAAAAAAAAAAAAABhY3NwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlkZXNjAAAA8AAAAHRyWFlaAAABZAAAABRnWFlaAAABeAAAABRiWFlaAAABjAAAABRyVFJDAAABoAAAAChnVFJDAAABoAAAAChiVFJDAAABoAAAACh3dHB0AAAByAAAABRjcHJ0AAAB3AAAADxtbHVjAAAAAAAAAAEAAAAMZW5VUwAAAFgAAAAcAHMAUgBHAEIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABvogAAOPUAAAOQWFlaIAAAAAAAAGKZAAC3hQAAGNpYWVogAAAAAAAAJKAAAA+EAAC2z3BhcmEAAAAAAAQAAAACZmYAAPKnAAANWQAAE9AAAApbAAAAAAAAAABYWVogAAAAAAAA9tYAAQAAAADTLW1sdWMA<KEY>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAf//Z)<jupyter_code>embedding_dim = 50 
num_filters = 5

inputs = Input(shape=(sequence_length,), dtype='int32')

# use a random embedding for the text
embedding_layer = Embedding(input_dim=max_words, 
                            output_dim=embedding_dim, 
                            input_length=sequence_length,
                            trainable=True)(inputs)

reshape = Reshape((sequence_length, embedding_dim, 1))(embedding_layer)

# relu activation
# The convolution window acts on the whole 50 dimensions 
conv_0 = Conv2D(num_filters, kernel_size=(3, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape) #Regularizer to apply an L2 penalty on the layer's kernel
conv_1 = Conv2D(num_filters, kernel_size=(4, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape)
conv_2 = Conv2D(num_filters, kernel_size=(5, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape)

# perform max pooling on each of the convoluations
maxpool_0 = MaxPool2D(pool_size=(sequence_length - 3 + 1, 1), strides=(1,1), padding='valid')(conv_0)
maxpool_1 = MaxPool2D(pool_size=(sequence_length - 4 + 1, 1), strides=(1,1), padding='valid')(conv_1)
maxpool_2 = MaxPool2D(pool_size=(sequence_length - 5 + 1, 1), strides=(1,1), padding='valid')(conv_2)

# concat and flatten
concatenated_tensor = Concatenate(axis=1)([maxpool_0, maxpool_1, maxpool_2])
flatten = Flatten()(concatenated_tensor)

# do dropout and predict
dropout = Dropout(0.5)(flatten)
output = Dense(units=3, activation='softmax')(dropout)<jupyter_output><empty_output><jupyter_text>![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEU<KEY>UDP12tonWOaoscE2afb5GlJhSXhWWF2FaaAM6boaFFDARee7eo0AedN+ZtvgFq1AFltVVLDhirkevfWeP58YPx44PBhjfmTBBxEoFILOM92yM8A9o8ECk7p3qRbVMgdmqxxXrpnbr4mARIgARIgARIgARIIYQL+EXAuAOk/x6J+lAivGLw+cge8FBuylvXC09FyPBqN+q+Bl55D7sbBaFyjKb5Jdk9OnL9jJJq5eqnFtJ5QeBhpxjx0rheFyMgo1O08v1AvOK3SCcR//HiRHnDu9srGz51quY53QDxXQUVsbCymTp3qRuSQVxRw5g3BVVAN2TldwHnf1qBBQFQUsGaNHnniCRVyMm+cpHnzgClTgLQ0jfmTBCqQAAVcMfBFwqV8DKy820PI3Qqsuh/I2l5MAe4iARIgARIgARIgARIIJQJ+FXDImI/O9UWIRaK6tyxDFhb3eA7PNXoa0ZGRiH66BxZ7jhtFPraPfB013/gWuzzMXf6eiXjT1Yut5tuTC/dKy9+GEU21113M6yOxw6OcNkIOlvV6ElFeQ1DdDUQB52YBcA44Txrle80hqOXjZed2tFgNNQFnQ7V/ywqqDz0EbNigex5/XIWczBsnSYTczJlARuHew3qQP0kgsAQo4HzgK9Jt0UXA4kvcQm7b28CRWe4hrD6chllIgARIgARIgARIgAScQcC/Ag4ZmN+5vgqvmu9i6iGPBRIyFuLTFzpg+pIv0Vh6wUXVQ6f4NJzJkZ+C0c1qocXYVK+ecxnYOnckBg/7Dsv2ePWZy9+Bkc1UwEW/9BU2uTvOuehSwJXnMaOAKw+twnkp4Arz8DWigCtKqsQ54IpmLd+eTz4B7r0X2LJFy8XEqJAbMUJjEXKyZQe6K3D5qs3c4UmAAs7Hds07DqQv1szpy3R4qqeQy+KqyD6SZDYSIAESIAESIAESqHACfhZwQMa8zq5hqI+jw2z3UKeMBV3xQse5SMvZhKEvR1u95Gq/Pw0HXQYuf8coNKvVAuNSi3RjKwop9xi2LZ6MQR+9iv9V1zneol8eSgFXlFS59lDAlQtXocwUcIVw+BxQwBVFFTAB532pdu2AO+4Adu3SI488okJu3DiNRcbJgg+5Rf6y4X0mxiRQbgIUcOVGBmTvBna0B3Z20sLHE1XIra9tcDIWIQESIAESIAESIAESCDYBvws4nJiHT1zzstXvvAA6uOkE5nV+EZ/8fBxAPpK/eRUx1rDQ1hhvCbd8bBvxGmq1GoeS/VsW9q2djdGftcPrjZuhw6DJWJy8HqNauIagNvkayUXcHXvAleeBooArD63CeSngCvPwNaKAK0oqaALO+9KtWwM33eReRfWBB1TIff+95pRFHVauBArO9Fv2PgNjEvCZAAWcz6hKzrhvGLDwd8CW5prn+Apg8/NA2sKSy/AICZAACZAACZAACZBAhRHwv4DDCfwcW88ahhr1dC8sk9FM6fHo9GI3JLimGsrfPhKvWwsrxKDp8K3Iz9+Kb5rWQqvxu72GnwqXAhxbPxk93ngSMVG18HK3qdiU5jJt+SkY5RqCGvPaCGyjgDurB4kCzhwfBZwZOwq4otwqTMB5V+XVV4HrrweOHNEjd9+tQi4uTuMFC4D164HTp71LMiaBMglQwJWJyLcMuceAU/s0r4i4hAhgyxsaZ+8EOETVN47MRQIkQAIkQAIkQAJBIBAAAQec+KkT6spqqNEN0X91DtLmdETjHovdq5Tmp2BMc9fcbS8MwpqNX+OVWq0xYU8Rg4bsTSPxRm1dOfXZbvNx2LPzheciDM3HYGeR4uwBV55niAKuPLQK56WAK8zD14gCrigpxwg4z6pJr7dGjYCrrwaOS09mAFWrqpD76SeNFy4Etm7V1/xJAmUQoIArA5DJ4ZObge3vASdWa2lbyO3ubXI2liEBEiABEiABEiABEvAzgYAIOBz/CZ3qymqo0Wg8eBniOryE3suyPKqej9SxLVBdhqFGP4VWrV9ArSKrpkr2E/i5U11d1CHmDXzruTyqHM7/FcOauIagthqPov6ufAJurmcVPWrrt5dz5ugX1ogI4L77/HZaf52IAs6cJAWcGTsKuKLcHCngvKsp88I98QRw5ZVApms56xtv1M+3xa4J4xctAnbv9i7JmAQsAhRwQXgQkl8FFlYBZK44SSLnfn0JEFHHRAIkQAIkQAIkQAIkEHQCgRFwOI6fXOIsulETNGn8OVZ6LayXv2ciWtfQBRQiI2ug9YQ9RYef5m7CkMa6YENknU8w3+scyN2Iwa7jMa0nYu/Z9ICr/h6mHfPsXheAtqCAM4bqaFEDgALOrGkd3a7x8WeEefZtt2Hr1q1B2YYOHYqIiAg8+OCDZlAropRIuNq1gcsvB06d0hpcc43yk3njJC1dChw8qK/5s9IToIAL0iOQe1QvVJADyOqpMkRVVlOVdGQGIMNUmUiABEiABEiABEiABIJCIEACDjge7xqGGhmNRv3XIMf7dvL3YfLbj1mroUZWb4lxu4vYMyB/G4Y31R5ukdXfxuQDhQVZ7s7v8E4d6WkXiZgSesAt7fGk9qCr8T6muRdlddUmBwu76Xx1kdHPoHdigLvAUcB5PwU+x44WNRRwPrejd0ZHt6uHgNt31VWIjY0NytaoUSNLwN1yyy3euJwf2ws0yDDVqCgVcnl5Ok/cFVeokNuwQe8jMRE4dsz598QaBoQABVxAsJZ+0oxfgJ2dNU9BtkvInQOcdL0nTxf+P1bpJ+NREiABEiABEiABEiCB8hIImIDD8Xh0FDkW/SK+WJ9bTL0KcGDKu6gp8qzY+dukSA7WffEiomWoamQMXug6E9syCoCCLOxeOgofNmuNli+qoIuq0wbjtrqGQrmulnskCQNcQ1Qjo+rh7RErsb+QY/MYChsZierPvIv+Y6cibtpkjPpiEKYlF1fvYm7F110UcL6SKpLP0aKGAq5Ie/m6w9Ht6iHgDl93Hb755pugbG+++aYl4O644w5fMTo3X77rDyuykMNDDwEi4UTSyXbJJcC55wLbtmn9V6wAMlwr9Tj3jlgzPxGggPMTSNPTnNoDbKgHrHB9zhScAhKvB2TYav5J07OyHAmQAAmQAAmQAAmQQCkEAifgcBzxHesg5qUh2FSCxyo4NA3v14rBq98kFx1+6qp0wYlfMPKtJ1wSLhKRMTVR+7EaeKLF55i9PRNHFnRHQ2tFVZlPrhbqPv0evkvNwrLez6BGtD3E1f07qtb7mOaxkkPBkQT0aFRde+JZoi8SkVG10eidQZhfdExrKSh9OEQB5wOk4rM4WtRQwBXfaD7sdXS7egg43HOPD3fjnywhMQec6a3K3HGS9u0D7r0X+MtfNJb9F18MnH8+8Ntvum/1aiDbe94BPcSfoU+AAs4hbVjgek8eidPhqUmunrci5FJ7AKdc70eHVJfVIAESIAESIAESIIFQJhBAAedHLPnp2LFsFiaPG4fxk2cgYd1eZJ4ZKZGPY8kLMW3iOIyfMhdJO9JQgu8rsUIFGTuRNPsHTBg7DhOnxiMpJb1EIVjiSXw5QAHnC6Vi8zha1FDAFdtmvux0dLtSwPnShGeXxxZsu3YB0uPv+uv1fDKP3IUX6nbokO5btw6wBd7ZXZWlHUCAAs4BjeBdhRMrgaNzde/BSS4hV1VjEXUybJWJBEiABEiABEiABEjAmEBoCDjj23NYQQo44wZxtKihgAvPdqWAM25X44InXUPftm4FbrpJNzmZ7L/gAu0ll56up9+0SYeyGl+MBSuSAAVcRdL34drpi4H1jwOpvTSzCLnFfwJ2dfWhMLOQAAmQAAmQAAmQAAkUR4ACrjgqgdpHAWdMlgLOGB2WL19uLR4wdepU85MEqKSj25UCLkCtXo7TymIOkmThhmuvBW67TeO0NOC884BLLwVkBVZJIu1On9bX/Ol4AhRwjm+iwhXc0kx7xO36VPcfjQdSewKn9hXOx4gESIAESIAESIAESKBEAhRwJaIJwAEKOGOojhY17AEXnu1KAWfcrgErePSonnrVKl3Q4Z//1PjwYeCcc4CrrwZyXGtup6QErBo88dkToIA7e4ZBP8Px5UDOAb2s9I5LiAB2ddNYFnWQeeOYSIAESIAESIAESIAESiRAAVcimgAcoIAzhkoBZ4yOPeBM0VHAmZILXjl7frilS4H/+z/gwQf12nv3AhERwN/+BtgrsdqLOwSvdrxSKQQo4EqBEwqHZEjquppA9m6trQi5JZcCR+eEQu1ZRxIgARIgARIgARKoEAIUcMHETgFnTJsCzhgdBZwpOgo4U3IVU66gANi/X689b57OF/fooxpLbzgRcnfe6a7bwYPu13wVdAIUcEFHHrgLFuQASVWBhHPdQm5bG2B3HyAvLXDX5ZlJgARIgARIgARIIMQIUMAFs8Eo4IxpU8AZo6OAM0VHAWdKzhnlZMXUPXu0LjNm6Iqq1atr/OuvKuQeekhjmTvu2DFn1LuS1CIYAq5JkyY455xzcNFFF+GKK64o1/a73/0OERER+OMf/1iucr5e57LLLrPqdv7554dPi2es13vJ2Q8knFdYyKUvBWQlVSYSIAESIAESIAESqMQEKOCC2fgUcMa0KeCM0VHAmaKjgDMl58xy2dnAbtdwue+/B0R81KmjdV23Djj3XMAWdNKbLiPDmfcRJrUKhoBr0KCBJdFEpDl5C5Mmdd9GQTZwYI6cLoQAACAASURBVAywo53us4Tc+cDSK4B8vq/coPiKBEiABEiABEigshGggAtmi1PAGdOmgDNGRwFnio4CzpRcaJQTwbZrl9b1229VwD31lMYrVqige+IJjWUeORF4TH4jEAwB17t3b0u8Pfnkk9i2bZujtsTERKtu0tMu7FP6MmD5jcAvMXqrspDDqvuBPQPD/tZ5gyRAAiRAAiRAAiTgSYACzpNGoF9TwBkTpoAzRkcBZ4qOAs6UXGiWkyGoO3dq3YcN0yGqzz2n8eLFgIiSF17QWIScDHFlMiYQDAE3aNAgS3K9/PLLxvUMVMEDBw5YdZPhsZUiyTDv3MN6qzI3nKygekbIHQIOTwNO51UKFLxJEiABEiABEiCBykuAAi6IbX9g9Gj9UhcRgYPXXYeBAwc6auvVqxdiY2PRvXt3R9VLOPXv39+qm9TPadykPl27drXq16dPH8fVj+1q9j6b2rLlmffrsRtuCNonRVxcnPXFvFq1akG7Ji9UDIEDB4AdO/RA//76LNgiRxZ5+MMfgGbN9LgIORm2yuQzgfz8fOs5l6GhgfrctIegUsD53CzByZh3Atg3HDgar9ezhdzGpzUWWcdEAiRAAiRAAiRAAmFIgAIuiI26d8SIM1/o9/7lL2eEkkglbmTAZ8BZz8DoF1888349fN11QfukoIALGuryXUjmj7OFXPfu+mzYAm7WLOCSS4B33tFzioyjRCiVb15e3hkB99FHHwXk38DatWtb16CAK7UpKv7g3iFA4nU6b5zUZvfnOkT18PSKrxtrQAIkQAIkQAIkQAJ+JEAB50eYZZ0qZ/r0M1/oc+68E3v27HHUNmfOHOtL0Lhx4xxVL+G0detWq25dunRxXN2kfoMHD7bqJ/P6OK1d586dy3Y1eK8dHj/+zPs19x//KOvt7bfjFHB+QxnYE23dCmzfrtfo2FGflTff1HjqVOCKK4AOHTRm77gibeE5BFXmZwvE56b0TJYedhRwRfA7b8dpkdauIair/6VDVGUhB0lHZgJHZwOSh4kESIAESIAESIAEQphAxQu4rP1Y/9N3GP5lP/T5/AuMnLEOh/NDmGhpVecccKXRKfUY54ArFU+pB5cvX24JuKkiBRyWHN2unAPOYU+Lw6uzYYNbyLVtq0JOfkuaNAm49lqgWzeN2TsOngIuMzNTufj5J+eA8zPQYJ0uLx3YNwzIdz0XK+90CblxWoP8k8GqCa9DAiRAAiRAAiRAAn4lUIECrgDH109Ah8aN0X7YcHR6LhqRkZGIjKyB10Yk+/UmHXMyCjjjpnC0qAEwfPhwS3JJTz2nJQo4wxahgDMEx2LWfHCrVrmFXKtWKuQ++kjhjBkD/O1vwOefV1pYFHCVbBEG0ye9IBfY0R5Iquoh5O4CVv8byHTev7emt8lyJEACJEACJEAClYNAhQm47ORxeKtuNBp0X4QTKMDRhO5oGCMCLhoN+iSFJ30KOON2pYAzRsdVUE3RUcCZkmM5bwJ5ecDSpe455Jo2VSH3ySea85tvgNtvBwYP9i4ZtjEFHAWc0cOdvQtY+Dtg0cWA3RNuRztd0IFDVI2QshAJkAAJkAAJkEDwCFSMgMvfifGtayEyqj66JmS57jYHuxePx9Bhk7HqYJiOQaWAM36yKeCM0VHAmaKjgDMlx3JlEcjOBmQl1ZQUzfnCCyrkevTQeMgQ4N57gZEjyzpTyB6ngKOAM354c48Bx+Zr8ROrdXiqp5ATScdEAiRAAiRAAiRAAg4kUCECLnfjYDSOjkRkTDOMSglT2VZcY1PAFUfFp30UcD5hKjYTh6AWi6XsnRRwZTNiDv8QyMgAZCXVnTv1fA0aqJDr21fjQYOAf/8bmDDBP9dzwFko4Cjg/PIYZu8Gtr8HSC84SSfWqJBbX9svp+dJSIAESIAESIAESMCfBCpAwOVj97iWiJH53qq/i6lHK9GqVhRwxs8uBZwxOvaAM0VHAWdKjuXOlsCxY4AsmpKaqmeqV0+F3BdfaNyvH/Df/2qes71WBZWngKOAC8ijt284sLAK8GsTPX3GOuDXl4D0xQG5HE9KAiRAAiRAAiRAAuUhUAECLger+jZEtAi4Gu0w40R5qhvieSngjBuQAs4YHQWcKToKOFNyLOdvAgcPAhMnAr/9pmd+7DEVcl9/rfFnnwE1a2ovOn9fO0Dno4CjgAvQowXkHgGkZ5ykbW9pjzhbyJ36Dch29TQNWAV4YhIgARIgARIgARIonkCFCLhlvZ5ClCXgPsCMjOIr5t5bgKz9m7E8IR4zps9AfEISkg/a88a5c515VZCLjKP7sDN5PVYtXYD4+DXY7xrlmp+WgqR5szF36TakVcTIVwq4M81U3hcUcOUl5s7PIahuFuV6RQFXLlzMHEQCu3cDo0YBe/fqRatVUyEnq6tK6tkT+N//gPmuebJ0r6N+UsBRwAXlgTy5AdjWBkhfqpfb9jaQcA6Q6ppvMSiV4EVIgARIgARIgARIQAkETcBlz+uM+rVqombNmqhhrXaqK57WqKn7atashUZ9k5BzpmUKcPzXGejfpiFqVf8fmrbvgX4DPke3dxujTkxtPP/+YPyc4iXi8lMwull1RIrcs7eaHyM+6yjWjvsYL9aOdu2vgRc+X4agd76jgDvTuuV9QQFXXmLu/BRwbhblekUBVy5czFyBBLZvB4YOBaSnnCSZLy4iApg0SeNPPwWefVZXYtU9Ff6TAo4CrkIewl9fARLOB9IW6OVTPgaSXwVObq6Q6vCiJEACJEACJEAClYtA0ARcfupSTJn8Hb77biI+b1lHe8DFNEG3cbJPtsmIW7Uf2jGtAIeW9Mdrj0chqs4bGLrqGNwzxeViT3xXPB8Tiah6LfH1qjSPYwByjyNlXh80qeGScDXa4PP+b6NVjylYu20y2rr2RzfojSS37QtOq1PAGXOmgDNGxyGopugo4EzJsVxFE9i0CRg4EDhyRGtyzz0q5GReOUldugCNGwMrV2pcAT8p4CjgKuCx00vmHAROnwZO5wFLL9chqraQOxoPyDBVJhIgARIgARIgARIIAIGgCTh33XOwrNeTpQ5Bzd81Ge/UiUJkZE20HJviknLuMwDHsbhHA2seuegnO2LOAbee01xpmPFBzTO93V7un4T0AqDgaBza1VIxF/PK10gO9jBUCjjPRizXawq4cuEqlJk94Arh8D2ggPOdFXM6m8Dq1YDME5eWpvW8/XYVcrLyqqRPPgFefx1Yt07jIPykgKOAC8JjVvYlTqwEUj5SIVeQ6xJy5wIZa7WsiDomEiABEiABEiABEvATAQcKuONY0MUl6B5vj5klrJKas6Y/novWYawvDFzrMXRVyOQgoWtdl+Rrj5lnxprmYOe8r9Gz62cYv+po4Z5zfgJa6mko4ErFU9pBCrjS6JR+jAKudD4lHqWAKxEND4Q4gWXLgK5dgQzXJKw33aRCbt48vbFOnYDWrYHNgRuWRwFHAee4d1HOfmD948Dym9xCLulmYEtzIL/MCYsddzusEAmQAAmQAAmQgPMIOE/Apc9BhzquXmotx2F3Sb3UMn9GJ1e+6Of6Y02h4aQeAk7mgMt2CHgKOOOGoIAzRschqKboKOBMybFcqBGQxRo+/hjIdv1jef31KuQWL9Y7kWNt2wIy15yfUrgIuEceeQS33HILnn32WTRt2tTn7fnnn0dERATOO+88n8uU5/ySt3r16lbd3nnnHT+1WiU5TcEpvdGjc3R4auIN7iGruz8HTu2rJCB4myRAAiRAAiRAAv4m4DgBl7O6HxpaPdsiUafzApTozvJ/xbAmMTrMtPpbmLTfcxgqBZzJg5KYmIjY2FhMtecJMjlJgMpQwJmDZQ84Q3YUcIbgWCykCciQOxma+sEHQG6u3spVV6mQW7FC4w4dgI8+AlJTjW81XARclSpVLJEmMs2p23333WfcTpW+4PFE4EicYjg8TYXc8r9pfDofsGVdpQdFACRAAiRAAiRAAr4QcJyAy4r/GDWtFUyj0aCP56qoXreTvwcTWrsEXHQTDN3s+qJgZaOA86LlU0gB5xOmYjMNHz7ckpdbt24t9nhF7qSAM6RPAWcIjsXCikBBASB/lHn3XSA/X3sCXXqpCjl7zjgRcrKww969Pt96uAi4v/zlL5Z4e/vtt9G3b19HbY899phVt4YNG/rcLsxYCoG0RcC6x4Cdn2gmEXJLLgV2dS2lEA+RAAmQAAmQAAmQgJuA8wTc3A54zBJwUXiqd6LX3G7uiiN/N8a1dAm4mKb4ZqvnWFUKOA9SPr+kgPMZVZGMFHBFkPi0w9E9GyngfGpDZqpkBPLygIkTgbffVhkn8R//<KEY>)<jupyter_code>model = Model(inputs=inputs, outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
tf.keras.utils.plot_model(model,  show_shapes=True)

batch_size = 50 
history = model.fit(X_train, y_train, epochs=20, batch_size=batch_size, verbose=1, validation_split=0.2, shuffle=True)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
y_hat = model.predict(X_test)
confusion_matrix(list(map(lambda x: np.argmax(x), y_test)), list(map(lambda x: np.argmax(x), y_hat)))
accuracy_score(list(map(lambda x: np.argmax(x), y_test)), list(map(lambda x: np.argmax(x), y_hat)))
from sklearn import metrics
print(metrics.classification_report( list(map(lambda x: np.argmax(x), y_test)),  list(map(lambda x: np.argmax(x), y_hat)), digits=3))<jupyter_output>precision    recall  f1-score   support

           0      0.721     0.103     0.181      1673
           1      0.969     0.963     0.966      2592
           2      0.674     0.980     0.799      3185

    accuracy                          0.777      7450
   macro avg      0.788     0.682     0.648      7450
weighted avg      0.787     0.777     0.718      7450<jupyter_text>Saving the model:<jupyter_code>#!mkdir -p home + saved_model
model.save(home + 'saved_model/model_1') 
model = tf.keras.models.load_model(home+'saved_model/model_1')

# Check architecture
model.summary()<jupyter_output><empty_output><jupyter_text>Model 2: glove with trainable embeddings creating a dictionary for the embadded words matrix<jupyter_code>embeddings_index = {}

f = open(os.path.join(home+"/archive/glove.6B.50d/full/glove.6B.50d.txt"))


for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = coefs
f.close()

print('Found %s word vectors.' % len(embeddings_index))
word_index = tokenizer.word_index
print('Found %s unique tokens.' % len(word_index))

max_features = 20000 # max number of words considered
num_words = min(max_features, len(embeddings_index)) +1
print(num_words)<jupyter_output>20001<jupyter_text>we run through all the words in our tokenizer and look for them in the glove model. If they exist we add them to our embedding matrix (ie our embeddings) if they don't then we assign a random vector<jupyter_code># create a matrix of zeros as embedding matrix
embedding_matrix = np.zeros((num_words, embedding_dim))
present=0
absent=0
# for each word in out tokenizer lets try to find that work in our glove model
for word, i in word_index.items():
    if i > max_features:
        continue
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        # if we find the word - add that words vector to the matrix
        embedding_matrix[i] = embedding_vector
        present+=1
    else:
        # if doesn't exist, assign a random vector
        embedding_matrix[i] = np.random.randn(embedding_dim)
        absent+=1
print(f' {present} word have been found, {absent} have not')<jupyter_output>17856 word have been found, 2144 have not<jupyter_text>We apply a CNN with the following characteristics<jupyter_code>embedding_dim = 50
num_filters = 5



inputs_2 = Input(shape=(sequence_length,), dtype='int32')
embedding_layer_2 = Embedding(num_words,
                            embedding_dim,
                            embeddings_initializer=Constant(embedding_matrix),
                            input_length=sequence_length,
                            trainable=True)(inputs_2)

reshape_3 = Reshape((sequence_length, embedding_dim, 1))(embedding_layer_2)

# relu activation
conv_0_3 = Conv2D(num_filters, kernel_size=(3, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape_3)
conv_1_3 = Conv2D(num_filters, kernel_size=(4, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape_3)
conv_2_3 = Conv2D(num_filters, kernel_size=(5, embedding_dim), activation='relu', kernel_regularizer=regularizers.l2(3))(reshape_3)

maxpool_0_3 = MaxPool2D(pool_size=(sequence_length - 3 + 1, 1), strides=(1,1), padding='valid')(conv_0_3)
maxpool_1_3 = MaxPool2D(pool_size=(sequence_length - 4 + 1, 1), strides=(1,1), padding='valid')(conv_1_3)
maxpool_2_3 = MaxPool2D(pool_size=(sequence_length - 5 + 1, 1), strides=(1,1), padding='valid')(conv_2_3)

concatenated_tensor_3 = Concatenate(axis=1)([maxpool_0_3, maxpool_1_3, maxpool_2_3])
flatten_3 = Flatten()(concatenated_tensor_3)

dropout_3 = Dropout(0.5)(flatten_3)
output_2 = Dense(units=3, activation='softmax')(dropout_3)
model_2 = Model(inputs=inputs_2, outputs=output_2)
model_2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model_2.summary())
batch_size = 50
history_2 = model_2.fit(X_train, y_train, epochs=20, batch_size=batch_size, verbose=1, validation_split=0.2)
plt.plot(history_2.history['accuracy'])
plt.plot(history_2.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

plt.plot(history_2.history['loss'])
plt.plot(history_2.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
y_hat_2 = model_2.predict(X_test)
confusion_matrix(list(map(lambda x: np.argmax(x), y_test)), list(map(lambda x: np.argmax(x), y_hat_2)))
accuracy_score(list(map(lambda x: np.argmax(x), y_test)), list(map(lambda x: np.argmax(x), y_hat_2)))
from sklearn import metrics
print(metrics.classification_report( list(map(lambda x: np.argmax(x), y_test)),  list(map(lambda x: np.argmax(x), y_hat_2)), digits=3))<jupyter_output>precision    recall  f1-score   support

           0      0.360     0.030     0.055      1673
           1      0.977     0.948     0.962      2592
           2      0.656     0.988     0.788      3185

    accuracy                          0.759      7450
   macro avg      0.664     0.655     0.602      7450
weighted avg      0.701     0.759     0.684      7450<jupyter_text>Saving the model:<jupyter_code>model_2.save(home + 'saved_model/model_2') 
model_2 = tf.keras.models.load_model(home + 'saved_model/model_2')

# Check architecture
model_2.summary()<jupyter_output><empty_output><jupyter_text>Test of new created instances:<jupyter_code>sentences =pd.DataFrame([["I do not trust him! is shameful"], 
                         ["good place for dinner! "], 
                         ["Italy is a member of european unio$n"], 
                         ["the candidate is untrustble and terrible!"],
                         ["the earth is a planet within the solar system"],
                         ["I'm very HAPPY for him"]])

#df['text'] = df['clean_comment'].apply(clean_str)

sentences['text'] = sentences[0].apply(clean_str)

token_sent = tokenizer.texts_to_sequences(sentences["text"].values)

sent_y = pad_sequences( token_sent , sequence_length)


sentences_hat = model.predict(sent_y)

indexes = tf.argmax(sentences_hat, axis=1)

for i in range(len(sentences)):
  print(indexes.numpy()[i], sentences['text'][i])<jupyter_output>1 i do not trust him is shameful
2 good place for dinner
1 italy is a member of european union
0 the candidate is untrustble and terrible
1 the earth is a planet within the solar system
0 im very happy for him
{'fraction_non_alphanumeric': 0.07944200944032367, 'fraction_numerical': 0.07666048550236008, 'mean_word_length': 8.457552809884415, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '510771', 'license': 'Unlicense', 'n_tokens_mistral': 12417, 'n_tokens_neox': 11414, 'n_words': 1137, 'path': 'Projects/Text_Mining_and_NLP_Prof_Marinai.ipynb', 'repo_name': 'gobrac/Portfolio---data-Analyst', 'size': '250910'}