<jupyter_start><jupyter_text>Exercise 1 - Word polarity detection with word embeddingThe objective is to build a system that takes as input a word and outputs a probability that theword has a positive or negative connotation (polarity). As illustrated in Figure 1, the system iscomposed of two parts in which the word embedding part will leverage on pre-trained vectorssuch as the one obtained with word2vec or GloVe. If you are not familiar with word embeddings,we recommend you to watch the class material including videos and pdf posted on Moodle. a) Read lexicons. Download lists of positive and negative words from the Opinion Lexiconavailable from https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html. A zipis also provided on Moodle. Listing 1 provides an example of function to read the lexicons.You may want to complete the code to remove lines that start with ;, that end with +and to remove empty lines. You should get 2005 positive words and 4783 negative words.<jupyter_code>def read_vocabulary_from_file (filename ):
  with open(filename , 'r', encoding ="ISO−8859−1") as f:
    content = f. readlines () # content is a list of lines
    content = [x.strip () for x in content] # removing newline chars

    # Remove elements containing ';' and '+'
    content = [x for x in content if ';' not in x and len(x) > 0]
    content = [x for x in content if '+' not in x and len(x) > 0]

    return content

# LOAD OPINION LEXICON
home_dir = 'opinion-lexicon-English/'
neg_file = home_dir + 'negative-words.txt'
pos_file = home_dir + 'positive-words.txt'

vocab_neg = read_vocabulary_from_file(neg_file)
vocab_pos = read_vocabulary_from_file(pos_file)
print("="*50)
print("OPINION LEXICON - ENGLISH")
print("="*50)
print("Loaded negative words: {} words".format(len(vocab_neg)))
print("Loaded positive words: {} words".format(len(vocab_pos)))<jupyter_output>==================================================
OPINION LEXICON - ENGLISH
==================================================
Loaded negative words: 4783 words
Loaded positive words: 2005 words<jupyter_text>b) Convert words into vectors. You can here go for two options : either by queryingan online API that returns you the vectors for a given word, or download a pre-trainedword-vector dictionary (word2vec, GloVe, etc.). The code provided in Listing 2 shows youhow to realise this using a GloVe embedding – zip also provided on Moodle.<jupyter_code>import numpy as np

# to get GloVe vectors: wget http://nlp.stanford.edu/data/glove.6B.zip
def load_glove_embeddings (path):
  embeddings = {}
  with open(path , 'r', encoding ='utf−8') as f:
    for line in f:
      values = line.strip ().split ()
      w = values[0]
      vectors = np.asarray(values[1:], dtype='float32')
      embeddings [w] = vectors
  return embeddings

# online query
import requests
import json

word = 'happy'
response = requests.get('https://icoservices.k8s.tic.heia-fr.ch/word-embedding/wordvector/word2vec/en/' + word)
vector = response.json ()
print(vector)

# off−line dictionary
word_dict = load_glove_embeddings ('./glove.6B.50d.txt')
word = 'happy'
vector = word_dict [word] # if word is in word_dict
print(vector)<jupyter_output>[5.6506757736206055, 12.520734786987305, -3.6179394721984863, 5.632171630859375, -3.351313352584839, -2.549443244934082, -2.328282117843628, 1.6070698499679565, -2.63830304145813, -1.2330397367477417, -3.712287425994873, 4.76062536239624, 1.0341637134552002, 2.15838885307312, -3.3286218643188477, -4.111255168914795, 1.6341111660003662, -1.8190534114837646, 3.094416379928589, 1.0175750255584717, -0.6583362817764282, 0.2975451350212097, -6.7455902099609375, -0.6072280406951904, -0.4232602119445801, -3.6135027408599854, 2.020463705062866, 8.311285972595215, 0.9904295206069946, 8.244004249572754, -1.4143904447555542, -6.177243232727051, -2.0402798652648926, 1.218432068824768, -0.2812301516532898, -2.2428207397460938, 2.5753173828125, -0.17160016298294067, -2.9521965980529785, -1.6330070495605469, 2.1099538803100586, -1.197441577911377, -5.9389567375183105, -2.6153135299682617, 0.42052891850471497, 1.6437393426895142, 2.9179530143737793, -0.4194609820842743, 3.8722822666168213, -2.620629072[...]<jupyter_text>c) Prepare the training and testing sets. Prepare the tensors X_train for trainingby taking the corresponding vectors of 1500 positive and 1500 negative words from thelexicon. Prepare the Y_train target output tensor corresponding to the training set. Youcan, for example use the target [1.0, 0.0] for a positive word and [0.0, 1.0] for a negativeword. For an embedding dimension of 50, the shapes of your X_train and Y_train tensorsshould be (3000,50) and (3000,2). In a similar way, prepare X_test and Y_test tensors.<jupyter_code>import random

# Negative [0.0, 1.0]
dataset_neg = [(l, [0.0, 1.0]) for l in vocab_neg if l in word_dict]
print(len(dataset_neg))

# Positive [1.0, 0.0]
dataset_pos = [(l, [1.0, 0.0]) for l in vocab_pos if l in word_dict]
print(len(dataset_pos))

# Shuffle dataset
random.shuffle(dataset_neg)
random.shuffle(dataset_pos)

# 1500 each for training
train_dataset = dataset_neg[:1500] + dataset_pos[:1500]
test_dataset = dataset_neg[1500:] + dataset_pos[1500:]

print()
print("Training dataset: {} items".format(len(train_dataset)))
print(train_dataset[:10])
print()
print("Test dataset: {} items".format(len(test_dataset)))
print(test_dataset[:10])
X_train = np.asarray([word_dict[x[0]] for x in train_dataset])
Y_train = np.asarray([x[1] for x in train_dataset])
    
print(X_train.shape)
print(X_train)
print(Y_train.shape)
X_test = np.asarray([word_dict[x[0]] for x in test_dataset])
Y_test = np.asarray([x[1] for x in test_dataset])

print(X_test.shape)
print(Y_test.shape)<jupyter_output>(3238, 50)
(3238, 2)<jupyter_text>d) Train and evaluate a classifier. Build a model, e.g. a double Dense layers in Keras(MLP) and train it. Report on the evolution of the loss and accuracy along the epochs.You should reach about 90% accuracy on the training set and 85% accuracy on the testset. Report on your model structure and fitting strategy.<jupyter_code>import tensorflow as tf

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn import metrics
# Define our model
model = Sequential()

model.add(Dense(128, input_shape=X_train[0].shape))
model.add(Dense(2, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='adam', 
              metrics=['accuracy'])

model.summary()
log = model.fit(X_train, Y_train, epochs=100, batch_size=128)
import matplotlib.pyplot as plt

plt.plot(log.history['accuracy'], label='Training')
plt.legend()
plt.grid()
loss, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Accuracy: %f' % (accuracy*100))<jupyter_output>Accuracy: 86.751080<jupyter_text>e) Analysis of results and discussions. Report on your experiments and comment yourclassification performances. Find 5 to 10 words which are not in your training set and onwhich the system thinks it is clearly positive or negative. Find some words on which thesystem hesitates, i.e. with output probabilities in range [0.4-0.6]). Are these words andoutputs making sense to you ?<jupyter_code>from scipy import spatial
import pandas as pd

Y_pred = model.predict(X_test)
print(Y_pred)
print(Y_test)

dist = spatial.distance.cosine(Y_pred[0], Y_test[0])
print(dist)

columns_name = ["word", "label", "pred", "dist"]
df_pred = pd.DataFrame(columns = columns_name)

for i in range(len(Y_test)):
    if Y_test[i][0]== 0:
        lbl = "neg"
    else:
        lbl = "pos"
    df_pred = df_pred.append({'word': test_dataset[i][0],         
                    'label': lbl,
                    'pred': Y_pred[i],
                    'dist': spatial.distance.cosine(Y_pred[i], Y_test[i])
                             },
                             ignore_index = True
                            )

# Top negative words
df_top_neg = df_pred[df_pred['label']=='neg'].sort_values(by=['dist'], ascending=True)
print(df_top_neg.head())
# Top positive words
df_top_pos = df_pred[df_pred['label']=='pos'].sort_values(by=['dist'], ascending=True)
print(df_top_pos.head())
# The most mistaken words
df_top_wrong = df_pred.sort_values(by=['dist'], ascending=False)
print(df_top_wrong.head())
# Words with hesitation
df_hes = df_pred[(df_pred['dist']>0.4) & (df_pred['dist']<0.6)]
print(df_hes)<jupyter_output>word label                      pred      dist
48      freakishly   neg  [0.60037655, 0.39962336]  0.445902
55          scarce   neg   [0.6565204, 0.34347966]  0.536429
86          madden   neg    [0.6914891, 0.3085109]  0.592558
117   unsettlingly   neg  [0.62143356, 0.37856644]  0.479750
119       inferior   neg    [0.6624995, 0.3375005]  0.546073
...            ...   ...                       ...       ...
3067   subsidizing   pos  [0.31554046, 0.68445945]  0.581340
3088       staunch   pos  [0.42361578, 0.57638425]  0.407788
3149       idolize   pos   [0.4160997, 0.58390033]  0.419660
3209      famously   pos  [0.39122415, 0.60877585]  0.459371
3218      morality   pos   [0.3078024, 0.69219756]  0.593686

[119 rows x 4 columns]
{'fraction_non_alphanumeric': 0.12135445111960677, 'fraction_numerical': 0.13904969961769525, 'mean_word_length': 4.2620689655172415, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '276982', 'license': 'MIT', 'n_tokens_mistral': 3934, 'n_tokens_neox': 3153, 'n_words': 894, 'path': 'exercises/exercise12/gr11_exercise12.ipynb', 'repo_name': 'PatCH0816/TSM_DeLearn', 'size': '9155'}