<jupyter_start><jupyter_text>Word embeddings<jupyter_code>import pandas as pd
import numpy as np
import lucem_illud_2020
import spacy
from nltk.tokenize import sent_tokenize
from lucem_illud_2020 import word_tokenize
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
import sklearn.metrics.pairwise
import sklearn.decomposition
import matplotlib.pyplot as plt
from adjustText import adjust_text<jupyter_output><empty_output><jupyter_text>Loading data:<jupyter_code>file = '../../data/clean/speech.pkl'
df = pd.read_pickle(file)
df['tokenized_sentences'] = df['cleaned text'].apply(sent_tokenize)
df['tokenized_words_in_sentences'] = df['tokenized_sentences'].apply(lambda x: [word_tokenize(s) for s in x])
df['normalized_words_concat'] = df['normalized_words'].apply(lambda x: ' '.join(x))
df.head()<jupyter_output><empty_output><jupyter_text>Analysis descriptionWe used word embeddings to explore the semantic spaces projected by each speech and the social and political insights derived from this result. Specifically, we projected the semantic space of each speech using document embeddings and focused in which speeches show a more unique meaning, providing hypotheses for the possible reasons why some speeches show innovative or similar contents.We started this process by using the `Doc2Vec` function of the `gensim` package to train a model containing the semantic space of each speech projected in one hundred "meanings". To get similitude metrics for the speeches, we calculated the cosine difference of the projection of every pair of speeches using the `cosine\_similarity` function of the `sklearn` package.<jupyter_code>speeches = [TaggedDocument(doc, [year]) for doc, year in zip(list(df['normalized_words_concat']), list(df['year']))]
model = Doc2Vec(speeches, vector_size = 100)
targetDocs = df['year']

heatmapMatrixD = []

for tagOuter in targetDocs:
    column = []
    tagVec = model.docvecs[tagOuter].reshape(1, -1)
    for tagInner in targetDocs:
        column.append(sklearn.metrics.pairwise.cosine_similarity(tagVec, model.docvecs[tagInner].reshape(1, -1))[0][0])
    heatmapMatrixD.append(column)
heatmapMatrixD = np.array(heatmapMatrixD)
fig, ax = plt.subplots(figsize=(25,35))
hmap = ax.pcolor(heatmapMatrixD, cmap='Reds')
#cbar = plt.colorbar(hmap)

#cbar.set_label('cosine similarity', rotation=270)
ax.set_xticks(np.arange(heatmapMatrixD.shape[1]) + 0.5, minor=False)
ax.set_yticks(np.arange(heatmapMatrixD.shape[0]) + 0.5, minor=False)

ax.set_xticklabels(targetDocs, minor=False, rotation=90, size=18)
ax.set_yticklabels(targetDocs, minor=False, size=18)
#fig.tight_layout()

plt.savefig('../../outputs/word_embeddings/doc2vec_cosine_similarity_matrix.png')<jupyter_output><empty_output><jupyter_text>Now we proceeded with obtaining the two principal components of the document embeddings projections of each speech. We then produced a scatter plot of the result, similar to what we did in our clustering analysis for subsection `Clustering`. This process allowed us to visualize the semantic space where each speech were projected, showing which speeches had different meanings than the rest. We used the `PCA` function of the `sklearn` package for this.<jupyter_code>model_result = np.zeros((len(df), 100))
years = list(df['year'])
for i, year in enumerate(years):
    model_result[i] = model[year]
pca = sklearn.decomposition.PCA(n_components = 2).fit(model_result)
reduced_pca = pca.transform(model_result)
# Plotting

presidents = list(df['president'].unique())
colors = list(plt.cm.rainbow(np.linspace(0,1, len(presidents))))
colors_p = [colors[presidents.index(l)] for l in df['president']]

plt.figure(figsize=(16,10))
fig = plt.figure(1)
ax = fig.add_subplot(111)
ax.set_frame_on(False)
plt.scatter(reduced_pca[:, 0], reduced_pca[:, 1], color = colors_p, alpha = 0.5)
plt.xticks(())
plt.yticks(())
texts = [plt.text(reduced_pca[:, 0][i],
                  reduced_pca[:, 1][i],
                  df['year'][i],
                  ha='center',
                  va='center',
                  size=14) for i in range(len(reduced_pca))]
adjust_text(texts)
plt.savefig('../../outputs/word_embeddings/doc2vec_pca_scatterplot_years.png')
# Plotting
plt.figure(figsize=(16,10))
fig = plt.figure(1)
ax = fig.add_subplot(111)
ax.set_frame_on(False)
plt.scatter(reduced_pca[:, 0], reduced_pca[:, 1], color = colors_p, alpha = 0.5)
plt.xticks(())
plt.yticks(())
texts = [plt.text(reduced_pca[:, 0][i],
                  reduced_pca[:, 1][i],
                  df['administration'][i],
                  ha='center',
                  va='center') for i in range(len(reduced_pca))]
adjust_text(texts)
plt.savefig('../../outputs/word_embeddings/doc2vec_pca_scatterplot_adm.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12305099030762748, 'fraction_numerical': 0.016645596291613992, 'mean_word_length': 5.346256684491979, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '381007', 'license': 'MIT', 'n_tokens_mistral': 1651, 'n_tokens_neox': 1541, 'n_words': 393, 'path': 'notebooks/analysis/Archive/word_embeddings-projections.ipynb', 'repo_name': 'ccsuehara/Peruvian-president-s-speeches', 'size': '4746'}