<jupyter_start><jupyter_text>Embeddings Neuronales**TODO**  Importanción de librería requeridas<jupyter_code>import os
import pickle

import warnings
import pickle
from collections import Counter

import numpy as np
import pandas as pd

import gensim.corpora as corpora
from gensim.models import FastText

from sklearn.metrics import pairwise_distances

from clustering_utils import vectorize, mbkmeans_clusters

warnings.filterwarnings("ignore", category=DeprecationWarning)
np.random.seed(42)

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Definición de variables globales<jupyter_code>TEXT_FILE_READ = 'docs/preprocessing_reddit_data.csv'
TEXT_SAVE_FILE = 'docs/reddit_data_fasttext.csv'
FILENAME_PICKLE = "docs/tmpreddit.pickle"
PICKLE_KMEANS = 'docs/models/fasttext_kmeans.model'

n_clusters = 120<jupyter_output><empty_output><jupyter_text>Lectura de los comentarios de RedditLos comentarios fueron previamente preprocesados (Ver en TODO).<jupyter_code>with open(FILENAME_PICKLE, 'rb') as f:
    df = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Vocabulario<jupyter_code># Create Dictionary
id2word = corpora.Dictionary(df['lemma_tokens'])

# Filtering Extremes
id2word.filter_extremes(no_below=2, no_above=.99)

# Creating a corpus object
corpus = [id2word.doc2bow(d) for d in df['lemma_tokens']]

processed_corpus = df['lemma_tokens']<jupyter_output><empty_output><jupyter_text>Entrenamiento del modelo FastText<jupyter_code>model = FastText(sentences=processed_corpus, vector_size=100, window=5, min_count=1, workers=1)
model.train(processed_corpus, total_examples=len(processed_corpus), epochs=100)
model.save("docs/models/fasttext.model")
# algunas predicciones
#model.get_nearest_neighbors('peron')
#model.get_nearest_neighbors('cristina')
#model.get_nearest_neighbors('nestor')
#model.get_nearest_neighbors('neoliberal')
#model.get_nearest_neighbors('malvinas')
#model.get_analogies('cristina', 'peronista', 'alberto')
#model.wv.most_similar('mapuche')<jupyter_output><empty_output><jupyter_text>Generación de vectores desde documentos<jupyter_code>vectorized_docs = vectorize(processed_corpus, model=model)
len(vectorized_docs), len(vectorized_docs[0])<jupyter_output><empty_output><jupyter_text>Generación de clusters<jupyter_code>clustering, cluster_labels = mbkmeans_clusters(
    X=vectorized_docs,
    k=n_clusters,
    mb=500,
    print_silhouette_values=True,
)
df_clusters = pd.DataFrame({
    "text": df["body"].values,
    "tokens": [" ".join(text) for text in processed_corpus],
    "cluster": cluster_labels
})

with open(PICKLE_KMEANS, 'wb') as f:
    pickle.dump(clustering, f)<jupyter_output>For n_clusters = 120
Silhouette coefficient: 0.01
Inertia:1525484.9615074536
Silhouette values:
    Cluster 107: Size:10 | Avg:0.26 | Min:-0.04 | Max: 0.39
    Cluster 43: Size:20 | Avg:0.15 | Min:0.02 | Max: 0.33
    Cluster 34: Size:216 | Avg:0.15 | Min:-0.03 | Max: 0.30
    Cluster 54: Size:145 | Avg:0.13 | Min:-0.08 | Max: 0.31
    Cluster 113: Size:159 | Avg:0.11 | Min:-0.08 | Max: 0.31
    Cluster 30: Size:91 | Avg:0.11 | Min:-0.04 | Max: 0.25
    Cluster 68: Size:69 | Avg:0.11 | Min:-0.05 | Max: 0.29
    Cluster 11: Size:85 | Avg:0.11 | Min:-0.01 | Max: 0.22
    Cluster 111: Size:35 | Avg:0.10 | Min:0.01 | Max: 0.26
    Cluster 112: Size:54 | Avg:0.10 | Min:-0.03 | Max: 0.26
    Cluster 27: Size:48 | Avg:0.09 | Min:-0.17 | Max: 0.30
    Cluster 39: Size:328 | Avg:0.09 | Min:-0.18 | Max: 0.24
    Cluster 89: Size:68 | Avg:0.09 | Min:-0.06 | Max: 0.30
    Cluster 17: Size:18 | Avg:0.08 | Min:-0.02 | Max: 0.21
    Cluster 101: Size:160 | Avg:0.08 | Min:-0.09 | Max: 0.31
    Cluster[...]<jupyter_text>*Top terms* por cluster (basado en los centroides de los clusters)<jupyter_code>print("Most representative terms per cluster (based on centroids):")
for i in range(n_clusters):
    tokens_per_cluster = ""
    most_representative = model.wv.most_similar(positive=[clustering.cluster_centers_[i]], topn=10)
    for t in most_representative:
        tokens_per_cluster += f"{t[0]} "
    print(f"Cluster {i}: {tokens_per_cluster}")<jupyter_output>Most representative terms per cluster (based on centroids):
Cluster 0: macri macri~~. macrium -macri \^macri macris mracri macri-peño mauriciomacri macr 
Cluster 1: acá acábo ac acv acaa acné aca \*sacá acabé acabás 
Cluster 2: ganar votar votastar voto votarar ganarir votos perderiar idiotar votarl 
Cluster 3: tener tenerla \-tener obtener abstener detener tenembaum retener sostener tenenbaum 
Cluster 4: calientenla calor caliente calo calentón calchaqui calienta escalofrío caliento agua 
Cluster 5: leer leeré leerlo leerme galeer leet lee leen libreer lees 
Cluster 6: fiacar bacar \-car orcar acar petacar facar cacar sacar evalúar 
Cluster 7: laempresanoesresponsablepordesgraciaspersonalnisupersonalestargenuinamenteinteresadoenloqueocurraenlavidadelcliente conformidad conectividad integridad invecilidad coeficiente intimidad descuidadamente entidad inconstitucionalidad 
Cluster 8: compralar comprendés compro compren comprendido compra comprar comprale comprate compremosle 
Cluster 9:[...]<jupyter_text>*Top terms* por cluster (basado en las palabras más frecuentes)<jupyter_code>for i in range(n_clusters):
    tokens_per_cluster = ""
    most_frequent = Counter(" ".join(df_clusters.query(f"cluster == {i}")["tokens"]).split()).most_common(5)
    for t in most_frequent:
        tokens_per_cluster += f"{t[0]}({str(t[1])}) "
    print(f"Cluster {i}: {tokens_per_cluster}")<jupyter_output>Cluster 0: macri(54) ah(7) macrista(4) /s(4) medio(3) 
Cluster 1: acá(148) aca(10) argentina(7) ver(7) sub(5) 
Cluster 2: ganar(139) voto(103) votar(73) perder(53) elección(27) 
Cluster 3: tener(160) él(11) año(11) tenés(10) tenia(7) 
Cluster 4: calor(83) agua(63) aire(32) team(30) frio(29) 
Cluster 5: leer(164) libro(21) comentario(17) escribir(11) él(11) 
Cluster 6: sacar(129) car(39) buscar(20) banco(14) gente(13) 
Cluster 7: gente(59) realidad(46) idea(38) cantidad(30) tenés(29) 
Cluster 8: comprar(148) compra(53) vender(38) compro(26) barato(25) 
Cluster 9: negro(57) chino(26) tiro(22) raro(16) puro(15) 
Cluster 10: vo(138) decir(11) favor(10) gracias(6) so(5) 
Cluster 11: the(123) of(30) is(11) vs(7) for(6) 
Cluster 12: elección(15) nación(13) inflación(12) acción(11) opción(8) 
Cluster 13: contar(34) matar(23) costar(17) faltar(15) control(11) 
Cluster 14: pagar(174) impuesto(66) cobrar(26) plan(23) pago(20) 
Cluster 15: mano(96) man(10) duro(5) humano(4) querer(4) 
Cluster 16: [...]<jupyter_text>Recupere los documentos más representativos (basados en los centroides de los clústeres) para un cluster en particular<jupyter_code>test_cluster = 0
most_representative_docs = np.argsort(
    np.linalg.norm(vectorized_docs - clustering.cluster_centers_[test_cluster], axis=1)
)
for d in most_representative_docs[:10]:
    print( df["body"].values[d])
    print("-------------")
test_v = vectorize([['defender', 'peso', 'siente', 'corazón', 'compro', 'pesos', 'tasa', 'fijo', 'año']], model=model)
prediction = clustering.predict(test_v)
print(prediction)
reddit = pd.read_csv(TEXT_FILE_READ)



def get_cluster(row):
    test_v = vectorize([str(row).split(" ")], model=model)
    return clustering.predict(test_v)

reddit['cluster'] = reddit.apply(lambda row: get_cluster(row['body_preprocessing']) , axis = 1)
# Show
reddit.head(10)
reddit.to_csv(TEXT_SAVE_FILE, index=False)
cluster_path = 'docs/test/fasttext_comments_per_cluster/'

os.makedirs(cluster_path,exist_ok=True)

for i in range(n_clusters):
    reddit[(reddit["cluster"] == i)][['flair', 'body']].to_csv(cluster_path + str(i) + '.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13057657187581528, 'fraction_numerical': 0.05922254109052961, 'mean_word_length': 5.976342129208371, 'pattern_counts': {'":': 3, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '180853', 'license': 'MIT', 'n_tokens_mistral': 3251, 'n_tokens_neox': 2927, 'n_words': 634, 'path': 'src/3c_pipeline_embedding_fasttext.ipynb', 'repo_name': 'EvaVillarrealGuzman/redditHateSpeech', 'size': '7666'}