<jupyter_start><jupyter_text>Download the spanish embeddings<jupyter_code># %cd embeddings
# Embeddings
# !wget -q http://dcc.uchile.cl/~jperez/word-embeddings/fasttext-sbwc.100k.vec.gz

# unzip
# !gzip -d -q fasttext-sbwc.100k.vec.gz

normalized_eval_path = '/content/drive/MyDrive/Colab Notebooks/regaetton_songs_nlp/data/normalized_eval_lyrics.csv'
split_train_path = '/content/drive/MyDrive/Colab Notebooks/regaetton_songs_nlp/data/normalized_train_split.csv'
scores_path = '/content/drive/MyDrive/Colab Notebooks/regaetton_songs_nlp/data/scores.csv'
import nltk
import pandas as pd

from nltk import word_tokenize
from gensim.models import KeyedVectors
from gensim.models.wrappers import FastText
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, recall_score

nltk.download('punkt')

def add_score(df, model, parameters, accuracy, recall):
    row = {'model': model, 'parameters': parameters,
           'accuracy': accuracy, 'recall': recall}
    df = df.append(row, ignore_index=True)

    return df<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>Load the data and the embeddings<jupyter_code>train = pd.read_csv(split_train_path)
eval = pd.read_csv(normalized_eval_path)
spanish_emb_path = '/content/drive/MyDrive/Colab Notebooks/regaetton_songs_nlp/embeddings/fasttext-sbwc.100k.vec'
spanish_emb = KeyedVectors.load_word2vec_format(spanish_emb_path)
emb_vocab = spanish_emb.vocab.keys()<jupyter_output><empty_output><jupyter_text>The first thing that we want to check is the amount of overlap between our vocab and the embeddings vocab.<jupyter_code>lyrics_vocab = list()
def get_vocab(lyrics):
    global lyrics_vocab

    words = lyrics.split()
    for word in words:
        if word not in lyrics_vocab:
            lyrics_vocab.append(word)

train.lyrics.apply(get_vocab)

def intersection(lst1, lst2): 
    overlap_words = set(lst1).intersection(lst2) 
    return overlap_words
overlap = intersection(emb_vocab, lyrics_vocab)
print(f'The amount of words in our vocab is {len(lyrics_vocab)}\nThe amount of words in both is {len(overlap)}') 
print(list(overlap)[:100])<jupyter_output>['lecciones', 'vendaval', 'manes', 'summer', 'carta', 'hechizos', 'saludo', 'lotería', 'sésamo', 'chiquillo', 'acontecimiento', 'trueno', 'cargar', 'pegaso', 'paz', 'focos', 'viñas', 'moleste', 'alzó', 'permito', 'mesita', 'progresado', 'tarima', 'requiere', 'sinceridad', 'enterarme', 'buzo', 'desenfreno', 'evoca', 'dijo', 'valdez', 'livia', 'sudo', 'creerlo', 'ena', 'evangelio', 'sinceramente', 'mitad', 'arrepiento', 'rumora', 'porción', 'desperate', 'reclame', 'lucho', 'hablemos', 'osamenta', 'podrida', 'determinación', 'queremos', 'suspirar', 'educado', 'recibes', 'galo', 'apartamentos', 'saqueando', 'motoristas', 'guardián', 'jaque', 'iluminaba', 'dancer', 'lados', 'mp', 'blade', 'cedió', 'unida', 'royal', 'bitch', 'quedaban', 'engañado', 'niños', 'maker', 'patiño', 'activos', 'ceder', 'comandando', 'advertimos', 'hermanos', 'anticipación', 'cae', 'acerqué', 'seeger', 'vel', 'bailar', 'sevilla', 'obscuro', 'imperfecto', 'oficios', 'constelación', 'continuan', 'talentoso', 'puesto',[...]<jupyter_text>The overlap is not enough to get a good embedding representation of each song, so the most recommendable thing to do is to create specific embeddings for our task.

Some of the reasons for this poor overlap can be :
- obviously that the domain in which the embeddings were trained is very different to our task domain. 
- Because of the nature of our data, which is input by users, we have a lot of spelling mistakes that, although, we have tried to solve some still pass the different checks. (example above: ena, qu, vel)
- finally, some lyrics have content in other languages, we also have done a lot of preprocessing to delete other languages but spanish. However, there are still some words that manage to past the different checks. (example above: summer)<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12761904761904763, 'fraction_numerical': 0.004047619047619047, 'mean_word_length': 5.7216, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '527650', 'license': 'MIT', 'n_tokens_mistral': 1511, 'n_tokens_neox': 1465, 'n_words': 390, 'path': 'models/pre_trained_embeddings.ipynb', 'repo_name': 'JonathanElejalde/reggaeton_songs_nlp', 'size': '4200'}