<jupyter_start><jupyter_text>First, we must clean the data of punctuation and alike<jupyter_code>import re
lyrics_clean = []
for lyrics in range(len(ds["Lyrics"])):
    r =  ds["Lyrics"][lyrics]
    lyrics_clean.append(re.sub("[^a-zA-Z1234567890çàãõáéíóúâêô']", " ", r.lower()))
ds["Lyrics"] = lyrics_clean
ds
words = []
for lyrics in ds["Lyrics"]:
    words.append(lyrics.split())

ds["Lyrics"] = words
list(ds["Title"])<jupyter_output><empty_output><jupyter_text>Now I will delete all songs that I know to be covers or to be instrumental (I could have done it automatically when creating the data but 1)that website had many mistakes in the song writer part; 2) the dataset is small enough<jupyter_code>takeout_songs = ['A Ordem Dos Templários (Instrumental)', 'A Whiter Shade Of Pale', 'Cherish', 
                 'Close The Door Lightly When You Go', 'Clothes Of Sand', 'Come Share My Life', 
                 'Desemprego', 'Dois Elefantes', 'Feedback Song For A Dying Friend', 'Gente Humilde',
                 'Gimme Shelter',  'Hey, That Is No Way To Say Goodbye', 'High Noon (Do Not Forsake Me)',
                 'Hoje a Noite Não Tem Luar', 'I Get Along Without You Very Well', 'Idade Média',
                 'If Tomorrow Never Comes', 'Juizo Final', 'La Solitudine', 'Metrópole', 'Meu Erro', 'My Girl',
                 'Nada Por Mim', 'Não Mais Que Três Semanas', 'No Fun', 'O Beco', 'O Grande Inverno da Rússia',
                 'O Que Eu Quero', 'On The Way Home / Rise  (pout-pourri)',  'Os Bons Morrem Jovens',
                 'Pais e Filhos / Stand By Me (medley)', 'Perfeição / Lithium (pout-pourri)', 'Più o Meno', 
                 'Pride Day',
                 'Que País É Esse / Cajuína / Pintinho Amarelinho / Aquele Abraço / Metal Contra As Nuvens (Pot-Pourri)',
                 'Riding Song', 'Sagrado Coração', '<NAME>', 
                 'Soldados / Blues da Piedade / Faz Parte do Meu Show / Nascente (Pot-Pourri)',
                 'Soldados / Help / Ball and Chain (Pot-Pourri)', 'Strani Amori', 'The Last Time I Saw Richard',
                 'Thunder Road', '<NAME>', 'Quem Inventou o Amor']
for title in takeout_songs:
    ds.drop(ds.index[(ds["Title"] == title)],axis=0,inplace=True)
ds[:10]
ds = ds.reset_index(drop=True) #reser indexes to drop the ones of the musics thrown away
lenghts = []
for lyrics in range(len(ds["Lyrics"])):
    lenghts.append(len(ds["Lyrics"][lyrics]))
plt.hist(lenghts, bins = 40)
plt.show<jupyter_output><empty_output><jupyter_text>Truncating at 400 words every single lyrics seems reasonable.<jupyter_code>X_train = []
for lyrics in range(len(ds["Lyrics"])):
    truncation = 400 - len(ds["Lyrics"][lyrics])
    if truncation >= 0:
        lyrics_new = ds["Lyrics"][lyrics] + ["<pad>" for component in range(truncation)]
    else:
        lyrics_new = ds["Lyrics"][lyrics][:400]
    X_train.append(lyrics_new)
X_train[10]<jupyter_output><empty_output><jupyter_text>Now, let us look at the new vocabulary<jupyter_code>from collections import Counter 
vocabulary = Counter()
for lyrics in X_train:
    vocabulary.update(lyrics)<jupyter_output><empty_output><jupyter_text>Let us use only 500 words<jupyter_code>vocabulary_new = vocabulary.most_common()[:500]<jupyter_output><empty_output><jupyter_text>Let us tokenize the vocabulary<jupyter_code>vocabulary_new_words = []
for word in range(len(vocabulary_new)):
    vocabulary_new_words.append(vocabulary_new[word][0])
vocabulary_new_words
words = tf.constant(vocabulary_new_words) 
word_ids = tf.range(len(vocabulary_new), dtype = tf.int64) 
vocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids)
num_oov_buckets = 200
table = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)<jupyter_output><empty_output><jupyter_text>Let us test the table<jupyter_code>table.lookup(tf.constant(["é saudade então e você galopa <pad>". split()]))<jupyter_output><empty_output><jupyter_text>All great, time to make the sets and then translate!<jupyter_code>def encode_words(data): 
    return table.lookup(data)
X_train_as_number = []
for lyrics in range(len(X_train)):
     X_train_as_number.append(encode_words(tf.constant(X_train[lyrics])))<jupyter_output><empty_output><jupyter_text>Now, let us make the actual X and y. For that, we will use the first 399 for X and the last 399 for y.<jupyter_code>len(X_train_as_number[0][:-1])
X = []
y = []
for lyrics in range(len(X_train_as_number)):
    X.append(X_train_as_number[lyrics][:-1])
    y.append(X_train_as_number[lyrics][1:])
X = np.array(X)
y = np.array(y)
X.shape, y.shape<jupyter_output><empty_output><jupyter_text>All right, time to train! We will do so in another notebook<jupyter_code>np.save("legiao-urbana-X", X)
np.save("legiao-urbana-y", y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12883564522908786, 'fraction_numerical': 0.010718789407313998, 'mean_word_length': 4.377401129943503, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '80413', 'license': 'Apache-2.0', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1682, 'n_words': 440, 'path': 'Legiao_Urbana_Lyrics_Data_Preparator.ipynb', 'repo_name': 'HLovisiEnnes/Mini_Projects', 'size': '4777'}