<jupyter_start><jupyter_text>MODELOS DE CATEGORIZACION 1. Introducción<jupyter_code>import IPython.display as ipd
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn as skl
import sklearn.utils, sklearn.preprocessing, sklearn.decomposition, sklearn.svm

data = pd.read_pickle("clean_data/track.pkl")
unpickled_df_features = pd.read_pickle("clean_data/features.pkl")<jupyter_output><empty_output><jupyter_text>(Copiado de Data Analysis) Analisis de FeaturesLos features fueron generados utilizando la libreria de librosa sobre mp3 de extractos de cada cancion.(Del data esta es la primera agrupacion)Los features generados son:- mfcc: Mel-frequency cepstral coefficients (MFCCs). The Mel frequency cepstral coefficients (MFCCs) of a signal are a small set of features (usually about 10–20) which concisely describe the overall shape of a spectral envelope. It models the characteristics of the human voice.- chroma_cens: Computes the chroma variant “Chroma Energy Normalized” (CENS). CENS features are robust to dynamics, timbre and articulation, thus these are commonly used in audio matching and retrieval applications. Chroma features are an interesting and powerful representation for music audio in which the entire spectrum is projected onto 12 bins representing the 12 distinct semitones (or chroma) of the musical octave.- tonnetz: Tonal centroid features (tonnetz). This representation uses the method to project chroma features onto a 6-dimensional basis representing the perfect fifth, minor third, and major third each as two-dimensional coordinates.- spectral_contrast: Each frame of a spectrogram S is divided into sub-bands. For each sub-band, the energy contrast is estimated by comparing the mean energy in the top quantile (peak energy) to that of the bottom quantile (valley energy). High contrast values generally correspond to clear, narrow-band signals, while low contrast values correspond to broad-band noise. - spectral_centroid: Each frame of a magnitude spectrogram is normalized and treated as a distribution over frequency bins, from which the mean (centroid) is extracted per frame.It indicates where the ”centre of mass” for a sound is located and is calculated as the weighted mean of the frequencies present in the sound. Consider two songs, one from a blues genre and the other belonging to metal. Now as compared to the blues genre song which is the same throughout its length, the metal song has more frequencies towards the end. So spectral centroid for blues song will lie somewhere near the middle of its spectrum while that for a metal song would be towards its end.- spectral_bandwidth: Compute p’th-order spectral bandwidth.- spectral_rolloff: The roll-off frequency is defined for each frame as the center frequency for a spectrogram bin such that at least roll_percent (0.85 by default) of the energy of the spectrum in this frame is contained in this bin and the bins below. This can be used to, e.g., approximate the maximum (or minimum) frequency by setting roll_percent to a value close to 1 (or 0).- rmse: Compute root-mean-square (RMS) value for each frame, either from the audio samples y or from a spectrogram S.- zcr: Zero-crossing rate of an audio time series -> The zero crossing rate is the rate of sign-changes along a signal, i.e., the rate at which the signal changes from positive to negative or back. This feature has been used heavily in both speech recognition and music information retrieval. It usually has higher values for highly percussive sounds like those in metal and rock.Para mas informacion sobre cada feature: [Librosa features](https://librosa.org/doc/main/feature.html)SpectrogramA spectrogram is a visual representation of the spectrum of frequencies of sound or other signals as they vary with time. Spectrograms are sometimes called sonographs, voiceprints, or voicegrams. When the data is represented in a 3D plot, they may be called waterfalls. In 2-dimensional arrays, the first axis is frequency while the second axis is time.Para cada feature se calcula:- kurtosis- max- mean- median- min- skew- std Conclusion:- zcr: Me interesa quedarme con esto ya que categoriza al metal y rock- spectral_centroid: el centro de masa del sonido, en metal esta sobre el final y en blues en medio- spectral rolloff: Es una medida de la señal que represental la frecuencia por debajo de el total spectral enegergy - mfcc: modelo de voz humana- chroma: representacion poderosa del audiowe will choose 5 features, i.e. Mel-Frequency Cepstral Coefficients, Spectral Centroid, Zero Crossing Rate, Chroma Frequencies, Spectral Roll-off.	filename	chroma_stft	rmse	spectral_centroid	spectral_bandwidth	rolloff	zero_crossing_rate	mfcc1	mfcc2	mfcc3	...	mfcc12	mfcc13	mfcc14	mfcc15	mfcc16	mfcc17	mfcc18	mfcc19	mfcc20	label  Feature extraction<jupyter_code>unpickled_df_features.head(5).style.format('{:.2f}')
features_columns = ['mfcc', 'chroma_cens', 'spectral_centroid',"spectral_bandwidth", 'spectral_rolloff', "zcr"]

clean_features = unpickled_df_features[features_columns]
clean_features.shape
clean_features_spectral_centroid= unpickled_df_features["spectral_centroid"]["mean"]
print(clean_features_spectral_centroid.head())
clean_features_spectral_centroid = clean_features_spectral_centroid.rename(columns={"01":"spectral_centroid"})

clean_features_spectral_bandwidth = unpickled_df_features["spectral_bandwidth"]["mean"]
print(clean_features_spectral_bandwidth.head())
clean_features_spectral_bandwidth = clean_features_spectral_bandwidth.rename(columns={"01":"spectral_bandwidth"})


clean_features_spectral_rolloff = unpickled_df_features["spectral_rolloff"]["mean"]
print(clean_features_spectral_rolloff.head())
clean_features_spectral_rolloff = clean_features_spectral_rolloff.rename(columns={"01":"spectral_rolloff"})

clean_features_zcr = unpickled_df_features["zcr"]["mean"]
print(clean_features_zcr.head())
clean_features_zcr = clean_features_zcr.rename(columns={"01":"zcr"})

clean_features_mfcc = unpickled_df_features["mfcc"]["mean"]
clean_features_mfcc_mean =clean_features_mfcc.mean(axis=1)
print(clean_features_mfcc_mean.head())
#clean_features_spectral_rolloff = clean_features_spectral_rolloff.rename(columns={"0":"mfcc"})


clean_features_chroma_cens= unpickled_df_features["chroma_cens"]["mean"]
clean_features_chroma_cens_mean=clean_features_chroma_cens.mean(axis=1)
print(clean_features_chroma_cens_mean.head())
#clean_features_chroma_cens_mean = clean_features_chroma_cens_mean.rename(columns={"0":"chroma"})

clean_features = pd.concat([clean_features_spectral_rolloff, clean_features_spectral_bandwidth, clean_features_spectral_centroid, clean_features_zcr, clean_features_mfcc_mean, clean_features_chroma_cens_mean  ], axis=1, join='inner')
features = clean_features.rename(columns={0: "mfcc",1:"chroma" })

print(features.columns)

data_full = pd.concat([data, features], axis=1, join='inner')

data_full.shape
data=data_full<jupyter_output><empty_output><jupyter_text>Preparamos los datos del modelo<jupyter_code>from matplotlib import offsetbox
import joblib
#from PIL import Image
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn import metrics
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Observamos la media y varianza de las variables:<jupyter_code>#Obtenemos las variables numericas del data:
print (data.columns)
print(data.info())

#dropeo location porque tiene muchos nulos 
data_complete = data_full.drop(labels="location",axis=1)

print("Media de las variables: ")
print(data_complete.mean(axis=0))

print('\n')


print("Varianza de las variables: ")
print(data_complete.var(axis=0))

print (data_complete.columns)
#Armo una lista de numericas para depsues hacer el fit transform en estas
lista_numero=["duration", "acousticness","album_tracks", "danceability","energy","instrumentalness", "liveness",  "speechiness","tempo","valence",'spectral_rolloff', 'spectral_bandwidth', 'spectral_centroid', 'zcr',
       'mfcc', 'chroma']

data_complete=data_complete[lista_numero]
# El argumento stratify nos permite generar una división que respeta la misma proporción entre clases en ambos sets

X = data_complete
Y = data['genre_top']
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state = 1237, stratify= Y)

#tengo qeu hacer un stratidfy aca para qeu en la division de train test tenga las mismo porcentaje de varialbes)
display(Y_train.value_counts(normalize=True).round(2))

display(Y_test.value_counts(normalize=True).round(2))
#Es necesario llevar a la misma escala, porque sino la que tiene mayor varianza va a pesar mas en PCA. Por eso normalizamos los datos
#en un modelo no hacemos en el test el fit_transform, solo hacemos transform. Porque ya tenemos la informacion de la media y varianza en el fit del train.


std_sclr = StandardScaler()
std_sclr_trained = std_sclr.fit(X_train)
X_train_numerical = std_sclr_trained.transform(X_train)
X_train_numerical_scaled = pd.DataFrame(X_train_numerical, columns = lista_numero)
X_train_numerical_scaled.head()
X_test_numerical = std_sclr_trained.transform(X_test)
X_test_numerical_scaled = pd.DataFrame(X_test_numerical, columns = lista_numero)
X_test_numerical_scaled.head()
print("Media de las variables: ")
print(X_train_numerical_scaled.mean(axis=0))

print('\n')

# Observamos nuevamente la varianza de las variables: como normalizamos la varianza es 1
print("Varianza de las variables: ")
print(X_train_numerical_scaled.var(axis=0))<jupyter_output>Media de las variables: 
duration              2.054668e-17
acousticness          8.022849e-17
album_tracks          5.641946e-16
danceability          2.689316e-16
energy                6.707861e-17
instrumentalness     -2.376314e-16
liveness              1.595708e-16
speechiness          -1.208808e-16
tempo                 2.483364e-16
valence               2.045747e-16
spectral_rolloff     -1.985932e-16
spectral_bandwidth    2.344191e-16
spectral_centroid     4.954861e-16
zcr                  -4.218721e-17
mfcc                 -2.156437e-17
chroma                1.941324e-16
dtype: float64


Varianza de las variables: 
duration              1.000143
acousticness          1.000143
album_tracks          1.000143
danceability          1.000143
energy                1.000143
instrumentalness      1.000143
liveness              1.000143
speechiness           1.000143
tempo                 1.000143
valence               1.000143
spectral_rolloff      1.000143
spectral_bandwidth    1.00014[...]<jupyter_text>Features del modeloX_test_numerical_scaled / X_train_numerical_scaled / Y_train, Y_test  1. Reduccion de dimensionalidad -> PCA<jupyter_code>model_pca = PCA().fit(X_train_numerical_scaled)

X_train_PCA = model_pca.transform(X_train_numerical_scaled)
X_test_PCA = model_pca.transform(X_test_numerical_scaled)

componentes=model_pca.n_components_
print("Componentes del modelo", model_pca.n_components_)
def plot_explained_variance(components_count, X):

    model_pca = PCA(components_count).fit(X)

    explained_variance = model_pca.explained_variance_ratio_

    #print(explained_variance)

    cumulative_explained_variance = np.cumsum(explained_variance)

    #print(cumulative_explained_variance)

    plt.plot(cumulative_explained_variance)
    plt.xlabel('número de componentes')
    plt.ylabel('% de varianza explicada');
plot_explained_variance(components_count = componentes, X = X_train_numerical_scaled)<jupyter_output><empty_output><jupyter_text>PCA para features musicales<jupyter_code>lista_features=['spectral_rolloff', 'spectral_bandwidth', 'spectral_centroid', 'zcr', 'mfcc', 'chroma' ]

std_sclr = StandardScaler()
std_sclr_trained = std_sclr.fit(X_train[lista_features])
X_train_numerical = std_sclr_trained.transform(X_train[lista_features])
X_train_numerical_scaled = pd.DataFrame(X_train_numerical, columns = lista_features)
X_train_numerical_scaled.head()
X_test_numerical = std_sclr_trained.transform(X_test[lista_features])
X_test_numerical_scaled = pd.DataFrame(X_test_numerical, columns = lista_features)
X_test_numerical_scaled.head()
model_pca = PCA().fit(X_train_numerical_scaled)

X_train_PCA = model_pca.transform(X_train_numerical_scaled)
X_test_PCA = model_pca.transform(X_test_numerical_scaled)

componentes=model_pca.n_components_
print("Componentes del modelo", model_pca.n_components_)
plot_explained_variance(components_count = componentes, X = X_train_numerical_scaled)<jupyter_output><empty_output><jupyter_text>con la regla del codo notamos que con 3 variables de PCA podemos explicar más de 95% y con 2 podemos explicar el 95%  Representacion grafica con 2 variables<jupyter_code>pca_digits_vis = PCA(n_components=2)
data_numero = pca_digits_vis.fit_transform(data_complete[lista_features])
print(data_complete[lista_features].shape)
print(data_numero.shape)
def plot_digits_pca(projection, generos):
    
    colors = ["#476A2A", "#7851B8", "#BD3430", "#4A2D4E", "#875525",
          "#A83683", "#4E655E", "#853541", "#3A3120", "#535D8E"]
    plt.figure(figsize=(10,10))
    plt.xlim(projection[:,0].min(), projection[:,0].max())
    plt.ylim(projection[:,1].min(), projection[:,1].max())

    for i in range(len(projection)):
        plt.xlabel('Primer Componente Principal')
        plt.ylabel('Segundo Componente Principal')
        plt.scatter(projection[i,0], projection[i,1], s=10) #color=color[genero[i]]
        
#No pude conectar un color con un genero distinto 
plot_digits_pca(data_numero, data.genre_top)<jupyter_output><empty_output><jupyter_text>2. NAIVE BAYES<jupyter_code>from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
import seaborn as sns
gnb = GaussianNB()

gnb.fit(X_train_numerical_scaled, Y_train)
Y_pred = gnb.predict(X_test_numerical_scaled)

Y_pred
round(accuracy_score(Y_test, Y_pred), 2)
sns.heatmap(confusion_matrix(Y_test, y_pred), annot=True, fmt='.0f')
plt.ylabel('Etiquetas reales')
plt.xlabel('Etiquetas predichas');<jupyter_output><empty_output><jupyter_text>3. KNN<jupyter_code># Importamos la clase KNeighborsClassifier de módulo neighbors
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(X_train, Y_train)
y_pred = knn.predict(X_test)
from sklearn.metrics import accuracy_score
accuracy_score(Y_test, y_pred).round(2)
from sklearn.model_selection import cross_val_score, KFold
kf = KFold(n_splits=12, shuffle=True, random_state=12)

scores_para_df = []

for i in range(1, 21):
    
    # En cada iteración, instanciamos el modelo con un hiperparámetro distinto
    model = KNeighborsClassifier(n_neighbors=i)
    
    # cross_val_scores nos devuelve un array de 5 resultados,
    # uno por cada partición que hizo automáticamente CV
    cv_scores = cross_val_score(model, X_train, Y_train, cv=kf)
    
    # Para cada valor de n_neighbours, creamos un diccionario con el valor
    # de n_neighbours y la media y el desvío de los scores
    dict_row_score = {'score_medio':np.mean(cv_scores),
                      'score_std':np.std(cv_scores), 'n_neighbors':i}
    
    # Guardamos cada uno en la lista de diccionarios
    scores_para_df.append(dict_row_score)
df_scores = pd.DataFrame(scores_para_df)
df_scores.head()
df_scores['limite_inferior'] = df_scores['score_medio'] - df_scores['score_std']
df_scores['limite_superior'] = df_scores['score_medio'] + df_scores['score_std']
df_scores.head()
# Graficamos los resultados
plt.plot(df_scores['n_neighbors'], df_scores['limite_inferior'], color='r')
plt.plot(df_scores['n_neighbors'], df_scores['score_medio'], color='b')
plt.plot(df_scores['n_neighbors'], df_scores['limite_superior'], color='r');
# Identificamos el score máximo
df_scores.loc[df_scores.score_medio == df_scores.score_medio.max()]
# Utilizamos sklearn para estandarizar la matriz de features
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
# Verificamos que las variables ahora tengan media 0 y desvío 1.
print('Medias:', np.mean(X_train, axis=0).round(2))
print('Desvio:', np.std(X_train, axis=0).round(2))
# Calculamos nuevamente los scores de cross validation,
# pero esta vez sobre los features estandarizados:

scores_para_df_standard = []

for i in range(1, 21):
    model = KNeighborsClassifier(n_neighbors=i)
    cv_scores = cross_val_score(model, X_train, Y_train, cv=kf)
    dict_row_score = {'score_medio':np.mean(cv_scores),
                      'score_std':np.std(cv_scores), 'n_neighbors':i}
    scores_para_df_standard.append(dict_row_score)
# Creamos el DataFrame a partir de la lista de diccionarios
df_scores_standard = pd.DataFrame(scores_para_df_standard)
df_scores_standard.head()
df_scores_standard['limite_superior'] = df_scores_standard['score_medio'] + df_scores_standard['score_std']
df_scores_standard['limite_inferior'] = df_scores_standard['score_medio'] - df_scores_standard['score_std']
df_scores_standard.head()

# Graficamos los resultados
plt.plot(df_scores_standard['n_neighbors'], df_scores_standard['limite_inferior'], color='r')
plt.plot(df_scores_standard['n_neighbors'], df_scores_standard['score_medio'], color='b')
plt.plot(df_scores_standard['n_neighbors'], df_scores_standard['limite_superior'], color='r');
# Identificamos el score máximo
df_scores_standard.loc[df_scores_standard.score_medio == df_scores_standard.score_medio.max()]
# Asignamos el valor del k óptimo a una variable
best_k = df_scores_standard.loc[df_scores_standard.score_medio == df_scores_standard.score_medio.max(), 'n_neighbors'].values[0]
best_k
# Elegimos el modelo óptimo de acuerdo a las pruebas de cross validation
model = KNeighborsClassifier(n_neighbors=best_k)

# Lo ajustamos sobre los datos de entrenamiento
model.fit(X_train, Y_train)
#Evaluamos qué accuracy obtenemos en train
accuracy_score(Y_train, model.predict(X_train)).round(2)
# Lo utilizamos para predecir en test
X_test = scaler.transform(X_test) # ¡Importantísimo estandarizar también los datos de test con las medias y desvíos aprendidos en train!
y_pred = model.predict(X_test)
# Evaluamos el accuracy del modelo en test
accuracy_score(Y_test, y_pred).round(2)<jupyter_output><empty_output><jupyter_text>KNN con 11 neighbors -> El modelo esta under fiteando- Acurr en train -> 0.72 - Acurr en test -> 0.67<jupyter_code># Obtenemos la matriz de confusión
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(Y_test, y_pred)
cm
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix

sns.set(rc={'figure.figsize':(30,30)})

# Graficamos la matriz de confusión
print(confusion_matrix(Y_test, y_pred))
plot_confusion_matrix(model,X_test, Y_test)
sns.heatmap(confusion_matrix(Y_test, y_pred), annot=True, fmt='.0f')
plt.ylabel('Etiquetas reales')
plt.xlabel('Etiquetas predichas');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08609898490506496, 'fraction_numerical': 0.021248619365697153, 'mean_word_length': 6.126686656671664, 'pattern_counts': {'":': 6, '<': 40, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '86921', 'license': 'MIT', 'n_tokens_mistral': 6605, 'n_tokens_neox': 6246, 'n_words': 1638, 'path': 'notebooks/Modelizacion Coti.ipynb', 'repo_name': 'jpyunis/digital-house-challenge-3', 'size': '19013'}