<jupyter_start><jupyter_text>Please **DO NOT** run any cells as everything (all final images, graphs, etc. are preloaded) Section 0 Preface for Imports, Data Handling, & Methodologies  Section 0.1 Preface for Write-Up Interpretation & AcknowledgementsFor the remainder of this notebook, each part 1) and 2) will be answered in line with notation for the following block of code to show:1. A brief statement (~paragraph) of what was done to answer the question (narratively explaining what you did in code to answer the question, at a high level).2. A brief statement (~paragraph) as to why this was done (why the question was answered in this way, not by doing something else. Some kind of rationale as to why you did x and not y or z to answer the question – why is what you did a suitable approach?).For 3) and 4) (below) the findings and interpretations will be provided in Results & Discussions for each question (model(s)) we ran.3. A brief statement (~paragraph) as to what was found. This should be as objective and specific as possible – just the results/facts. Do make sure to include numbers and a figure (=a graph or plot) in your statement, to substantiate and illustrate it, respectively. As the unsupervised methods often yield visualizable results, be sure to include a figure. 4. A brief statement (~paragraph) as to what you think the findings mean. This is your interpretation of your findings and should answer the original question.Code was used from my own Github repository, found at `www.github.com/sunnydigital/IDS_F21`, including code derived from <NAME> from Introduction to Data Science, Fall 2021. Most of the code falling under the aforementioned two categories surrounds the `PCA` and `k-means` analysis plots.All code used in this analysis attributable to Introduction to Machine Learning is not cited - we feel it is fair to use code from the course.The author refers to all analysis performed in the first-person plural tense 'we,' as the author believes it to be weird to say *'I'* did anything, given all *I* did was learn from the amazing instructors & TAs :) Thank you. Section 0.2 Imports & Installation of Packages, Libraries, Seaborn Settings, and Dataset(s)Below we set the random seed to the numeric portion of my NYU ID: `N12664675` import packages & libraries as well as set the settings for `seaborn` plots<jupyter_code>!pip install xgboost
!pip install impyute
!pip install missingno
!pip install eli5
!pip install scikit-optimize
!pip install tune-sklearn ray[tune]
!pip install scikit-plot
!pip install colorcet==3.0.0
import random
random.seed(12664675)
import sys
import os
import time
import re

import numpy as np
import pandas as pd

'''import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.modules.activation import Softmax
import torch.optim as optim'''

from statsmodels.distributions.empirical_distribution import ECDF
import statsmodels.api as sm

import scipy.stats as st
import statsmodels.api as sm ## Need revision for Windows use
from scipy.stats import zscore
from scipy.spatial.distance import squareform

from sklearn.inspection import permutation_importance
from sklearn.linear_model import LinearRegression, Ridge, Lasso, LogisticRegression, Perceptron
from sklearn.neural_network import MLPClassifier, MLPRegressor
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.model_selection import train_test_split, cross_validate, KFold, cross_val_score, RepeatedKFold, GridSearchCV
from sklearn.metrics import mean_squared_error as mse
from sklearn.metrics import accuracy_score, roc_auc_score, make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn import tree, ensemble, metrics, calibration
from sklearn.svm import SVC, LinearSVC
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE, MDS
from sklearn.cluster import KMeans, DBSCAN
from sklearn.metrics import silhouette_samples
from sklearn.impute import KNNImputer

from IPython import display

from eli5.sklearn import PermutationImportance
from eli5 import show_prediction
import eli5

from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_objective
from skopt import BayesSearchCV

'''from tune_sklearn import TuneSearchCV, TuneGridSearchCV
import ray.tune as tune'''

from tqdm import tqdm

from matplotlib import pyplot as plt
from matplotlib import mlab
from matplotlib.pyplot import figure
from matplotlib.colors import ListedColormap
from mpl_toolkits.mplot3d import Axes3D
import scikitplot as skplt
import seaborn as sns; sns.set_theme(color_codes=True); sns.set_style("whitegrid")
import graphviz
import colorcet as cc

from impyute.imputation.cs import fast_knn

import xgboost as xgb
data = 'musicData.csv'
df = pd.read_csv(data)

colors = ['#000099', '#ffff00']
fig, ax = plt.subplots(figsize=(100,10))

sns.heatmap(df.isnull(), cmap=sns.color_palette(colors), ax=ax)
df_map = df[['instance_id', 'artist_name', 'track_name', 'obtained_date']]
df = df.loc[:, ~df.columns.isin(['instance_id', 'artist_name','track_name','obtained_date'])]
df.head(10)<jupyter_output><empty_output><jupyter_text>Section 0.4 Handling Null/String & Categorical DataBelow, we check the presence of `NA`'s and any non-numeric values in the dataframe and afterwards output a description of the dataframe.<jupyter_code>lambda_apply = lambda x: pd.to_numeric(x, errors='coerce')
df[['popularity', 'acousticness', 'danceability', 'duration_ms', 
    'energy', 'instrumentalness', 'liveness', 'loudness',
    'speechiness', 'tempo', 'valence']] = df[['popularity', 'acousticness', 'danceability', 'duration_ms', 
                                              'energy', 'instrumentalness', 'liveness', 'loudness',
                                              'speechiness', 'tempo', 'valence']].apply(lambda_apply).replace(-1, np.NaN)
to_impute = df[['popularity', 'acousticness', 'danceability', 'duration_ms', 
                'energy', 'instrumentalness', 'liveness', 'loudness',
                'speechiness', 'tempo', 'valence']]

imputer = KNNImputer(n_neighbors=5)
impute_fit = imputer.fit_transform(to_impute)

df_drop = df

df_drop[['popularity', 'acousticness', 'danceability', 'duration_ms', 
         'energy', 'instrumentalness', 'liveness', 'loudness',
         'speechiness', 'tempo', 'valence']] = impute_fit

df_drop.dropna(inplace=True)<jupyter_output><empty_output><jupyter_text>Below, we create a combined column out of `key` and `mode`, signifying the gradable key, mode (e.g. A major or C minor) combination in music, denoting the colloquial `key` a song is written in.<jupyter_code>df_drop['key'] = df_drop['key'] + '_' + df_drop['mode']
df_drop = df_drop.drop(labels=['mode'], axis='columns').reset_index(drop=True)
df_drop['tempo'].unique()
## To check how many nulls there are in a column
df_drop['tempo'].isnull().sum()<jupyter_output><empty_output><jupyter_text>In this section we use the `LabelEncoder` package from `sklearn.preprocessing` and the `get_dummies` package from `pandas` to obtain numerical data for features with string values and dummy features for categorical data.<jupyter_code>key_le = LabelEncoder().fit(df_drop['key'])
key_encode = pd.DataFrame(data=key_le.transform(df_drop['key']), columns=['key'])

genre_le = LabelEncoder().fit(df_drop['music_genre'])
genre_encode = pd.DataFrame(data=genre_le.transform(df_drop['music_genre']), columns=['music_genre'])

key_inverse = lambda x: key_le.inverse_transform(x)
genre_inverse = lambda x: genre_le.inverse_transform(x)
df_encode = df_drop

df_encode['key'] = key_encode['key']
df_encode['music_genre'] = genre_encode['music_genre']
print(key_encode.isnull().sum())
print(genre_encode.isnull().sum())

df_encode.isnull().sum()<jupyter_output>key    0
dtype: int64
music_genre    0
dtype: int64<jupyter_text>Section 0.5 Train Test Split Using Stratification<jupyter_code>X = df_encode.loc[:,:'music_genre']
y = df_encode.loc[:,'music_genre']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, stratify=y)<jupyter_output><empty_output><jupyter_text>Section 0.6 Standardization & NormalizationFor this dataset, (and the applicable functions of dimensionality reduction through PCA, t-SNE, MDS; and clustering through k-means and DBSCAN) I have chosen to standardize through the below mapping:$$ \vec{X}_{\textrm{Z-Score}} = \frac{\vec{X} - \mu_{\vec{X}}}{\sigma_{\vec{X}}} $$where each $\vec{X}$ is a given column of data.<jupyter_code>X_train.dtypes
X_train = st.zscore(X_train)
X_test = st.zscore(X_test)<jupyter_output><empty_output><jupyter_text>Section 1.0 PCA Fitting and Principal Component Analysis<jupyter_code>ordinal = lambda n: "%d%s" % (n,"tsnrhtdd"[(n//10%10!=1)*(n%10<4)*n%10::4])

pca, pca_t = PCA().fit(X_train), PCA().fit(X_test)

eig_vals, eig_vals_t = pca.explained_variance_, pca_t.explained_variance_
loadings, loadings_t = pca.components_ * -1, pca_t.components_ * -1
rotated_data, rotated_data_t = pca.fit_transform(X_train) * -1, pca_t.fit_transform(X_test) * -1

covar_explained, covar_explained_t = eig_vals / sum(eig_vals) * 100, eig_vals_t / sum(eig_vals_t) * 100

for n in range(len(covar_explained)):
    explained = covar_explained[n].round(5)
    print(f'{ordinal(n+1)} PC explains {explained}% of variance')
num_col = len(X_train.columns)
x = np.linspace(1,num_col, num_col)
plt.bar(x, eig_vals, color='gray')
plt.plot([0,num_col], [1,1], color='orange')
plt.xlabel('Principal Component')
plt.ylabel('Eigenvalue')
plt.title('Wines.csv Principal Components')
plt.show()
threshold = 1

print('Number of factors selected by Kaiser criterion:', np.count_nonzero(eig_vals > threshold))

print('Number of factors selected by Elbow criterion: 3')

threshold = 90
eig_sum = np.cumsum(covar_explained)
print('Number of factors to account for at least 90% variance:', np.count_nonzero(eig_sum < threshold) + 1)
look_at = 9
for i in range(0,look_at):
    
    plt.subplot(3,3,i+1)
    plt.bar(x, loadings[i,:])
    
    plt.title(f'Principal Component {i+1}', fontsize=10)
    plt.xlabel('Predictor', fontsize=8)
    plt.ylabel('Loading', fontsize=8)
    
    plt.tight_layout()
for index in range(len(X_train.columns)):
    print(f'Predictor {index+1}: {X_train.columns[index]}')
sns.scatterplot(x=rotated_data[:,0], y=rotated_data[:,1], hue=y_train, s=10, marker='x', palette='Paired')

plt.title('Comparison of First Two PCs')
# Set x-axis label
plt.xlabel('Principal Component 1')
# Set y-axis label
plt.ylabel('Principal Component 2')
%matplotlib notebook

fig = plt.figure()
ax = fig.gca(projection='3d')
fig.add_axes(ax)

cmap = ListedColormap(sns.color_palette('Paired', 10).as_hex())

sc = ax.scatter(rotated_data[:,0],
                rotated_data[:,1],
                rotated_data[:,2],
                s=40, marker='x', c=y_train.to_list(), cmap=cmap, alpha=0.8)

ax.set_xlabel('Principal Component 1')
ax.set_ylabel('Principal Component 2')
ax.set_zlabel('Principal Component 3')

ax.legend(*sc.legend_elements(), bbox_to_anchor=(1.05, 1), loc=2)

plt.show()
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Section 2.0 K-Means Clustering<jupyter_code>x = np.column_stack((rotated_data[:,0],rotated_data[:,1]))

num_clusters = 9 # how many clusters are we looping over? (from 2 to 10)
Q = np.empty([num_clusters,1])*np.NaN # init container to store sums
# Compute kMeans:
for i in range(2, 11): # Loop through each cluster (from 2 to 10!)
    kMeans = KMeans(n_clusters = int(i)).fit(x) # compute kmeans using scikit
    cId = kMeans.labels_ # vector of cluster IDs that the row belongs to
    cCoords = kMeans.cluster_centers_ # coordinate location for center of each cluster
    s = silhouette_samples(x,cId) # compute the mean silhouette coefficient of all samples
    Q[i-2] = sum(s) # take the sum
    # Plot data:
    plt.subplot(3,3,i-1)
    plt.hist(s,bins=20) 
    plt.xlim(-0.2,1)
    plt.ylim(0,10000)
    plt.xlabel('Silhouette score')
    plt.ylabel('Count')
    plt.title(f'Clust: {i}, Sum: {int(Q[i-2])}') # sum rounded to nearest integer
    plt.tight_layout() # adjusts subplot padding
plt.plot(np.linspace(2,10,9),Q)
plt.xlabel('Number of Clusters')
plt.ylabel('Sum of Silhouette Scores')
plt.show()
num_clusters = np.argmax(Q) + 2
kMeans = KMeans(n_clusters = num_clusters).fit(np.column_stack((rotated_data[:,0],
                                                                rotated_data[:,1])))
cId = kMeans.labels_ 
cCoords = kMeans.cluster_centers_ 

for i in range(num_clusters):
    plotIndex = np.argwhere(cId == int(i))
    plt.plot(rotated_data[plotIndex,0],rotated_data[plotIndex,1],'x',markersize=5, alpha=0.1, zorder=2)
    plt.plot(cCoords[int(i-1),0], cCoords[int(i-1),1],'o', markersize=5, color='black')  
    plt.xlabel('Principal Component 1')
    plt.ylabel('Principal Component 2')    
    plt.title('Comparison of First Two PCs')
%matplotlib widget

num_clusters = np.argmax(Q) + 2
kMeans = KMeans(n_clusters = num_clusters).fit(np.column_stack((rotated_data[:,0],
                                                                rotated_data[:,1],
                                                                rotated_data[:,2]))) 
cId = kMeans.labels_ 
cCoords = kMeans.cluster_centers_ 

fig = plt.figure(figsize=(6,6))
ax = Axes3D(fig, auto_add_to_figure=False)
fig.add_axes(ax)

# Plot the color-coded data:
for i in range(num_clusters):
    
    plotIndex = np.argwhere(cId == int(i))
    
    sc = ax.scatter(rotated_data[plotIndex,0],
                    rotated_data[plotIndex,1],
                    rotated_data[plotIndex,2],
                    label=f'Cluster {i+1}',
                    s=40, marker='x', alpha=0.05)
    
    sc = ax.scatter(cCoords[int(i-1),0],
                    cCoords[int(i-1),1],
                    cCoords[int(i-1),2],
                    color='black', s=70, alpha=1.0)
    
ax.legend(loc='best')
ax.set_xlabel('Principal Component 1')
ax.set_ylabel('Principal Component 2')
ax.set_zlabel('Principal Component 3')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Section 3.0 Xtreme Gradient Boost (XGBoost) Fitting & Analysis  Appendix Appendix 1.0 Depreciated Code<jupyter_code>df_drop = df.drop(index=df[(df['duration_ms'] == -1)].index, axis=0).reset_index(drop=True)
string_data = ['key']
categorical_data = ['mode', 'music_genre']

smap = lambda x: re.search('\w+(?!_)', x)[0]
cmap = lambda x: re.search('\w+(?=_name)|mode|((?<=music_)\w+)', x)[0]
encode_string = {}
for i, col in zip(range(len(string_data)), string_data):
    encode_string[f'encode_{smap(col)}'] = LabelEncoder().fit(df[[col]])
    
encode_categorical = {}
for i, col in zip(range(len(categorical_data)), categorical_data):
    encode_string[f'encode_{cmap(col)}'] = LabelEncoder().fit(df[[col]])
df_process = pd.get_dummies(data=df_encode, prefix='mode', columns=['mode'])

df_process = df_process[['popularity', 'acousticness', 'danceability', 'duration_ms', 
                         'energy', 'instrumentalness', 'key', 'liveness', 'loudness', 'mode_Major', 'mode_Minor', 
                         'speechiness', 'tempo', 'valence', 'music_genre']]

df_process[['tempo', 'mode_Major', 'mode_Minor']] = df_process[['tempo', 'mode_Major', 'mode_Minor']].apply(pd.to_numeric)
tts_dict = {}
for i in range(10):
    tts = train_test_split(df_encode.loc[(df_encode['music_genre'] == i), :'music_genre'], 
                           df_encode.loc[(df_encode['music_genre'] == i), 'music_genre'],
                           test_size=0.1)

    tts_dict[f'genre_{i}_tts'] = tts<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1233175233944366, 'fraction_numerical': 0.015254454557108063, 'mean_word_length': 4.734288864388093, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '556250', 'license': 'MIT', 'n_tokens_mistral': 5457, 'n_tokens_neox': 5184, 'n_words': 1406, 'path': 'Capstone Project/Sunny Son IML Capstone-RIEGSIL.ipynb', 'repo_name': 'sunnydigital/IML_S22', 'size': '15610'}