<jupyter_start><jupyter_text>Music Recommendation> I pledge my Honor that I have abided by the Stevens Honor System. - <NAME> & <NAME>  datasetTo download the dataset follow the instructions here:- https://www.kaggle.com/c/kkbox-music-recommendation-challenge/data?select=members.csv.7zIf you are running archlinux:- git clone https://aur.archlinux.org/kaggle-api.git- cd kaggle-api- makepkg -si- Go to the first link and create a kaggle account and agree to the competition rules- go to your account page on kaggle and create an api key and save the kaggle.json file in the folder ~/.kaggle/- kaggle competitions download -c kkbox-music-recommendation-challenge imports:<jupyter_code>import os
import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow.data as tfd


from loguru import logger
from tqdm import tqdm

from typing import List, Any, Tuple, Optional, Dict
datapath: str = os.path.join('..', 'data')
def list_files(directory: str, extension: str) -> List[str]:
    all_files = os.listdir(directory)
    return [os.path.join(directory, file) for file in all_files if file.split('.')[-1] == extension]
datasets: Dict[str, pd.DataFrame] = dict()
for filepath in tqdm(list_files(datapath, 'csv'), ascii=True, desc="Loading data from disk"):
    datasets[os.path.basename(filepath).split('.')[0]] = pd.read_csv(filepath)
print(f"Loaded {len(datasets)} csv files")
for key, value in datasets.items():
    print(f"length of dataset {key} is {len(value)}")
# Let's remove the 'sample_submission' dataset
_ = datasets.pop('sample_submission')
for key, value in datasets.items():
    print(f"Information for dataset: {key}")
    print("Description")
    print(value.describe())
    print('\n')
    print("dataframe 'head'")
    print(value.head())
    print('\n\n-------------------------------------------------------\n\n')<jupyter_output>Information for dataset: songs
Description
        song_length      language
count  2.296320e+06  2.296319e+06
mean   2.469935e+05  3.237800e+01
std    1.609200e+05  2.433241e+01
min    1.850000e+02 -1.000000e+00
25%    1.836000e+05 -1.000000e+00
50%    2.266270e+05  5.200000e+01
75%    2.772690e+05  5.200000e+01
max    1.217385e+07  5.900000e+01


dataframe 'head'
                                        song_id  song_length genre_ids  \
0  CXoTN1eb7AI+DntdU1vbcwGRV4SCIDxZu+YD8JP8r4E=       247640       465   
1  o0kFgae9QtnYgRkVPqLJwa05zIhRlUjfF7O1tDw0ZDU=       197328       444   
2  DwVvVurfpuz+XPuFvucclVQEyPqcpUkHR0ne1RQzPs0=       231781       465   
3  dKMBWoZyScdxSkihKG+Vf47nc18N9q4m58+b4e7dSSE=       273554       465   
4  W3bqWd3T+VeHFzHAUfARgW9AvVRaF4N5Yzm4Mr6Eo/o=       140329       726   

        artist_name                            composer     lyricist  language  
0  張信哲 (<NAME>)                                  董貞          何啟弘       3.0  
1         BLACKPINK  TEDD[...]<jupyter_text>description- we have a list of users, their personal information, the songs that they liked and didnt like, and where they accesed the song- we also have metadata about each song in the dataset Dataset descriptions:- members.csv    - user ID, location, gender, - song_extra_info + songs.csv    - song_id, song name, artist name, composer, lyricist, genre id, song length, language- train.csv:    - userid, songid, where they found the song, whether they liked it or nowThis will be used to create clusters of songs, or song groupsRecommendation dataset:- userid, song rating, songs cluster, songs feature vector<jupyter_code>len(datasets['songs'])
datasets["songs"]
raw = pd.DataFrame()
# categorical variables
raw["artist_name"] = datasets["songs"]["artist_name"]
raw["composer"] = datasets["songs"]["composer"] 
raw["lyricist"] = datasets["songs"]["lyricist"]
raw["genre_id"] = datasets["songs"]["genre_ids"]
raw["language"] = datasets["songs"]["language"]
raw["song_length"] = datasets["songs"]["song_length"]
raw["song_id"] = datasets["songs"]["song_id"]
raw
import re

def only_letters(tested_string):
    match = re.match("^[a-zA-Z]*$", str(tested_string))
    return match is not None
def only_numbers(tested_string):
    match = re.match("^[0-9]*$", str(tested_string))
    return match is not None
raw['artist_name'] = raw[raw['artist_name'].apply(only_letters)]['artist_name']
raw['lyricist'] = raw[raw['lyricist'].apply(only_letters)]['lyricist']
raw['composer'] = raw[raw['composer'].apply(only_letters)]['composer']
raw['genre_id'] = raw[raw['genre_id'].apply(only_numbers)]['genre_id']
raw<jupyter_output><empty_output><jupyter_text>We are working with a massive dataset, and to allocate enough memory to run an MCA on even just raw cat would take over 1.4 TiBso lets drop everything with an NA<jupyter_code>raw.dropna(axis=0, how='any', inplace=True)
raw = raw[(raw.artist_name != 'Various Artists') & (raw.artist_name != 'Various')]
raw = raw[raw.language != -1]<jupyter_output><empty_output><jupyter_text>One other way to reduce the number of categories for MCA is to filter our dataframe by the top 1000 artists<jupyter_code>num_artists=100
num_composers=500
num_lyricists=500
num_genres=500
num_languages=3
raw = raw[raw.artist_name.isin(raw['artist_name'].value_counts(sort=True).index.tolist()[:num_artists])]
raw = raw[raw.composer.isin(raw['composer'].value_counts(sort=True).index.tolist()[:num_composers])]
raw = raw[raw.lyricist.isin(raw['lyricist'].value_counts(sort=True).index.tolist()[:num_lyricists])]
raw = raw[raw.genre_id.isin(raw['genre_id'].value_counts(sort=True).index.tolist()[:num_genres])]
raw = raw[raw.language.isin(raw['language'].value_counts(sort=True).index.tolist()[:num_languages])]
raw
raw.artist_name.unique()
raw.composer.unique()
raw.lyricist.unique()
raw.genre_id.unique()
raw.language.unique()
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
raw["song_length"] = scaler.fit_transform(raw["song_length"].values.reshape(-1, 1))
raw
from prince.famd import FAMD
famd = FAMD(
    n_components=3,
    n_iter=5,
    copy=True,
    check_input=True,
    engine='auto',
    random_state=42
)
famd = famd.fit(raw.drop(['song_id'], axis='columns'))
famd.row_coordinates(raw)
ax = famd.plot_row_coordinates(
    X = raw,
    ax=None,
    figsize=(20, 20),
    x_component=0,
    y_component=2,
    color_labels=["Artist: {}".format(t) for t in raw["artist_name"]],
    ellipse_outline=False,
    ellipse_fill=True,
    show_points=True
)
ax.get_figure().savefig('famd_row_coordinates_100_artists_song_length.svg')
famd.eigenvalues_
famd.column_correlations(raw)
continuous_coordinates = pd.DataFrame()
continuous_coordinates["song_id"] = raw["song_id"]
continuous_coordinates = pd.concat([continuous_coordinates, famd.row_coordinates(raw)], axis=1)
continuous_coordinates = continuous_coordinates.rename({0:'X0', 1:'X1', 2:'X2'}, axis=1)
continuous_coordinates
train = datasets["train"]
train
df = pd.merge(train, continuous_coordinates, on='song_id')
df
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["user_id"] = le.fit_transform(df["msno"].values)
df.dropna(axis=0, how='any', inplace=True)
df['target'] = df['target'].replace([1], 'liked')
df['target'] = df['target'].replace([0], 'disliked')
for colname in ["X0", "X1", "X2"]:
    le = LabelEncoder()
    df[colname] = le.fit_transform(df[colname].values)
df
df.to_csv("cleaned_song_and_user_data.csv")
mfa = prince.mfa.MFA(
    groups={
        'Song Information': ["X0", "X1", "X2"],
        'User Discovery Info': ["source_system_tab", "source_screen_name", "source_type", "target"] 
    },
    n_components=3,
    n_iter=5,
    copy=True,
    check_input=True,
    random_state=42
)
mfa = mfa.fit(df.drop(['song_id', 'msno', 'user_id'], axis='columns'))
mfa.row_coordinates(df)<jupyter_output><empty_output><jupyter_text>gradient boostalternative to pca above<jupyter_code>import lightgbm as lgb
import gc
sei = pd.read_csv('../data/song_extra_info.csv')
members = pd.read_csv('../data/members.csv', parse_dates=['registration_init_time','expiration_date'])
songs = pd.read_csv('../data/songs.csv')
train = pd.read_csv('../data/train.csv')
test = pd.read_csv('../data/test.csv')
members['expiration_year'] = members['expiration_date'].dt.year
members.drop(columns = ['registration_init_time' , 'expiration_date'] , inplace = True)
members['membership_days'] = members['expiration_date'].subtract(members['registration_init_time']).dt.days.astype(int)
members['registration_year'] = members['registration_init_time'].dt.year
members.head()
train = train.merge(songs , on='song_id' , how='left')
train = train.merge(members , on = 'msno' , how='left')
train = train.merge(sei , on = 'song_id' , how='left')
test  = test.merge(songs , on='song_id' , how='left')
test = test.merge(members , on = 'msno' , how = 'left')
test =  test.merge(sei , on = 'song_id' , how = 'left')
test['song_length'].fillna(test['song_length'].mean() , inplace = True)
test['song_length'] = test['song_length'].astype(np.uint32)
test['language'].fillna(test['language'].mode().values[0] , inplace= True)
test['language'] = test['language'].astype(np.int8)

train['song_length'].fillna(train['song_length'].mean() , inplace = True)
train['song_length'] = train['song_length'].astype(np.uint32)
train['language'].fillna(train['language'].mode().values[0] , inplace= True)
train['language'] = train['language'].astype(np.int8)

print(train.columns)
train.head()
train['genre_ids'].fillna('no_genre_id' , inplace= True)
train['genre_ids_count'] = train['genre_ids'].apply(lambda x: genre_count(x)).astype(np.int8)
test['genre_ids'].fillna('no_genre_id' , inplace= True)
test['genre_ids_count'] = test['genre_ids'].apply(lambda x: genre_count(x)).astype(np.int8)
def artist_count(art) -> int:
    """
    get number of artists
    """
    return 0 if art == 'no_artist_name' else art.count('|')+art.count('/') + art.count('//') + art.count(';') + 1

train['artist_name'].isnull().value_counts()
train['artist_name'].fillna('no_artist_name' , inplace = True)
train['artist_count'] = train['artist_name'].apply(lambda x : artist_count(x)).astype(np.int8)
test['artist_name'].fillna('no_artist_name' , inplace = True)
test['artist_count'] = test['artist_name'].apply(lambda x : artist_count(x)).astype(np.int8)
train_col = list(train.columns)
test_col = list(test.columns)

label_encoded = ['source_system_tab', 'source_screen_name',
       'source_type','gender']
drop = ['msno', 'song_id' , 'isrc','artist_name',
       'composer', 'lyricist','name','genre_ids']
min_max_scaling = ['number_of_time_played', 'user_activity_msno','membership_days', 'song_length']

for feature in label_encoded:
    le = LabelEncoder()
    le.fit(train[feature].values + test[feature].values)
    train[f] = le.transform(list(train[feature].values))
    test[f] = le.transform(list(test[feature].values))

for feature in min_max_scaling:
    ms = MinMaxScaler()
    train[f] = ms.fit_transform(train[[feature]])
    test[f] = ms.transform(test[[feature]])
for col in train.columns:
    if train[col].dtype == object:
        train[col] = train[col].astype('category')
        test[col] = test[col].astype('category')
print(train.columns)
train.head(10)
X_train = train.drop(columns = ['target'] , axis = 1)
Y_train = train['target'].values
X_test = test.drop(columns = ['id'] , axis = 1)

ids = test['id'].values
train_set = lgb.Dataset(X_train , Y_train)
model = lgb.train({
    'objective': 'binary',
    'metric': 'binary_logloss',
    'boosting': 'gbdt',
    'learning_rate': 0.3 ,
    'verbose': 0,
    'num_leaves': 108,
    'bagging_fraction': 0.95,
    'bagging_freq': 1,
    'bagging_seed': 1,
    'feature_fraction': 0.9,
    'feature_fraction_seed': 1,
    'max_bin': 256,
    'max_depth': 10,
    'num_rounds': 200,
    'metric' : 'auc'
}, train_set=train_set,  valid_sets=train_set, verbose_eval=5)
predictions = model.predict(X_test)
print(predictions)
print('Saving Predictions')
submission = pd.DataFrame({
    'id': ids,
    'target': pred_test
})
submission['target'] = pred_test
submission.to_csv('submission.csv' , index = False , float_format ='%.5f' )
submission.head()<jupyter_output>[0.38419875 0.31085878 0.19812878 ... 0.42768898 0.38944534 0.39090245]
Saving Predictions
{'fraction_non_alphanumeric': 0.16075242319697716, 'fraction_numerical': 0.032528339083292265, 'mean_word_length': 5.0182896688087, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '275252', 'license': 'MIT', 'n_tokens_mistral': 4651, 'n_tokens_neox': 4367, 'n_words': 862, 'path': 'assignments/final_project/src/WSDM_Analysis.ipynb', 'repo_name': 'jschmidtnj/ma544-final-project', 'size': '12195'}