<jupyter_start><jupyter_text>About this notebookThis notebook contains brief and general description of the dataset. We use this one to know what features we can use to answer our research questions.**Note that at this point, we peek to raw dataset in pickle format, no other processing has been made**  Libraries<jupyter_code>import pandas as pd
from IPython.display import display<jupyter_output><empty_output><jupyter_text>Vars<jupyter_code>pickle_dataset_dir = "./datasets/pickle_files"<jupyter_output><empty_output><jupyter_text>Functions Function to observe dataset structure in general<jupyter_code>def debug_dataframe(df):
    print("Dataset size: ", df.shape)
    print("Dataset columns: ", df.columns)
    print("\nPreview:")
    display(df.head(3))<jupyter_output><empty_output><jupyter_text>Function to read one single dataset pickle file<jupyter_code>def read_dataset(dataset_file):
    df = pd.read_pickle(pickle_dataset_dir + "/df_pickle_" + dataset_file)
    return df<jupyter_output><empty_output><jupyter_text>Main  1. Million Song Dataset (MSD)Million song dataset consists of the following:1. msd_songs : one million songs data with main feature columns2. msd_summary : one million songs data with aside feature columns3. msd_artist_location : contains location per artist_id, with latitude and longitude entries4. msd_artist_mbtag : contains mbtag of artist_id, note that artist_id is not unique5. msd_artist_similarity : contains artist similarity of certain artis_id, note that artist_id is not unique6. msd_artist_term : contains terms of artist_id, note that artist_id is not unique7. msd_tracks_per_year : contains summary of msd_songs; with only 'year', 'track_id', 'artist_name', 'song_title' columns8. msd_unique_artist : list of all unique artists in msd dataset9. msd_unique_tracks : list of all unique tracks in msd dataset  1.1 msd_songs<jupyter_code>df_msd_songs = read_dataset("msd_songs")
debug_dataframe(df_msd_songs)<jupyter_output>Dataset size:  (1000000, 11)
Dataset columns:  Index(['track_id', 'title', 'song_id', 'release', 'artist_id', 'artist_mbid',
       'artist_name', 'duration', 'artist_familiarity', 'artist_hotttnesss',
       'year'],
      dtype='object')

Preview:<jupyter_text>1.2 msd_summary<jupyter_code>df_msd_summary = read_dataset("msd_summary")
debug_dataframe(df_msd_summary)<jupyter_output>Dataset size:  (1000000, 53)
Dataset columns:  Index(['analysis_sample_rate', 'audio_md5', 'danceability', 'duration',
       'end_of_fade_in', 'energy', 'idx_bars_confidence', 'idx_bars_start',
       'idx_beats_confidence', 'idx_beats_start', 'idx_sections_confidence',
       'idx_sections_start', 'idx_segments_confidence',
       'idx_segments_loudness_max', 'idx_segments_loudness_max_time',
       'idx_segments_loudness_start', 'idx_segments_pitches',
       'idx_segments_start', 'idx_segments_timbre', 'idx_tatums_confidence',
       'idx_tatums_start', 'key', 'key_confidence', 'loudness', 'mode',
       'mode_confidence', 'start_of_fade_out', 'tempo', 'time_signature',
       'time_signature_confidence', 'track_id', 'analyzer_version',
       'artist_7digitalid', 'artist_familiarity', 'artist_hotttnesss',
       'artist_id', 'artist_latitude', 'artist_location', 'artist_longitude',
       'artist_mbid', 'artist_name', 'artist_playmeid', 'genre',
       'idx_artist_terms', 'idx_sim[...]<jupyter_text>1.3 msd_artist_location<jupyter_code>df_msd_artist_location= read_dataset("msd_artist_location")
debug_dataframe(df_msd_artist_location)<jupyter_output>Dataset size:  (13850, 5)
Dataset columns:  Index(['artist_id', 'lat', 'long', 'Name', 'Location'], dtype='object')

Preview:<jupyter_text>1.4 msd_artist_mbtag<jupyter_code>df_msd_artist_mbtag = read_dataset("msd_artist_mbtag")
debug_dataframe(df_msd_artist_mbtag)<jupyter_output>Dataset size:  (24777, 2)
Dataset columns:  Index(['artist_id', 'mbtag'], dtype='object')

Preview:<jupyter_text>1.5 msd_artist_similarity<jupyter_code>df_msd_artist_similarity = read_dataset("msd_artist_similarity")
debug_dataframe(df_msd_artist_similarity)<jupyter_output>Dataset size:  (2201916, 2)
Dataset columns:  Index(['artist_id', 'similliar_artist_id'], dtype='object')

Preview:<jupyter_text>1.6 msd_artist_term<jupyter_code>df_msd_artist_term = read_dataset("msd_artist_term")
debug_dataframe(df_msd_artist_term)<jupyter_output>Dataset size:  (1109381, 2)
Dataset columns:  Index(['artist_id', 'term'], dtype='object')

Preview:<jupyter_text>1.7 msd_tracks_per_year<jupyter_code>df_msd_tracks_per_year = read_dataset("msd_tracks_per_year")
debug_dataframe(df_msd_tracks_per_year)<jupyter_output>Dataset size:  (515576, 4)
Dataset columns:  Index(['year', 'track_id', 'artist_name', 'song_title'], dtype='object')

Preview:<jupyter_text>1.8 msd_unique_artists<jupyter_code>df_msd_unique_artists = read_dataset("msd_unique_artists")
debug_dataframe(df_msd_unique_artists)<jupyter_output>Dataset size:  (44745, 4)
Dataset columns:  Index(['artist_id', 'artist_mbid', 'track_id', 'artist_name'], dtype='object')

Preview:<jupyter_text>1.9 msd_unique_tracks<jupyter_code>df_msd_unique_tracks = read_dataset("msd_unique_tracks")
debug_dataframe(df_msd_unique_tracks)<jupyter_output>Dataset size:  (1000000, 4)
Dataset columns:  Index(['track_id', 'song_id', 'artist_name', 'song_title'], dtype='object')

Preview:<jupyter_text>2. Secondhand Song Dataset (SHS)After careful analysis, we decide to not utilize this dataset as it does not support our main research questions.  2.1 SHS - Not Used<jupyter_code># Not used<jupyter_output><empty_output><jupyter_text>3. MusicX Match (MXM)MusicX match dataset consists of the following:1. mxm_lyrics : contains detailed lyrics of a song, per word count per song. (track_id) is not unique  3.1 mxm_lyrics<jupyter_code>df_mxm_lyrics = read_dataset("mxm_lyrics")
debug_dataframe(df_mxm_lyrics)<jupyter_output>Dataset size:  (19045332, 5)
Dataset columns:  Index(['track_id', 'mxm_tid', 'word', 'count', 'is_test'], dtype='object')

Preview:<jupyter_text>4. Lastfm Dataset (LASTFM)Lastfm dataset consists of the following:1. lastfm_similars : contains similars tracks (with score) relative to "track_id"  4.1 lastfm_similars<jupyter_code>df_lastfm_similars = read_dataset("lastfm_similars")
debug_dataframe(df_lastfm_similars)<jupyter_output>Dataset size:  (584897, 2)
Dataset columns:  Index(['track_id', 'similars_entry'], dtype='object')

Preview:<jupyter_text>5. Taste ProfileTaste profile dataset consists of the following:1. train_triplets : contains per song clicks per user. Note that (user_id) is not unique  5.1 train_triplets<jupyter_code>df_train_triplets = read_dataset("train_triplets")
debug_dataframe(df_train_triplets)<jupyter_output>Dataset size:  (48373586, 3)
Dataset columns:  Index(['user_id', 'song_id', 'play_count'], dtype='object')

Preview:
{'fraction_non_alphanumeric': 0.11120892018779342, 'fraction_numerical': 0.020246478873239437, 'mean_word_length': 8.125836680053547, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '650535', 'license': 'MIT', 'n_tokens_mistral': 2540, 'n_tokens_neox': 2448, 'n_words': 491, 'path': 'project/00raw_datasets_observation.ipynb', 'repo_name': 'sanadhis/ADA-2017', 'size': '6816'}