<jupyter_start><jupyter_text>importSpotifyData - This notebook is built upon the an eaieler project where I pulled simmilar data from the Spotify Web API: [getSpotifyData.ipynb](https://github.com/sreidy/DATA601_HW2/blob/main/getSpotifyData.ipynb) Table Of Contents   Importing Packages<jupyter_code>import spotipy
import pandas as pd
from spotipy.oauth2 import SpotifyClientCredentials


import getpass
import pickle
import json
import sys
def drawProgressBar(percent, barLen = 20):
    # percent float from 0 to 1. 
    sys.stdout.write("\r")
    sys.stdout.write("[{:<{}}] {:.0f}%".format("=" * int(barLen * percent), barLen, percent * 100))
    sys.stdout.flush()<jupyter_output><empty_output><jupyter_text>Seting up Spotify Web API Credentials & Using Spotipy - We will need the following:     + [A spotify Account](https://developer.spotify.com/)     + Create an App in the [Spotify Developer Dashboard](https://developer.spotify.com/dashboard/applications)      + Within the newly create Spotify App         + Spotify Client ID          + Spotify Client Secret<jupyter_code>CLIENT_ID = getpass.getpass(prompt = "your-spotify-client-id")
CLIENT_SECRET = getpass.getpass(prompt = "your-spotify-client-secret")<jupyter_output>your-spotify-client-secret········<jupyter_text>Get Oauth Token<jupyter_code>import warnings;
warnings.filterwarnings('ignore');

token = SpotifyClientCredentials(client_id=CLIENT_ID, client_secret=CLIENT_SECRET)
cache_token = token.get_access_token()
#save token as sp, will be used in future API calls
sp = spotipy.Spotify(cache_token)<jupyter_output><empty_output><jupyter_text>Test Spotify Web API Token<jupyter_code># Enter artist name 
artist_name = input("Name of Artist to Search: ")
#Search for an Artists and print out 20 popular tracks 
results = sp.search(q=artist_name, limit=20)
for i, t in enumerate(results['tracks']['items']):
    print(' ', i, t['name'])<jupyter_output>0 My Bloody Valentine
  1 When You Sleep
  2 Only Shallow
  3 Sometimes
  4 Soon
  5 To Here Knows When
  6 Loomer
  7 I Only Said
  8 Only Shallow
  9 Come in Alone
  10 Touched
  11 What You Want
  12 Blown a Wish
  13 Lose My Breath
  14 My Bloody Valentine
  15 Off Your Face
  16 Soft as Snow (But Warm Inside)
  17 Cupid Come
  18 Honey Power (EP Version)
  19 (When You Wake) You're Still in a Dream<jupyter_text>Geting Audio Feature Data  - Expanding upon the code I wrote for an [ealier project](https://github.com/sreidy/DATA601_HW2/blob/main/getSpotifyData.ipynb)  - Spotify handles genre by providing a list of genres that a particualr artists is associated with      + from the spotify documentaiton              ... list of the genres the artist is associated with. For example: "Prog Rock" , "Post-Grunge". (If not yet classified, the array is empty.)  - We will need to find a way to manualy group simmilar genres and also use a the Artist_id of a track to pull and data about genre   Audio Features The Spotify Web Api provided the following Audio Features for each track [more info here](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/)- track_id : a spotify primary key; unique for each track- artist: name of artist- album: name of album- trackName: title of track- acousticness: A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.- danceability: Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. - duration_ms: The duration of the track in milliseconds.- energy: a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. - instrumentalness: Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context.- key: The estimated overall key of the track. Integers map to pitches using standard [Pitch Class notation](https://en.wikipedia.org/wiki/Pitch_class) - liveness: Detects the presence of an audience in the recording.- loudness: The overall loudness of a track in decibels (dB). - mode: indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.- speechiness: Speechiness detects the presence of spoken words in a track.- tempo: BPM of track- time_signature: An estimated overall time signature of a track. - valence: A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. - category_id: The Spotify Category ID of the track - popularity: The value will be between 0 and 100, with 100 being the most popular. The artist’s popularity is calculated from the popularity of all the artist’s tracks.<jupyter_code>###### Example 
# a spotify playlist ID 
# Playlist ID of any playlist can be found by looking at the spotify URL string 
playlistID_1 = "37i9dQZF1DX4jsulumEbDn"

# Get All tracks in the playlist 
tracks = sp.user_playlist_tracks("spotify",playlistID_1)['items']

#get the first song in the playlist
song = tracks[0]

#get song meta data 
#spotify track_id (primary key)
track_id = song['track']['id']
track_name = song['track']['name']
# For tracks where there is more than one listed artist, we will take the first listed artist 
artist = song["track"]["album"]["artists"][0]["name"]
artist_id = song['track']['artists'][0]['id']

# Get list of genres from the artist 
genre_list = sp.artist(artist_id)['genres']


print(track_name + " : " + artist )
print(genre_list)

sp.artist(artist_id)['popularity']
sp.audio_features(track_id)
#query_spotify_playlist: Creates a dataframe of songs and assorted metadata from a playlist.

# Parameters 
# :creator: (string) username who authored playlist 
# :playlist_id: (string) a unique primary key for the playlist (found from playlist URL extention)
# :track_feature_list: (array) a list of strings of different features to be queried in dataframe 
#                      learn more about track feautres here
#                     https://developer.spotify.com/documentation/web-api/reference/tracks/get-several-audio-features/
# :metadata_list: (array) [Default] a list of mettadata to be queried, same for all tracks

# Returns 
# :playlist_df: a pandas dataframe of the playlist 
 
def query_spotify_playlist(creator, playlist_id,track_feature_list,
                           metadata_list = ["track_id","artist","artist_id","album","trackName"]):
    
    colNames = metadata_list + track_feature_list
    
    #set DataFrame for the playlist 
    playlist_df = pd.DataFrame(columns = colNames)
    
    #pull playlist data from Spotify Web API
    playlist_raw_dict = sp.user_playlist_tracks(creator,playlist_id)
    playlist = playlist_raw_dict['items']
    check_if_next = playlist_raw_dict['next']
    total_len = playlist_raw_dict['total']
    
    #Handling playlist of over 100 songs, spotify web api limits to 100 tracks at a time
    if check_if_next != None:
        spotify_offset = 100
        while check_if_next != None:
            next_playlist_chunk = sp.user_playlist_tracks(creator,playlist_id,offset = spotify_offset)
            playlist.extend(next_playlist_chunk['items'])
            check_if_next = next_playlist_chunk['next']
            spotify_offset += 100

            
            
    n = len(playlist)
    count = 0
    #loop over all items in a playlist 
    for i in playlist:
        progress = count / n 
        drawProgressBar(progress)
        
        playlist_elem = {}
        
        # Metadata (hardcoded locations of mettadata)
        playlist_elem["track_id"] = i["track"]["id"]
        playlist_elem["artist"] = i["track"]["album"]["artists"][0]["name"]
        playlist_elem["artist_id"] = i["track"]["album"]["artists"][0]["id"]
        playlist_elem["album"] = i["track"]["album"]["name"]
        playlist_elem["trackName"] = i["track"]["name"]
        
        # Artist Genre list, 
        artist = sp.artist(i["track"]["album"]["artists"][0]["id"])
        track_genre_list = artist['genres']
        playlist_elem['genre'] = str(track_genre_list)
        
        # Artist Popularity
        artist_pop = artist['popularity']
        playlist_elem['popularity'] = artist_pop
        
        # Track Audio Features
        audio_features = sp.audio_features(playlist_elem["track_id"])[0]
        for feature in track_feature_list:
            playlist_elem[feature] = audio_features[feature]
            
        #print(playlist_elem) 
        # add track data to playlist dataframe     
        track_df = pd.DataFrame(playlist_elem, index = [0])
        playlist_df = pd.concat([playlist_df, track_df], ignore_index = True)
        
        count += 1
        
    return playlist_df 
playlistID_1 = "37i9dQZF1DX4jsulumEbDn"
audioFeatureList = ["acousticness","danceability","duration_ms","energy","instrumentalness","key","liveness","loudness","mode","speechiness","tempo","time_signature","valence"]

playlist1_df = query_spotify_playlist("spotiy",playlistID_1, audioFeatureList)
playlist1_df
#geting list of genres back from data frame
eval(playlist1_df['genre'][0])<jupyter_output><empty_output><jupyter_text>- Idea for Future project: Use a differnt API to pull song lyrics<jupyter_code>playlist_of_everything = "69fEt9DN5r4JQATi52sRtq"

anti_folk_playlist = "1SXOzFKlMx7ODmxI1jnxux"

playlist_df = query_spotify_playlist("thesoundsofspotify",anti_folk_playlist,audioFeatureList)
playlist_df<jupyter_output><empty_output><jupyter_text>Simplifying Genre  - Spotify dose not curently have a large list of every genre defined on the service - For the sake of building a clasifer it would be best to simplify each track to one genre  - [Every Noise](http://everynoise.com/) visilizes all the differnt spotify genres. Wow there are alot All playlists were generated by using the Every Noise web app, by first searching the genre and then every noise would construct a spotify playlist that is representative of that genre  List of playlists for to pull<jupyter_code>Genre_list = ['pop','dance_pop','rap','rock','hip_hop','country','edm','indie_pop','indie_rock','punk','metal','alternative_rock','jazz']
audioFeatureList = ["acousticness","danceability","duration_ms","energy","instrumentalness","key","liveness","loudness","mode","speechiness","tempo","time_signature","valence"]
Every_Noise_Playlists = {
    'The Sound of Pop' : ["The Sounds of Spotify","6gS3HhOiI17QNojjPuPzqc",'pop'],
    'The Sound of Dance Pop' : ["The Sounds of Spotify","2ZIRxkFuqNPMnlY7vL54uK",'dance_pop'],
    'The Sound of Rock' : ["The Sounds of Spotify","7dowgSWOmvdpwNkGFMUs6e",'rock'],
    'The Sound of Rap' : ["The Sounds of Spotify","6s5MoZzR70Qef7x4bVxDO1", 'rap'],
    'The Sound of Hip Hop' : ["The Sounds of Spotify","6MXkE0uYF4XwU4VTtyrpfP",'hip_hop'],
    'The Sound of Country' : ["The Sounds of Spotify","4mijVkpSXJziPiOrK7YX4M",'country'],
    'The Sound of EDM' : ["The Sounds of Spotify","3pDxuMpz94eDs7WFqudTbZ",'edm'],
    'The Sound of Indie folk' : ["The Sounds of Spotify","1aYiM4zLmBuFq0Fg6NQb6a",'indie_folk'],
    'The Sound of Indie Rock' : ["The Sounds of Spotify","4XXr357Jej7eUBh7XPK8hb",'indie_rock'],
    'The Sound of Punk' : ["The Sounds of Spotify","17qQT0G3yFjOJ02wWZaNCw","punk"],
    'The Sound of Metal' : ["The Sounds of Spotify","3pBfUFu8MkyiCYyZe849Ks","metal"],
    'The Sound of Alternitive Rock' : ["The Sounds of Spotify","3dlw4x21qVajwZLPNHtS3u",'alternative_rock'],
    'The Sound of Jazz' : ["The Sounds of Spotify", "5EyFMotmvSfDAZ4hSdKrbx",'jazz']
}
#query_multiple_spotify_playlists: Takes a dictionary of playlists and returns a dictionary of pandas dataframes 

# Parameters 
# :playlists: (dict) Key -> (str) Playlist name: 
#                            [:creator: (str)username who authored playlist,
#                            playlist_id: (string) a unique primary key for the playlist (found from playlist URL extention)
#                            category_id: (string) The Spotify category ID (genera)

# :track_feature_list: (array) a list of strings of different features to be queried in dataframe 
#                      learn more about track feautres here
#                     https://developer.spotify.com/documentation/web-api/reference/tracks/get-several-audio-features/
# :metadata_list: (array) [Default] a list of mettadata to be queried, same for all tracks

# Returns 
# :playlist_dict: a dictionary of a pandas dataframe of the playlist
#                Key -> (str) Playlist name: [dataframe of tracks in playlist]


def query_multiple_spotify_playlists(playlists,track_feature_list,
                                    metadata_list = ["track_id","artist","artist_id","album","trackName"], echo = False):
    
    # Create playlist_dict to hold all output playlists 
    playlist_dict = {}
    
    #Loop over every playlist in playlists  
    for playlist in playlists.items():
        
        playlist_name = playlist[0];playlist_creator = playlist[1][0];playlist_id = playlist[1][1];playlist_category_id = playlist[1][2];
        
        # Print curent playlist being pulled from API
        if echo:
            print(playlist_name + " " + playlist_id + " " + playlist_category_id)
        
        # if API unable to find playlist data, skip playlist 
        try:
            #query track data from playlist, save as dataframe 
            playlist_df = query_spotify_playlist(playlist_creator, playlist_id,track_feature_list)
        
            #lable track data with spotify category ID (genera)
            playlist_df = playlist_df.assign(category_id = playlist_category_id )
        
            #add playlsit_df to playlist_dict
            playlist_dict[playlist_name] = playlist_df
            
        except:
            print("playlist Error: Unable to query " + playlist_name + " by " + playlist_creator)
            print(playlist_id)
            continue

        
    #check if all playlists dataframes added to dictionary     
    #assert len(playlists) == len(playlist_dict)
    #return dict of data_frames 
    return playlist_dict
    
all_playlists = query_multiple_spotify_playlists(Every_Noise_Playlists,audioFeatureList, echo = True)
spotify_df = pd.concat(all_playlists)
spotify_df.head()
# Change data types of the following Columns, duration_ms -> int, instrumentalness -> float, 
# key -> int, mode -> int, time_signature -> int
spotify_df = spotify_df.astype({'duration_ms': int, 'instrumentalness': float,'key': int,'mode': int,'time_signature': int})
spotify_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
MultiIndex: 10901 entries, ('The Sound of Pop', 0) to ('The Sound of Jazz', 397)
Data columns (total 21 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   track_id          10901 non-null  object 
 1   artist            10901 non-null  object 
 2   artist_id         10901 non-null  object 
 3   album             10901 non-null  object 
 4   trackName         10901 non-null  object 
 5   acousticness      10901 non-null  float64
 6   danceability      10901 non-null  float64
 7   duration_ms       10901 non-null  int64  
 8   energy            10901 non-null  float64
 9   instrumentalness  10901 non-null  float64
 10  key               10901 non-null  int64  
 11  liveness          10901 non-null  float64
 12  loudness          10901 non-null  float64
 13  mode              10901 non-null  int64  
 14  speechiness       10901 non-null  float64
 15  tempo             10901 non-null  float64
 1[...]<jupyter_text>Save spotify_df to csv and pickle<jupyter_code>spotify_df.to_pickle("spotify_df.pkl")
spotify_df.to_csv("spotify_df.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09592492284436606, 'fraction_numerical': 0.020721798828494047, 'mean_word_length': 3.67, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 38, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '310574', 'license': 'MIT', 'n_tokens_mistral': 5155, 'n_tokens_neox': 4888, 'n_words': 1500, 'path': 'Notebooks/importSpotifyData.ipynb', 'repo_name': 'sreidy/spotify_datascience_project', 'size': '15877'}