<jupyter_start><jupyter_text>"Spotify genre popularity prediction"> "<NAME>"- toc: true- branch: master- badges: true- comments: true- categories: [fastpages, jupyter]- image: images/some_folder/your_image.png- hide: false- search_exclude: true- metadata_key1: metadata_value1- metadata_key2: metadata_value2  0 PROLOGUE Data Collection:This Dataset was collected from Kaggle.com- Link: https://www.kaggle.com/iamsumat/spotify-top-2000s-mega-dataset Context- This dataset contains audio statistics of the top 2000 tracks on Spotify. The data contains 15 columns each describing the track and it's qualities. Songs released from 1956 to 2019. AcknowledgementsThis data is extracted from the Spotify playlist - Top 2000s on PlaylistMachinery(@plamere) using Selenium with Python. More specifically, it was scraped from http://sortyourmusic.playlistmachinery.com/.  Content Variables1. Index: ID2. Title: Name of the Track3. Artist: Name of the Artist4. Top Genre: Genre of the track5. Year: Release Year of the track6. Beats per Minute(BPM): The tempo of the song7. Energy: The energy of a song - the higher the value, the more energtic. song8. Danceability: The higher the value, the easier it is to dance to this song.9. Loudness: The higher the value, the louder the song.10. Valence: The higher the value, the more positive mood for the song.11. Length: The duration of the song.12. Acoustic: The higher the value the more acoustic the song is.13. Speechiness: The higher the value the more spoken words the song contains15. Popularity: The higher the value the more popular the song is. Goals:1. Most popular Genres and Artists of all time from 1950s to 2000s?2. Is there a trend in genres preferred back in the day vs now?3. What other variables have an impact on the most popular genres? 4. The goal is to build a predictive model to determine song/genre popularity  1 IMPORTING LIBRARIES<jupyter_code>import pandas as pd
import numpy as np
from scipy import stats, special

import seaborn as sns
from seaborn import pairplot, heatmap
import matplotlib.pyplot as plt

from sklearn import model_selection, metrics, linear_model, tree, datasets, feature_selection
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.naive_bayes import GaussianNB<jupyter_output><empty_output><jupyter_text>2 DATA DESCRIPTION & CLEANING 2.1 DATA DESCRIPTION<jupyter_code># Loading datadataset

spotify_df = pd.read_csv('data/Spotify-2000.csv') 
# Visual Display of the first 5 lines of the dataset

spotify_df.head()
# Overview of Dataset information and data types

spotify_df.info()  
# Overview of Dataset numerical data

spotify_df.describe()
# Number of genres that have featured in the all time top 2000.

len(spotify_df["Top Genre"].unique())
# Number of times each genre features in the all time top 2000.

spotify_df["Top Genre"].value_counts()[:25].plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Raw Dataset  Summary: The Dataset contains:- 1994 entries- 1994 non-null entries- 15 total variable columns- 149 Genre entries Data Types:- 4 categorical columns- 11 numerical columns Numerical Data:- The data set is between years 1994 - 2019. Just about 63 years worth of most popular songs as classified by spotify..- min Popularity of a song is 11 and max is 100. Initial Analysis & Progression:1. It's clear that Rock music seems to be the all time favourite genre with the most features. But that being said the data is lob-sidded towards pre-2000s and music taste does tend to change over the years so this needs to be investigated further in the EDA.  2.2 Data Cleaning Action:1. Convert column data types.2. Remove unecessary columns. 3. Adjust column titles.4. consolidate genre column as there are many variations of a single genre e.g. 'dutch pop' and 'dance pop' or 'album rock'and alternative rock. we will make these columns just 'pop' or just 'rock' as to provide a more accurate summarised representation of that genres.<jupyter_code>#Converting Length (Duration) to an integer data type

spotify_df.replace(',','', regex=True, inplace=True)
spotify_df['Length (Duration)']= spotify_df['Length (Duration)'].apply(pd.to_numeric,errors='coerce')
print('Length (Duration) is now a -->',spotify_df['Length (Duration)'].dtype, 'data type')
#Removing the Index column.
spotify_df.drop(columns = ['Index'], inplace = True)

#Converting all column titles to lowercase.
spotify_df.columns = map(str.lower, spotify_df.columns)

#Coverting column names to have no space between, if they do, replace space with period "."
spotify_df.rename(columns = {'top genre' : 'genre', 'beats per minute (bpm)':'beats_per_minute','loudness (db)': 'loudness','length (duration)': 'duration'}, inplace = True)<jupyter_output><empty_output><jupyter_text><jupyter_code>spotify_df.info()
spotify_df.head(3)<jupyter_output><empty_output><jupyter_text>Consolidating genre column<jupyter_code># function to split the genre column
    
def genre_splitter(genre):
    result = genre.copy()
    result = result.str.split(" ",1)
    for i in range(len(result)):
        if (len(result[i]) > 1):
            result[i] = [result[i][1]]
    return result.str.join('')

#loop until the genre cannot be split any further

new_genre = spotify_df['genre'].copy()
while(max((new_genre.str.split(" ", 1)).str.len()) > 1):
    new_genre = genre_splitter(new_genre)
    
print('New Total of Genres from 146 to -->', len(new_genre.unique()))
print(new_genre.value_counts())
#inputting new column values from new_genre to genre in dataframe.

spotify_df['genre'] = new_genre
spotify_df['genre']<jupyter_output><empty_output><jupyter_text>3 EXPLORATORY ANALYSIS & VISUALISATIONS 3.1 Most popular Genres & Artists from 1950s to 2000s?<jupyter_code># Create a function top_10, which takes a single parameter for a column. 
# Group the data by the desired column input, sum the values the remaining columns, sort sum values by 'Popularity' column 
# from highest to lowest, the print the top 10 rows. 

def top_10(column):
    top_10_songs = spotify_df.groupby([column]).sum().sort_values('popularity', ascending=False).head(10)
    return(top_10_songs[['popularity']])  # Only show 'popularity' column.

top_10('genre')
top_10('artist')
# Shows the number of data entries for each genre

spotify_df['genre'].value_counts().head()<jupyter_output><empty_output><jupyter_text>Conclusion:- The above shows the accumulation of most popular Genres & Artists of all time according to the spotify data set. We can also see there has a significant amounts of Rock entries compared to the rest of the genres. So there was always a high possibility this will be the most popular overall.  - But in terms of 'pure' popularity as seen below, values from 0-100, the story is a little different. We can see that pop music and its many variations has a majority in populularity in more recent years. This may indicate a shift in popularity over the years as well as musical listening trends due to advancements in technology.<jupyter_code>pure_popularity = spotify_df.sort_values('popularity', ascending=False).head(10)
pure_popularity[['genre', 'year', 'popularity']]<jupyter_output><empty_output><jupyter_text>3.2 Is there a trend/shift in genres prefered to pre-2000s vs now over the years?- To approach this, I will split the dataset into quarters (n/4) and track the popularity change in genre, as well as the genre entry count over the years.   1/4 of dataset - Years 1956-1979<jupyter_code>#spotify_df.sort_values('year').iloc[0:500,:] - Placeholder to check year span.

def year_block(index_1,index_2):
    block = spotify_df.sort_values('year').iloc[index_1:index_2,:]
    return block[['genre', 'year', 'popularity']].sort_values('popularity', ascending=False)
   #return block[['genre', 'year', 'popularity']].groupby('genre').sum().sort_values('popularity', ascending=False).head()
    
year_block_1 = year_block(0,500) 
year_block_1.head()<jupyter_output><empty_output><jupyter_text>Genre count<jupyter_code>def genre_count(year_block):
    return year_block['genre'].value_counts().head()
    
genre_count(year_block_1)<jupyter_output><empty_output><jupyter_text>Pie chart - year span 1956-1979:<jupyter_code># Pie chart, indicating genre count between year span 1956-1979:

sizes = genre_count(year_block_1).values
labels = genre_count(year_block_1).index
explode = [0.1,0,0,0,0]

plt.pie(sizes, labels=labels, explode=explode, shadow=True)

plt.title('1/4 Dataset - Genre Count - Years 1956-1979')

plt.show()<jupyter_output><empty_output><jupyter_text>2/4 of dataset - Years 1979- 1994<jupyter_code>#spotify_df.sort_values('year').iloc[501:1001,:] - Placeholder to check year span.
    
year_block_2 = year_block(501,1001) 
year_block_2.head()<jupyter_output><empty_output><jupyter_text>Genre count<jupyter_code>genre_count(year_block_2)<jupyter_output><empty_output><jupyter_text>Pie chart - year span 1979- 1994:<jupyter_code>sizes = genre_count(year_block_2).values
labels = genre_count(year_block_2).index
explode = [0.1,0,0,0,0]

plt.pie(sizes, labels=labels, explode=explode, shadow=True)

plt.title('2/4 Dataset - Genre Count - Years 1979-1994')

plt.show()<jupyter_output><empty_output><jupyter_text>3/4 of dataset - Years 1994 - 2008<jupyter_code>#spotify_df.sort_values('year').iloc[1001:1501,:] - Placeholder to check year span.
    
year_block_3 = year_block(1001,1501) 
year_block_3.head()<jupyter_output><empty_output><jupyter_text>Genre count<jupyter_code>genre_count(year_block_3)<jupyter_output><empty_output><jupyter_text>Pie chart - year span 1994-2008:<jupyter_code>sizes = genre_count(year_block_3).values
labels = genre_count(year_block_3).index
explode = [0.1,0,0,0,0]

plt.pie(sizes, labels=labels, explode=explode, shadow=True)

plt.title('3/4 Dataset - Genre Count - Years 1994 - 2008')

plt.show()<jupyter_output><empty_output><jupyter_text>4/4 of dataset - Years 2008 - 2019<jupyter_code>#spotify_df.sort_values('year').iloc[1501:,:] - Placeholder to check year span.
    
year_block_4 = year_block(1501,1993) 
year_block_4.head()<jupyter_output><empty_output><jupyter_text>Genre count<jupyter_code>genre_count(year_block_4)<jupyter_output><empty_output><jupyter_text>Pie chart - year span 2008-2019:<jupyter_code>sizes = genre_count(year_block_4).values
labels = genre_count(year_block_4).index
explode = [0.1,0,0,0,0]

plt.pie(sizes, labels=labels, explode=explode, shadow=True)

plt.title('4/4 Dataset - Genre Count - Years 2008-2019')

plt.show()<jupyter_output><empty_output><jupyter_text>Analysis:- As per above, we can clearly see a shift in genre preference, from Rock leaning into pop music. The level of popularity for pop music is also much higher than that of rock music at it's peak.  3.3 What other features have an impact on the most popular genres?- We shall look to explore numerical variables such as the audio features and look for correlation bewteen each feature in order to help define usable features to build a model. - First let's remove the 'artist' column as all values are unique and will not help with our classification model.<jupyter_code>spotify_df.drop(columns = ['artist','title','year'], inplace = True)
spotify_df.head()<jupyter_output><empty_output><jupyter_text>Note:- Music genres that have a single values would make our model inefficient, since it does not have enough data to work off of, so these values and corresponding rows in the original dataframe will be removed. - Putting all of these instances into a broader "Other" category is another potential solution, but I decided against it because there is probably minimal similarity between all of these music genres.  3.3.1  Genre's with a value count less than 20x shall be removed<jupyter_code>unique = spotify_df['genre'].unique()
to_remove = [] 

# genres that have a single instance only will be placed within the to_remove array
for genre in unique:
    if spotify_df['genre'].value_counts()[genre] < 20: 
        to_remove += [genre]

print('Genre Values to be removed from data set =', len(to_remove))<jupyter_output>Genre Values to be removed from data set = 56<jupyter_text>Now to replace our original genre column with the updated version<jupyter_code>spotify_df.set_index(["genre"],drop = False, inplace = True)
for name in to_remove:
    type(name)
    spotify_df.drop(index = str(name), inplace = True)
    
spotify_df['genre'].value_counts()
plt.figure(figsize=(20,10))
sns.heatmap(spotify_df.corr(),annot=True,cmap='BrBG')
plt.show()<jupyter_output><empty_output><jupyter_text>Analysis: Target Feature: 'Popularity'From the above heat map we can see that the strongest features that correlate with Popularity is:- loudness (17%)- daceability (13%)- energy (12%)- valence (10%) Variable relationship: We can see the strongest relationship between the variables excl. paopularity are:- loudness & energy (74%)- valence & danceability (53%)- valence & energy (43%)  3.3.2  As this is a dataset of already the most popular songs of all time up now, we will classify the target feature: 'popularity' into bins, to help better classify our data for our predictive modeling.- 1 = 'most popular' - 0 = 'least popular'<jupyter_code>spotify_df['popularity'] = pd.qcut(spotify_df['popularity'], q=2, labels=[0, 1]) 
spotify_df.head()<jupyter_output><empty_output><jupyter_text>Now we need to one hot-encode (get dummy variables) for the genre feature:<jupyter_code>spotify_df = spotify_df.reset_index(drop=True)
spotify_df
def dummies(df, column, prefix):
    df = df.copy()
    dummies = pd.get_dummies(df[column], prefix=prefix)
    df = pd.concat([df, dummies], axis=1)
    df = df.drop(column, axis=1)
    return df
    
spotify_df = dummies(spotify_df, 'genre', 'genre')
spotify_df.head()<jupyter_output><empty_output><jupyter_text>Splitting & Scaling Data<jupyter_code>y = spotify_df.loc[:'popularity']
X = spotify_df.drop('popularity', axis=1)
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=20)
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)<jupyter_output>(1382, 21) (1382,)
(346, 21) (346,)<jupyter_text>Modeling & TrainingDue to the nature of the dataset, we will be implementing classification type machine learning:- LogisticRegression- DecisionTreeClassifier- Naive Bayes<jupyter_code># Creating Models

logr_model = linear_model.LogisticRegression()
dtree_model = tree.DecisionTreeClassifier()
nb_model = GaussianNB()
# Training the models

logr_model.fit(X_train, y_train)
dtree_model.fit(X_train, y_train)
nb_model.fit(X_train, y_train)
# Accuracy of trained models with training data

logr_train_acc = logr_model.score(X_train, y_train)
dtree_train_acc = dtree_model.score(X_train, y_train)
nb_train_acc = nb_model.score(X_train, y_train)

print(round(logr_train_acc*100,2))
print(round(dtree_train_acc*100,2))
print(round(nb_train_acc*100,2))
# Validating models with k-fold cross validation method

kf = model_selection.KFold(n_splits=10, shuffle=True, random_state=24)

accuracy_logr = cross_val_score(logr_model, X_train, y_train, scoring="accuracy", cv=kf)
accuracy_dtree = cross_val_score(dtree_model, X_train, y_train, scoring="accuracy", cv=kf)
accuracy_nb = cross_val_score(nb_model, X_train, y_train, scoring="accuracy", cv=kf)
# Accuracy of Models on train data with K-fold Cross Validation

print("Logistic Regression model with Cross Validation:", round(accuracy_logr.mean()*100,2),"%")
print("Decision Tree model with Cross Validation:", round(accuracy_dtree.mean()*100,2),"%")
print("Naive Bayes model with Cross Validation:", round(accuracy_nb.mean()*100,2),"%")
objects = ('Logistic Regression', 'Decision Tree Classifier', 'Naive Bayes')
y_pos = np.arange(len(objects))
performance = [62.81,57.46,62.37]

plt.bar(y_pos, performance, align='center')
plt.xticks(y_pos, objects)
plt.ylabel('Accuracy')
plt.title('Accuracy of Test Data Model')

plt.show()<jupyter_output><empty_output><jupyter_text>Conslusion - Our chosen model is Logistic Regression, so let's evaluate our trained model on the test data set:<jupyter_code># using the chosen Logistic Regression model due to higher accuracy:
round(logr_model.score(X_test, y_test)*100,2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10369187296516985, 'fraction_numerical': 0.030222986669942872, 'mean_word_length': 5.939471440750213, 'pattern_counts': {'":': 0, '<': 109, '<?xml version=': 0, '>': 113, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '180647', 'license': 'Apache-2.0', 'n_tokens_mistral': 5663, 'n_tokens_neox': 5220, 'n_words': 1655, 'path': '_notebooks/2021-09-08-tl-spotify-genre-and-popularity-prediction.ipynb', 'repo_name': 'snafis/data-projects-portfolio', 'size': '16287'}