<jupyter_start><jupyter_text>Spotify project  Dataset DescriptionThe dataset I will be using for this analysis is from Kaggle and can be accessed here: https://www.kaggle.com/zaheenhamidani/ultimate-spotify-tracks-dbMost of the columns are self-explanatory, and represent a feature of a song (e.g. genre, name, loudness, acousticness, duration, etc.) Popularity column is an integer number between 0 and 100.The goal of this project is to predict how popular a song would be, based on its features. Spotify might be interested in popularity prediction to decide which songs to recommend to their users. Moreover, this analysis would help them make data-driven decisions when deciding things like how much to pay for song licenses.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import seaborn as sns
import sklearn as skl
from scipy.stats import norm


songs = pd.read_csv('/Users/aliyazhaxylykbayeva/Documents/GitHub/personal-projects/SpotifyFeatures.csv')
songs = songs.dropna(thresh=1)
songs.head()<jupyter_output><empty_output><jupyter_text>Data Exploration and CleaningI'll do some data exploration and clean the dataset before analyzing it any further to get a better idea of what I'm working with.<jupyter_code>genres = songs.groupby('genre').count()
genres.head()
genres = genres.dropna(thresh=1)
genres = genres[['artist_name']]
genres = genres.rename(columns={'artist_name':'count'})
genres.plot.bar()
songs = songs.replace("Children’s Music", "Children's Music")
genres = songs.groupby('genre').count()
genres = genres[['artist_name']]
genres = genres.rename(columns={'artist_name':'count'})
genres.plot.bar()
x = songs[['popularity','danceability','duration_ms','energy','instrumentalness','loudness','liveness','speechiness','tempo', 'valence']]

corr_mat = x.corr()
f, ax = plt.subplots(figsize =(7, 7)) 
sns.heatmap(corr_mat)<jupyter_output><empty_output><jupyter_text>Random ForestI'll train the algorithm and fit the model to get a better idea of feature importance when it comes to predicting popularity.  Edit: As I was doing this part and ran two random forests and CV Grid Search, I realized that the most logical approach was to analyze this dataset splitting it up by genre. People who are into Electronic Dance Music probably care about danceability the most, while people who are into classical music put more importance on instrumentalness.<jupyter_code>y = songs['popularity']
x = songs[['danceability','duration_ms','energy','instrumentalness','loudness','liveness','speechiness','tempo', 'valence']]
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=1)
randomforest = RandomForestRegressor(max_depth=10, random_state=0, n_estimators=200, n_jobs=-1)
randomforest.fit(x_train, y_train)
yhat_test=randomforest.predict(x_test)
res_hat = yhat_test-y_test
rmse1=np.sqrt(sum(res_hat**2)/len(yhat_test))<jupyter_output><empty_output><jupyter_text>Root mean squared error is 14.6878, which, in my opinion, is very good given that we only analyzed the song by its quantitative features.<jupyter_code>y.describe()
randomforest.feature_importances_<jupyter_output><empty_output><jupyter_text>Based on feature importance, I'll engineer new features that will allow for a more accurate prediction. In this case, duration, loudness, and speechiness are the most important.<jupyter_code>x = x.assign(loudness_duration=(x['loudness']*x['duration_ms']))
x = x.assign(loudness_speechiness=(x['loudness']*x['speechiness']))
x = x.assign(speechiness_duration=(x['speechiness']*x['duration_ms']))
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=1)
randomforest2 = RandomForestRegressor(max_depth=10, random_state=0, n_estimators=200, n_jobs=-1)
randomforest2.fit(x_train, y_train)
yhat_test=randomforest2.predict(x_test)
res_hat = yhat_test-y_test
rootmeansq=np.sqrt(sum(res_hat**2)/len(yhat_test))<jupyter_output><empty_output><jupyter_text>Analyzing popularity patterns by genreI could see from running 2 random forests that accuracy of the predictions was not getting significantly better with the engineered features, so I decided to analyze each genre on its own.<jupyter_code>genre=np.unique(songs['genre'])
rmse2=[]
for i in genre:
    temp=songs[songs.genre == i]
    y = temp['popularity']
    x = temp[['danceability','duration_ms','energy','instrumentalness','loudness','liveness','speechiness','tempo', 'valence']]
    x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=1)
    randomforest = RandomForestRegressor(max_depth=10, random_state=0, n_estimators=200, n_jobs=-1)
    randomforest.fit(x_train, y_train)
    yhat_test=randomforest.predict(x_test)
    res_hat = yhat_test-y_test
    rmse2.append((round(np.sqrt(sum(res_hat**2)/len(yhat_test)),3)))
genre=genre.tolist()
genre.append("Overall")
rmse2.append(rmse1)
import seaborn
import matplotlib

plt.figure(figsize=(10, 15))
seaborn.barplot(x=rmse2,y=genre)<jupyter_output><empty_output><jupyter_text>Even with less data in each training set, when analyzing by genre, the testing error (RMSE) was lower than the overall error for every single genre. While some genres like Children's Music or Reggaeton are harder to predict with high accuracy, this experiment has proven that it is very important to take descriptive features like genre into account.Now, I'd like to ask another interesting question - what is the effect of each song characteristic on its popularity on average. To perform that analysis, I'll use linear regression.<jupyter_code>import statsmodels.api as sm

# Add important features
x=songs[['danceability','duration_ms','energy','instrumentalness','loudness','liveness','speechiness','tempo', 'valence']]
x['duration']=x['duration_ms'].div(60000)
# Popularity is likely nonlinear in duration, so I added a column with squared values of duration.
x['duration_sq']= np.square(x['duration'])
x=x.drop(axis=1,columns='duration_ms')

y=songs['popularity']

# Train and fit linear regression
lm=sm.OLS(exog=x, endog=y, hasconst=True)
lm_res = lm.fit()
lm_res.summary()<jupyter_output>/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """
/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  import sys
{'fraction_non_alphanumeric': 0.10869877785765636, 'fraction_numerical': 0.010927390366642703, 'mean_word_length': 6.842164599774521, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '502047', 'license': 'Unlicense', 'n_tokens_mistral': 2325, 'n_tokens_neox': 2218, 'n_words': 636, 'path': 'Spotify Project.ipynb', 'repo_name': 'zhaxylykbayeva/zhaxylykbayeva.github.io', 'size': '6955'}