<jupyter_start><jupyter_text>1. Business UnderstandingThe audio streaming service was founded in 2006 by Swedish startup Spotify Technology S.A., and is available in more than 90 different countries. For the Spotify company, one of the most important potentials and challenges is to compile a unique personalized playlist for each user that matches the listener's music taste. Consideration is given to solving the above challenges by using Spotify AI and machine learning algorithms to generate a customized playlist for the user.The overarching question of this case study is: How are machine learning technologies used at Spotify to improve the customer's listening experience?  2. Data and Data UnderstandingThe collected data for Spotify is merged into a unified data set and checked for the problem to see if insights can be extracted from the information. The attributes have different ranges. This data is carefully prepared so that the final dataset can be created or processed using data mining methods. For this purpose, the data is loaded and analyzed. Subsequently, these in turn form the basis for the next phase of modeling.  2.1 Import of Relevant Modules<jupyter_code># @hidden_cell
# The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.
# from project_lib import Project
# project = Project(project_id='148a63b8-8cd0-4ae9-acd7-382d6f8da2d1', project_access_token='<KEY>')
# pc = project.project_context
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>2.2 Read Data<jupyter_code>#Fetch the local file
# my_file = project.get_file("data.csv")

#Read the CSV data file from the object storage into a pandas DataFrame
#my_file.seek(0)

#raw_data = pd.read_csv(my_file)
raw_data = pd.read_csv("https://storage.googleapis.com/ml-service-repository-datastorage/Generation_of_Individual_Playlists_Generation-of-Individual-Playlists-data.csv")

raw_data.head()
#raw_data.head(5)
#descriptive statistics for all columns
raw_data.describe(include = 'all')<jupyter_output><empty_output><jupyter_text>2.3 Data Cleaning<jupyter_code># check for duplicate rows
raw_data[raw_data.duplicated(keep = False)]<jupyter_output><empty_output><jupyter_text>Check for missing values<jupyter_code>raw_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>Select predictors<jupyter_code>#predictive modeling
#plot a correlation matrix to find out which variables are correlated to each other
f,ax=plt.subplots(figsize = (18,18))
sns.heatmap(raw_data.corr(),annot= True,linewidths=0.2, fmt = ".1f",ax=ax)
plt.xticks(rotation=90)
plt.yticks(rotation=0)
plt.title('Correlation Map')
plt.show()<jupyter_output><empty_output><jupyter_text>Drop features<jupyter_code>raw_data= raw_data.drop(['name', 'release_date'], axis=1)
raw_data = raw_data.drop(['id'], axis=1)
raw_data = raw_data.drop(['artists', 'energy'], axis=1)
raw_data = raw_data.drop(['year'], axis=1)
raw_data.head()
f,ax=plt.subplots(figsize = (18,18))
sns.heatmap(raw_data.corr(),annot= True,linewidths=0.5,fmt = ".1f",ax=ax)
plt.xticks(rotation=90)
plt.yticks(rotation=0)
plt.title('Correlation Map')
plt.show()
#Detect outliers and handle them
raw_data.hist(figsize=(25,25), bins=50)
#raw_data.describe()<jupyter_output><empty_output><jupyter_text>3. Data Preparation<jupyter_code>df_dummies = pd.get_dummies(raw_data, drop_first=True) # 0-1 encoding for categorical values
df_dummies.head()<jupyter_output><empty_output><jupyter_text>4. Data Modeling<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn import metrics
target = df_dummies['popularity']
predictors = df_dummies.drop(['popularity'], axis=1)<jupyter_output><empty_output><jupyter_text>4.1 Test and Training sets<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(predictors, target, test_size=0.2, random_state=365)<jupyter_output><empty_output><jupyter_text>Standardize columns before using models<jupyter_code>scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>4.2 Linear Regression model and evaluation<jupyter_code>reg = LinearRegression()
reg.fit(X_train, y_train)
print('training performance')
print(reg.score(X_train,y_train))
print('test performance')
print(reg.score(X_test,y_test))
y_pred = reg.predict(X_test)
test = pd.DataFrame({'Predicted':y_pred,'Actual':y_test})
fig= plt.figure(figsize=(16,8))
test = test.reset_index()
test = test.drop(['index'],axis=1)
plt.plot(test[:50])
plt.legend(['Actual','Predicted'])
sns.jointplot(x='Actual',y='Predicted',data=test,kind='reg',);<jupyter_output><empty_output><jupyter_text>4. Deployment<jupyter_code># Wurde nicht durchgeführt, da es optional war<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1064, 'fraction_numerical': 0.017, 'mean_word_length': 7.9144385026737964, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '454854', 'license': 'MIT', 'n_tokens_mistral': 1704, 'n_tokens_neox': 1622, 'n_words': 423, 'path': 'Marketing/Generation of Individual Playlists/notebook.ipynb', 'repo_name': 'AlexRossmann/machine-learning-services', 'size': '5037'}