<jupyter_start><jupyter_text><jupyter_code># importing libraries for data analysis and manipulation
import pandas as pd
import numpy as np
# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# To ignore potential useless warnings from Scipy (SciPy issue #5998)
import warnings
warnings.filterwarnings(action="ignore", message="^internal gelsd")

# to make this notebook's output identical at every run
np.random.seed(42)
# Code to read csv file into colaboratory:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)<jupyter_output><empty_output><jupyter_text>> Inserting the dataset for regression and classification (the split between train and test set is already done).<jupyter_code>downloaded = drive.CreateFile({'id':'1faxDrc-DqcciP0PghJbJJts9CRa0bL82'}) 
downloaded.GetContentFile('CS98XClassificationTest.csv') 
classification_test = pd.read_csv('CS98XClassificationTest.csv') 

downloaded = drive.CreateFile({'id':'1TQq8nRRdowS_aULA1SKviqAisw9NWIUd'}) 
downloaded.GetContentFile('CS98XClassificationTrain.csv') 
classification_train = pd.read_csv('CS98XClassificationTrain.csv') 

downloaded = drive.CreateFile({'id':'1tmBD1ct5ig0UF7iIPmcM0MqDwJpYvtpo'}) 
downloaded.GetContentFile('CS98XRegressionTest.csv') 
regression_test = pd.read_csv('CS98XRegressionTest.csv') 

downloaded = drive.CreateFile({'id':'1Zgg12bV-xXozSRI5Kj4631gwbBe2K8By'}) 
downloaded.GetContentFile('CS98XRegressionTrain.csv') 
regression_train = pd.read_csv('CS98XRegressionTrain.csv')<jupyter_output><empty_output><jupyter_text>Regression Task The regression task is about building a model which predicts the popularity of a song, hence: > Model should learn from the data and be able to predict the popularity of a song given other metrics. It is a supervised learning task since we are given labelled training examples. It is also a multiple regression problem, since we are going to use multiple features which will help the model make predictions about the popularity of a song. Furthermore, it s a univariate regression problem (not multivariate) because we are only trying to predict a single feature (i.e. popularity) for each song. Lastly, this is batch learning (not online learning) since we do not have flow of data coming from external source.  Exploration of the dataset > Getting the shape and the 5 first rows:<jupyter_code>regression_train.shape
regression_test.shape # we can understand that the split of test and test set was 80/20 (453+114=567 rows totally)
regression_train.head()
regression_test.head()<jupyter_output><empty_output><jupyter_text>> Information about the features of the dataset: >> We notice there are 15 columns:- 3 of them (title, artist, top genre) with categorical values- the rest 12 columns contain numemical values- 'pop', i.e. song's popularity, is the target value (label)- the other 14 attributes can be used as predictors<jupyter_code>regression_train.info()
# summary statistics for numerical features

regression_train.describe()<jupyter_output><empty_output><jupyter_text>> Exploration of the categorical attributes (title, artist, top genre):<jupyter_code>regression_train['title'].nunique()

regression_train['artist'].nunique() # totally 345 artists
regression_train['top genre'].nunique() # totally 86 top genres
# top 10 most popular songs
top_10_popular=regression_train.sort_values(by=["pop"], ascending=False).head(10)
top_10_popular<jupyter_output><empty_output><jupyter_text>Visualisation of the dataset<jupyter_code># For visualising distributional values
import seaborn as sns
import matplotlib.pyplot as plt

regression_train["top genre"].value_counts().nlargest(5).sort_values(ascending=True).plot.barh()
plt.ylabel("Top genre")
plt.xlabel("Count")
plt.title("Top Genres (top 5)")
plt.show()
# Plot histograms 
%matplotlib inline
import matplotlib.pyplot as plt

regression_train.hist(bins=50, figsize=(12,7))
plt.show()
# Plot boxplot of particular feautures
regression_train_boxplot= regression_train[['bpm', 'nrgy', 'dnce', 'dB', 'live', 'val', 'spch', 'pop']]
sns.boxplot(data=regression_train_boxplot)
plt.xlabel('Features')
plt.ylabel('Value')
plt.show()
# violin plot between popularity and loudness regarding the top 10 popular songs
plt.figure(figsize=(15,10))
sns.violinplot(x= "pop", y="dB", data= top_10_popular)
plt.show()<jupyter_output><empty_output><jupyter_text>>Heatmap of the dataset: >>The darker the colour(closest to 1), the stronger is the correlation. Here we notice that popularity has strong correlation with energy(nrgy), danceability(dnce), loudness(dB) (and duration, but probably not important):<jupyter_code># Plot linear correlation matrix
fig, ax = plt.subplots(figsize=(15,10))
sns.heatmap(regression_train.corr(), annot=True, cmap='YlGnBu', vmin=-1, vmax=1, center=0, ax=ax)
plt.title('Linear Correlation Matrix')
plt.show()<jupyter_output><empty_output><jupyter_text>> Fitting linear lines between the numerical attributes which seemed to have strong correlation in heatmap<jupyter_code>'''
Fit line between popoluarity and energy
'''
fig, ax = plt.subplots() 
fit = np.polyfit(regression_train['nrgy'], regression_train['pop'], deg=1)
print(fit)
ax.plot(regression_train['nrgy'], fit[0] * regression_train['nrgy'] + fit[1], color='green')
ax.scatter(regression_train['nrgy'],regression_train['pop'])
plt.xlabel("Energy")
plt.ylabel("Popularity") 
plt.title('Fitting the line between popularity and energy')
plt.show()

'''
Fit line between popoluarity and loudness
'''
fig, ax = plt.subplots() 
fit = np.polyfit(regression_train['dB'], regression_train['pop'], deg=1)
print(fit)
ax.plot(regression_train['dB'], fit[0] * regression_train['dB'] + fit[1], color='green')
ax.scatter(regression_train['dB'],regression_train['pop'])
plt.xlabel("Loudness")
plt.ylabel("Popularity") 
plt.title('Fitting the line between popularity and loudness')
plt.show()


'''
Fit line between popoluarity and danceability
'''
fig, ax = plt.subplots() 
fit = np.polyfit(regression_train['dnce'], regression_train['pop'], deg=1)
print(fit)
ax.plot(regression_train['dnce'], fit[0] * regression_train['dnce'] + fit[1], color='green')
ax.scatter(regression_train['dnce'],regression_train['pop'])
plt.xlabel("Danceability")
plt.ylabel("Popularity") 
plt.title('Fitting the line between popularity and danceability')
plt.show()

'''
Fit line between popoluarity and duration
'''
fig, ax = plt.subplots() 
fit = np.polyfit(regression_train['dur'], regression_train['pop'], deg=1)
print(fit)
ax.plot(regression_train['dur'], fit[0] * regression_train['dur'] + fit[1], color='green')
ax.scatter(regression_train['dur'],regression_train['pop'])
plt.xlabel("Duration")
plt.ylabel("Popularity") 
plt.title('Fitting the line between popularity and duration')
plt.show()<jupyter_output>[ 0.18428789 49.67363777]<jupyter_text>> Exploring the attribute "energy" more in depth:<jupyter_code>import seaborn as sns
color= sns.color_palette()
sns.set_style('darkgrid')
sns.distplot(regression_train.nrgy)
plt.show()<jupyter_output><empty_output><jupyter_text>Further Correlations > Below we will use the standard correlation coefficient (Pearson's r) between every pair of attributes compared to 'popularity', with the corr() method: > Standard correlation coefficient calculates strength of linear relationship:    * close to +1 it indicates a strong positive correlation    * close to -1 means a strong negative one<jupyter_code>corr_matrix= regression_train.corr()
corr_matrix
corr_matrix['pop'].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>> So from above, we can notice that popularity has strong positive linear relationship with: duration, loudness (dB), energy and danceability (and in lower degree with speechiness)<jupyter_code># histogram of loudness

regression_train['dB'].hist()
# Histogram of energy

regression_train['nrgy'].hist()
#scatter plot between energy and loudness which seem to be important indicators for song's popularity

regression_train.plot(kind='scatter', x='nrgy', y='dB', alpha=0.4)
plt.show
# alpha for transparency
regression_train.plot(kind='scatter', x= 'dB', y='pop', alpha=0.6)
plt.show()<jupyter_output><empty_output><jupyter_text>Preparing the data for machine learning algorithms > Calling the isolated dataframe as: 'spotify'<jupyter_code>spotify= regression_train.drop('pop', axis=1) # isolating the target which is 'pop'
spotify_labels= regression_train['pop'].copy()               # target variable/label is 'pop'<jupyter_output><empty_output><jupyter_text>Data Cleaning  Feature Engineering: feature selection<jupyter_code>spotify = spotify[['top genre', 'nrgy', 'dnce', 'dB', 'dur']] # the final train predictos we are keeping

spotify.shape<jupyter_output><empty_output><jupyter_text>> Most ML algorithms cannot work with missing features so this is an important step:<jupyter_code>spotify.isnull().any()
spotify.isnull().sum()<jupyter_output><empty_output><jupyter_text>> We can view the 15 missing values in the column 'top genre'<jupyter_code>missing_values_regr= spotify[spotify.isnull().any(axis=1)]
missing_values_regr<jupyter_output><empty_output><jupyter_text>> Replacing those with 'adult standards', as in exploration of the dataset it was the most famous top genre and, thus, it seems to be one good choice:<jupyter_code>spotify['top genre'].fillna(value='adult standards',inplace=True)
spotify.isnull().sum()<jupyter_output><empty_output><jupyter_text>> Below we are going to use 'SimpleImputer' which can handle the missing values. Based on the way we going to fill its parameters, we are going to specify how we want to handle the missing values in the unseed data (test data): Although in this dataset we can know that we have missing values only in top genre in test data (as we will see below), we are continuing this way for the generalisation of the model. If the test data had missing numerical values, then they would be filled by the median value:<jupyter_code>from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
spotify.info()
spotify_num= spotify.drop(['top genre'], axis=1) #the numerical attributes
spotify_num
spotify.info()
imputer.fit(spotify_num)
imputer.statistics_
spotify_num.median().values
X = imputer.transform(spotify_num)
spotify_tr = pd.DataFrame(X, columns=spotify_num.columns,
                          index=spotify.index)
imputer.strategy
spotify_tr = pd.DataFrame(X, columns=spotify_num.columns,
                          index=spotify_num.index)
spotify_tr.head(2)
# a sample of how inputer works
spotify.info()
# categorical attributes which we will hold for the construction of the model
categorical_attribute= spotify[['top genre']]<jupyter_output><empty_output><jupyter_text>> Process for using OneHoTencoder for converting categorical into numerical values:<jupyter_code>from sklearn.preprocessing import OrdinalEncoder

ordinal_encoder = OrdinalEncoder()
spotify_cat_encoded = ordinal_encoder.fit_transform(categorical_attribute)
spotify_cat_encoded[:5]
from sklearn.preprocessing import OneHotEncoder

cat_encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
spotify_cat_hot1 = cat_encoder.fit_transform(categorical_attribute)
spotify_cat_hot1

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder<jupyter_output><empty_output><jupyter_text>> Machine Learning Algorithms do not perform well when the input numerical attributes have different scales. Below we are going to scale the numerical attributes, and construct a pipeline which can link the tranformations both of numerical and categorical values in the test set:<jupyter_code># We create the preprocessing pipelines for both numerical and categorical data
# credits to: https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html
# we are using the 'most frequent' strategy with the rational that out categorical attributes are
# very possible to be repeated as the song becomes more popular (one popular singer might own many popular songs in the test set)
numeric_features = list(spotify_num)
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())])

categorical_features = list(categorical_attribute)
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

full_pipeline = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)])

spotify_final= full_pipeline.fit_transform(spotify) #spofify_final : final and transformed train predictors-features<jupyter_output><empty_output><jupyter_text>> OneHotEncoder returns a sparse matrix while num_pipeline a dense matrix. In this case of mixing the numerical with categorical transformations, ColumnTransformer returns either a sparse or a dense matrix. Here it returned a sparse matrix:<jupyter_code>spotify_final
# we can convert to an array if needed: spotify_final= spotify_final.toarray()<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code>from sklearn.linear_model import LinearRegression
lin_reg= LinearRegression()
lin_reg.fit(spotify_final, spotify_labels)<jupyter_output><empty_output><jupyter_text>Root Mean Square Error (RMSE)> Measures the standard deviation of the errors the systems makes in its predictions:<jupyter_code>from sklearn.metrics import mean_squared_error
spotify_predictions= lin_reg.predict(spotify_final)<jupyter_output><empty_output><jupyter_text>>> The interval of popularity scores (from the data exploration) was between 26 and 84. Here RMSE is 9.27. We could say it not a bad result, but neither a good one:<jupyter_code>lin_mse = mean_squared_error(spotify_labels, spotify_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse<jupyter_output><empty_output><jupyter_text>> Using r2_score to conclude if the dataset is linear. If close to 1, the dataset is linear by nature. Here it is not negative, but it has not a high value neither:<jupyter_code>from sklearn.metrics import r2_score
r2_score(spotify_predictions, spotify_labels)<jupyter_output><empty_output><jupyter_text>MAE > Calculating the Mean Absolute Error (MAE), which is smaler than that of the RMSE, since RMSE raises the errors to its square values<jupyter_code>from sklearn.metrics import mean_absolute_error

lin_mae = mean_absolute_error(spotify_labels, spotify_predictions)
lin_mae<jupyter_output><empty_output><jupyter_text>Training a Decision Tree Regressor<jupyter_code>from sklearn.tree import DecisionTreeRegressor

tree_reg= DecisionTreeRegressor(random_state=42)
tree_reg.fit(spotify_final, spotify_labels)
spotify_predictions = tree_reg.predict(spotify_final)
tree_mse = mean_squared_error(spotify_labels, spotify_predictions)
tree_rmse = np.sqrt(tree_mse)
tree_rmse<jupyter_output><empty_output><jupyter_text>> Obviously this cannot be a valid result and this model is extremely overfitting the data >> Evaluation of Decision tree using Cross-Validation with K-folds = 10:<jupyter_code>from sklearn.model_selection import cross_val_score

scores = cross_val_score(tree_reg, spotify_final, spotify_labels,
                         scoring="neg_mean_squared_error", cv=10)

tree_rmse_scores = np.sqrt(-scores)
# the scoring function is the opposite of the MSE so that's why we use '-scores'
tree_rmse_scores
def display_tree_scores(tree_scores):
    print("Scores:", tree_scores)
    print("Mean:", tree_scores.mean())
    print("Standard deviation:", tree_scores.std())

display_tree_scores(tree_rmse_scores)<jupyter_output>Scores: [15.94760443 13.24189662 16.09347694 16.98365227 15.43084933 15.99791653
 13.40480842 14.57089946 16.77696834 11.39395551]
Mean: 14.984202785724648
Standard deviation: 1.7084601132034862<jupyter_text>> Here we can see than DecisionTreeRegressor is performing much worse than linear regression  > Below it is an effort to change decision tree's parameters. Decision Trees are prone to overfitting when dealing with regression tasks, for this reason we set min_samples_leaf=10, and max_depth=1 which gave better results than the default.>> It performs better than the previous DecisionTreeRegressor, however the final result is still not satisfactory.<jupyter_code>from sklearn.tree import DecisionTreeRegressor
tree_reg= DecisionTreeRegressor( random_state=42, min_samples_leaf=10, max_depth=1)
tree_reg.fit(spotify_final, spotify_labels)


from sklearn.model_selection import cross_val_score

scores = cross_val_score(tree_reg, spotify_final, spotify_labels,
                         scoring="neg_mean_squared_error", cv=10)

tree_rmse_scores = np.sqrt(-scores)
# the scoring function is the opposite of the MSE so that's why we use '-scores'
tree_rmse_scores
def display_tree_scores(tree_scores):
    print("Scores:", tree_scores)
    print("Mean:", tree_scores.mean())
    print("Standard deviation:", tree_scores.std())

display_tree_scores(tree_rmse_scores)<jupyter_output>Scores: [13.03497615 10.79426206 10.19277763 11.5861052  10.40096828 11.92517844
 12.06441073 11.89422954 13.55243587 10.74488677]
Mean: 11.619023067905678
Standard deviation: 1.0523359813729696<jupyter_text>Evaluation in the Test Set<jupyter_code>regression_test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 114 entries, 0 to 113
Data columns (total 14 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   Id         114 non-null    int64 
 1   title      114 non-null    object
 2   artist     114 non-null    object
 3   top genre  113 non-null    object
 4   year       114 non-null    int64 
 5   bpm        114 non-null    int64 
 6   nrgy       114 non-null    int64 
 7   dnce       114 non-null    int64 
 8   dB         114 non-null    int64 
 9   live       114 non-null    int64 
 10  val        114 non-null    int64 
 11  dur        114 non-null    int64 
 12  acous      114 non-null    int64 
 13  spch       114 non-null    int64 
dtypes: int64(11), object(3)
memory usage: 12.6+ KB<jupyter_text>> Sychronizing the test's shape with that of the train set:<jupyter_code>x= regression_test[['top genre','nrgy', 'dnce', 'dB', 'dur']] # test predictors
x.info()
x.isnull().any()<jupyter_output><empty_output><jupyter_text>> For example this command: "x['top genre'].fillna(value='adult standards',inplace=True)" is not needed to be executed because of the aforementioned pipeline transformations. If executed then the warning is: "A value is trying to be set on a copy of a slice from a DataFrame". > Running the full pipeline to tranform the test data.>> Before we used 'fit_transform', now we are calling only 'transform' for the successfull transition:<jupyter_code>x_final=full_pipeline.transform(x)<jupyter_output><empty_output><jupyter_text>>The number of rows is normal to be different and it depends on the initial split which was 80%-20% between train and test set. The number of columns are transformed because of the previous steps, but the important here is that both train and test predictors to have the same number of columns (the given inputs compared to the unseen data of test set):<jupyter_code>x_final.shape
spotify_final.shape
pop_predictions= lin_reg.predict(x_final)
pop_predictions<jupyter_output><empty_output><jupyter_text>>Exporting the results of 'pop' and 'Id' into a csv file:<jupyter_code>len(list(regression_test['Id']))
len(list(pop_predictions))
regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions})
#regr_results
#regr_results.to_csv('GroupN.csv', index = False)
len(regr_results)<jupyter_output><empty_output><jupyter_text>Random Forest Regressor<jupyter_code>from sklearn.ensemble import RandomForestRegressor
forest_reg= RandomForestRegressor()
forest_reg.fit(spotify_final, spotify_labels)
pop_predictions = forest_reg.predict(spotify_final)
forest_mse = mean_squared_error(spotify_labels, pop_predictions)
forest_rmse = np.sqrt(forest_mse)
forest_rmse<jupyter_output><empty_output><jupyter_text>> The ForestRegressor is overfitting the test data, since the RMSE in test set was approximately double reaching 8.45 (the model is performing better in training that in test data)<jupyter_code>from sklearn.model_selection import cross_val_score

scores = cross_val_score(forest_reg, spotify_final, spotify_labels,
                         scoring="neg_mean_squared_error", cv=10)

forest_reg_scores = np.sqrt(-scores)
# the scoring function is the opposite of the MSE so that's why we use '-scores'
forest_reg_scores
def display_forest_reg_scores(forest_reg_scores):
    print("Scores:", forest_reg_scores)
    print("Mean:", forest_reg_scores.mean())
    print("Standard deviation:", forest_reg_scores.std())

display_forest_reg_scores(forest_reg_scores)


model_rfr= RandomForestRegressor(n_estimators=500, max_leaf_nodes=16, n_jobs=-1)
model_rfr.fit(spotify_final,spotify_labels)
pop_predictions_rfr= model_rfr.predict(x_final)
#pop_predictions_rfr
regr_results_rfr= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions_rfr})
#regr_results_rfr 

#regr_results_rf= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions_rfr})
#regr_results_rf 
#regr_results_rf.to_csv('rf.csv', index = False) #### RMSE 8.45 in test set<jupyter_output><empty_output><jupyter_text>> Combining Random Forest Regressor with grid search for detecting the best estimator:<jupyter_code>from sklearn.model_selection import GridSearchCV

param_grid = [
    # try 12 (3×4) combinations of hyperparameters
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    # then try 6 (2×3) combinations with bootstrap set as False
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]

forest_reg = RandomForestRegressor(random_state=42)
# train across 5 folds, that's a total of (12+6)*5=90 rounds of training 
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error',
                           return_train_score=True)
grid_search.fit(spotify_final, spotify_labels)
#best hyperparameter
grid_search.best_params_

grid_search.best_estimator_
# score of each hyperparameter combination tested during the grid search
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_distribs = {
        'n_estimators': randint(low=1, high=200),
        'max_features': randint(low=1, high=8),
    }

forest_reg = RandomForestRegressor(random_state=42)
rnd_search = RandomizedSearchCV(forest_reg, param_distributions=param_distribs,
                                n_iter=10, cv=5, scoring='neg_mean_squared_error', random_state=42)
rnd_search.fit(spotify_final, spotify_labels)
cvres = rnd_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
feature_importances = grid_search.best_estimator_.feature_importances_
feature_importances
grid_model = grid_search.best_estimator_

grid_model = grid_search.best_estimator_
x_final=full_pipeline.transform(x)
pop_predictions_grid= grid_model.predict(x_final)

#pop_predictions_grid
regr_results_grid= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions_grid})
#regr_results_grid 
#****************************************************************************************************
#****************************************************************************************************
## 8.11
#regr_results_grid.to_csv('gr.csv', index = False)<jupyter_output><empty_output><jupyter_text>Stochastic Gradient Descent (SGD) Regressor> Gradient Descent is a different way of training a linear regression model, which is better suited for cases where there is a large number of features or too many training instances to fit in memory. Here, none of these hold true, however, we are going to try it:> The main idea here is to tweak SGD's parameters iteratively in order to minimise the cost function.>> Performing SGD linear regression using SGDRegressor(): maximum epochs= 1000, or until the loss drops less than 0.001 (eta=1e-3), default learning scedule, does not use regularisation (penalty=none) and learning rate "η"(eta)=0.1<jupyter_code>from sklearn.linear_model import SGDRegressor
sgd_regressor= SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0=0.1, random_state=42)
sgd_regressor.fit(spotify_final, spotify_labels)

sgd_pop_predictions= sgd_regressor.predict(x_final)<jupyter_output><empty_output><jupyter_text>> As we can notice (rmse in test data=16.88) SGD is not appropriate here<jupyter_code>#regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': sgd_pop_predictions})
#regr_results 
#regr_results.to_csv('-&-.csv', index = False)
#rmse = 16.88<jupyter_output><empty_output><jupyter_text>Polynomial Features<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
poly_features= PolynomialFeatures(degree=2, include_bias=False)
X_poly= poly_features.fit_transform(spotify_final)
lin_reg=LinearRegression()
lin_reg.fit(X_poly, spotify_labels)
Y_poly= poly_features.fit_transform(x_final)
poly_pop_predictions= lin_reg.predict(Y_poly)<jupyter_output><empty_output><jupyter_text>> As exprected, using high-degree polynomial would not prevent the training data from overfitting the test data<jupyter_code>#regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': poly_pop_predictions})
#regr_results 
#regr_results.to_csv('dwe.csv', index = False)
# rmse = 16.23<jupyter_output><empty_output><jupyter_text>SVM Linear Regression> SVM in general can perform linear and non-linear regression, classification and outlier detection. >> C: hyperparameter of the modelUsing Support Vector Regressor with kerner='linear':<jupyter_code>from sklearn.svm import SVR
svm_reg = SVR(kernel="linear")
svm_reg.fit(spotify_final, spotify_labels)
pop_predictions = svm_reg.predict(spotify_final)
svm_mse = mean_squared_error(spotify_labels, pop_predictions)
svm_rmse = np.sqrt(svm_mse)
svm_rmse<jupyter_output><empty_output><jupyter_text>Setting epsilon=1.5.Epsilon specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value<jupyter_code>from sklearn.svm import LinearSVR

svm_reg= LinearSVR(epsilon=1.5, random_state=42)

svm_reg.fit(spotify_final, spotify_labels)
svm_pop_predictions= svm_reg.predict(x_final)
regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': svm_pop_predictions})
#regr_results 
#regr_results.to_csv('-&-.csv', index = False)
#rmse = 8.84!! but still is < 8.11 of grid search
from sklearn.model_selection import cross_val_score

scores = cross_val_score(svm_reg, spotify_final, spotify_labels,
                         scoring="neg_mean_squared_error", cv=10)

svm_reg_scores = np.sqrt(-scores)
# the scoring function is the opposite of the MSE so that's why we use '-scores'
svm_reg_scores
def display_svm_reg_scores(svm_reg_scores):
    print("Scores:", svm_reg_scores)
    print("Mean:", svm_reg_scores.mean())
    print("Standard deviation:", svm_reg_scores.std())

display_svm_reg_scores(svm_reg_scores)
svm_reg= LinearSVR(epsilon=1.5, random_state=42, C=12)

svm_reg.fit(spotify_final, spotify_labels)
svm_pop_predictions= svm_reg.predict(x_final)<jupyter_output><empty_output><jupyter_text>> Cross Evaluation using cross_val_svore<jupyter_code>from sklearn.model_selection import cross_val_score

scores = cross_val_score(svm_reg, spotify_final, spotify_labels,
                         scoring="neg_mean_squared_error", cv=10)

svm_reg_scores = np.sqrt(-scores)
# the scoring function is the opposite of the MSE so that's why we use '-scores'
svm_reg_scores
def display_svm_reg_scores(svm_reg_scores):
    print("Scores:", svm_reg_scores)
    print("Mean:", svm_reg_scores.mean())
    print("Standard deviation:", svm_reg_scores.std())

display_svm_reg_scores(svm_reg_scores)


regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': svm_pop_predictions})
#regr_results 
#regr_results.to_csv('-&-.csv', index = False)

#***************************************************************************************************************
#rmse = 7.86967 So far the 2nd best model 
#***************************************************************************************************************<jupyter_output><empty_output><jupyter_text>LinearSVR with an RBF kernel> Using randomised search with cross validation to find the appropriate hyperparameter values for C and gamma:<jupyter_code>from sklearn.svm import LinearSVR

lin_svr = LinearSVR(random_state=42)
lin_svr.fit(spotify_final, spotify_labels)
pop_pred=lin_svr.predict(x_final)
#len(pop_pred)
from sklearn.svm import SVR
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import reciprocal, uniform

param_distributions = {"gamma": reciprocal(0.001, 0.1), "C": uniform(1, 15)}
rnd_search_cv = RandomizedSearchCV(SVR(), param_distributions, n_iter=10, verbose=2, cv=3, random_state=42)
rnd_search_cv.fit(spotify_final, spotify_labels)
rnd_search_cv.best_estimator_
pop_pred = rnd_search_cv.best_estimator_.predict(x_final)
regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_pred})
#regr_results 
#regr_results.to_csv('cc.csv', index = False)
#***************************************************************************************************************

# RMSE in test: 7.83639. So far the best model 
#***************************************************************************************************************
spotify_final_dense= spotify_final.toarray()
x_final_dense= x_final.toarray()<jupyter_output><empty_output><jupyter_text>Gradient Boosting Regressor> Boosting hypothesis refers to any Ensemble method that can combine several weak learners into a strong learner: It works by sequentially adding predictors to an ensemble, each one correcting its predecessor (popular methods: Adaptive Boost, Gradient Boosting). They are both suitable for regression and classification.> Gradient Boosting Regressor works by adding predictos to an ensemble, each one correcting its predecessor, by fitting the new predictor to the residual errors made by the previous predictor (and not by increasing the relative weight of misclassified training instances as in Adaboost).>> n_estimators= numbers of trees >> max_depth, min_samples_leaf: hyperparameters controlling the growth of Desicion Trees<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor

gbr= GradientBoostingRegressor(max_depth=2, n_estimators=100, learning_rate=0.1, random_state=42)
gbr.fit(spotify_final_dense, spotify_labels)
pop_predictions_gbr= gbr.predict(x_final_dense)
regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions_gbr})
#regr_results 
#regr_results.to_csv('gb.csv', index = False)
#rmse= 8.9<jupyter_output><empty_output><jupyter_text>XGBoost> Extreme Gradient Boosting (Optimised Implementation of Gradient Boosting)<jupyter_code>import xgboost<jupyter_output><empty_output><jupyter_text>> It requires dense arrays:<jupyter_code>xgb_reg = xgboost.XGBRegressor(random_state=42)
xgb_reg.fit(spotify_final_dense, spotify_labels)
pop_predictions_gb= xgb_reg.predict(x_final_dense)
regr_results= pd.DataFrame({'Id': regression_test['Id'], 'pop': pop_predictions_gb})
#regr_results 
#regr_results.to_csv('cc.csv', index = False)
# RMSE= 8.1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11495254395518904, 'fraction_numerical': 0.02237435817644313, 'mean_word_length': 6.878401569012013, 'pattern_counts': {'":': 2, '<': 187, '<?xml version=': 0, '>': 249, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '47', 'dir': 'jupyter-structured-clean-dedup', 'id': '502755', 'license': 'MIT', 'n_tokens_mistral': 10925, 'n_tokens_neox': 10017, 'n_words': 2727, 'path': 'Spotify_Songs_Popularity_Regression/Spotify_Songs_Popularity_Regression.ipynb', 'repo_name': 'dimi-fn/Spotify-Songs', 'size': '32135'}