<jupyter_start><jupyter_text>In this notebook, we load the preprocessed dataset as created in 'Project_preprocessing' and build a Support Vector Regressor model. The preprocessed dataset is split into train and test sets. 1. Naive Baseline - Predictions on the test set are performed using 'Naive Baseline' model (mean strategy). The baseline Root mean square error (RMSE) is thus established.2. Default model - The default Support Vector Regressor model is evaluated with 5 fold cross validation on train set. The model is then fit on the entire train data and predictions are performed on test data. RMSE using default model on test data is reported.3. Tuned model - Lastly, we optimize the parameters of the model using 5 cross validation. An extensive parameter search is performed using several grid searches. 'neg_root_mean_squared_error' is used as scoring function during grid search. The 'best estimator' model obtained after grid searches with 5 fold cross validation is finally used to perform predictions on the test set. RMSE obtained using tuned model on test set is compared with that of Default model and Naive Baseline model.4. Finally, we reduce dimensions using PCA, repeat steps 2 & 3 above and compare RMSE scores.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
#read preprocessed dataset as pandas dataframe
dataset = pd.read_csv('melbourne-airbnb.csv')
#examine first 5 rows
dataset.head()
#split into a features dataframe and target series
X = dataset.drop('price',axis=1)
y = dataset['price']
#split into train and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)<jupyter_output><empty_output><jupyter_text>Naive Baseline<jupyter_code>#create naive baseline
from sklearn.metrics import mean_squared_error, r2_score
mean_price = np.mean(y)
y_test_naive = np.repeat(mean_price, len(y_test))
naive_rmse = mean_squared_error(y_test, y_test_naive)
naive_rmse = np.sqrt(naive_rmse)
naive_r2 = r2_score(y_test,y_test_naive)
print('Naive Baseline RMSE on test set: ' + str(round(naive_rmse,2)))
print('Naive Baseline R2 on test set: ' + str(round(naive_r2,2)))<jupyter_output>Naive Baseline RMSE on test set: 75.75
Naive Baseline R2 on test set: -0.0<jupyter_text>Support Vector Machine - Default parameters Support Vector machines are based on a distance metric, therefore they require the data to be scaled for optimal performance. We will perform 'RobustScaling' as we have some outliers in the dataset. Also, we will perform a transformation of scaled data to reduce skewness. The steps of scaling and transformation will be included in the machine learning pipeline.<jupyter_code>#evaluate svr with default parameters on test set
from sklearn.svm import SVR
from sklearn.preprocessing import RobustScaler,PowerTransformer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

steps = [('scaler', RobustScaler()),('transformer', PowerTransformer(method='yeo-johnson')), ('SVR', SVR())]
pipeline = Pipeline(steps)

# evaluate default model with 5 fold cross validation on train set
scores = cross_val_score(pipeline,X_train,y_train, cv=5, scoring='neg_root_mean_squared_error')
mean_score = abs(np.mean(scores))

#fit model on entire train set and predict on test set
pipeline.fit(X_train,y_train)
y_pred_def = pipeline.predict(X_test)

# evaluate predictions made on test set
svr_def_r2 = r2_score(y_test,y_pred_def)
svr_def_rmse = np.sqrt(mean_squared_error(y_test, y_pred_def))

print('SVR (default parameters) mean cross validated RMSE on train set: ' + str(round(mean_score,2)))
print()
print('SVR (default parameters) R2 score on test set: ' + str(round(svr_def_r2,1)))
print()
print('SVR (default parameters) RMSE score on test set: ' + str(round(svr_def_rmse,2)))<jupyter_output>SVR (default parameters) mean cross validated RMSE on train set: 60.63

SVR (default parameters) R2 score on test set: 0.4

SVR (default parameters) RMSE score on test set: 58.09<jupyter_text>Support Vector Machine - hyperparameter optimizationPerform hyperparameter tuning using grid search with 5 fold cross validation on train set. Evaluate tuned model on test set and calculate RMSE.<jupyter_code>from sklearn.model_selection import GridSearchCV

steps = [('scaler', RobustScaler()),('transformer', PowerTransformer(method='yeo-johnson')), ('SVR', SVR())]
pipeline = Pipeline(steps)

#define hyperparameters to tune
parameters = {'SVR__kernel': ['linear'], 'SVR__C':[1,10,100], 'SVR__gamma':[1.0,0.01,0.001]}

#perform grid search using cross validation
grid1_lin = GridSearchCV(pipeline, param_grid=parameters, cv=5, scoring='neg_root_mean_squared_error')
grid1_lin.fit(X_train, y_train)
print('Best parameters with grid1')
print(grid1_lin.best_params_)
grid1_rmse = abs(grid1_lin.best_score_)
best_std = grid1_lin.cv_results_['std_test_score'][grid1_lin.best_index_]
print('Mean cross validated RMSE with best estimator grid1: '+ str(round(grid1_rmse,2)) + ', Standard deviation: ' + str(round(best_std,2)))
#further tuning of parameters
steps = steps = [('scaler', RobustScaler()),('transformer', PowerTransformer(method='yeo-johnson')), ('SVR', SVR())]
pipeline = Pipeline(steps)

#define hyperparameters to tune
parameters = {'SVR__kernel': ['rbf'],'SVR__C':[1,10,100], 'SVR__gamma':[1.0,0.01,0.001]}

#perform grid search using cross validation
grid2_rbf = GridSearchCV(pipeline, param_grid=parameters, cv=5, scoring='neg_root_mean_squared_error')
grid2_rbf.fit(X_train, y_train)
print('Best parameters with grid2')
print(grid2_rbf.best_params_)
grid2_rmse = abs(grid2_rbf.best_score_)
best_std = grid2_rbf.cv_results_['std_test_score'][grid2_rbf.best_index_]
print('Mean cross validated RMSE with best estimator grid2: '+ str(round(grid2_rmse,2)) + ', Standard deviation: ' + str(round(best_std,2)))
#predict using model with tuned hyperparameters
y_pred_rbf = grid2_rbf.predict(X_test)
rbf_r2 = r2_score(y_test,y_pred_rbf)
rbf_rmse = mean_squared_error(y_test, y_pred_rbf)
rbf_rmse = np.sqrt(rbf_rmse)
print('R2 score using tuned SVR(rbf) on test set: ' + str(round(rbf_r2,1)))
print()
print('RMSE using tuned SVR(rbf) on test set: ' + str(round(rbf_rmse,2)))
#plot predicted vs actual values and residuals plot

residuals = y_test-y_pred_rbf

sns.set(style="whitegrid")
fig,(ax1,ax2) = plt.subplots(1,2, figsize= (14,6))
sns.regplot(x = y_pred_rbf, y=y_test,scatter_kws={'s':5}, ax=ax1)
ax1.set_xlabel('Predicted values',fontsize=12)
ax1.set_ylabel('Actual values',fontsize=12)
ax1.set_title('Predicted vs Actual values', fontsize = 16)
sns.residplot(x=y_pred_rbf, y=residuals,scatter_kws={'s':5}, ax=ax2)
ax2.set_xlabel('Predicted values',fontsize=12)
ax2.set_ylabel('Residuals',fontsize=12)
ax2.set_title('Residuals plot', fontsize=16);<jupyter_output><empty_output><jupyter_text>Support Vector Machine - Reduced Dimensions In this section, we will reduce dimensions using Principal Component Analysis and then evaluate the performance of SVR models (default and after tuning of hyperparameters). We will first choose the number of principal components based on explained variance and then include PCA as a step in the machine learning pipeline.<jupyter_code>#perform PCA on dataset to determine optimal number of components
from sklearn.decomposition import PCA
scaler = RobustScaler() #since our dataset has a few outliers, this is a better scaling model than StandardScaler()

X_scaled = scaler.fit_transform(X)

pca = PCA()
pca.fit_transform(X_scaled)
#function to draw scree plot (explained variance by principal components)
def var_explained():
    import numpy as np
    from matplotlib.pyplot import figure, show
    from matplotlib.ticker import MaxNLocator

    ax = figure().gca()
    ax.plot(np.cumsum(pca.explained_variance_ratio_))
    ax.xaxis.set_major_locator(MaxNLocator(integer=True))
    plt.xlabel('Number of Components')
    plt.ylabel('Cumulative Explained Variance')
    plt.axvline(x=9, linewidth=1, color='r', alpha=0.5)
    plt.title('Explained Variance of PCA by Component')
    show()

var_explained()<jupyter_output><empty_output><jupyter_text>We can account for ~95% of variance if we select 9 principal components.  SVR (default parameters) after PCAAn important preprocessing step before reducing dimensions with PCA is to scale the data. Since our dataset has some outliers, we will use RobustScaler() method. Also, we will perform a transformation of scaled data to reduce skewness and then reduce dimensions using PCA. The steps of scaling,transformation and PCA will be included in the machine learning pipeline.<jupyter_code>#reduce dimensions using PCA and then evaulate SVR model
steps = [('scaler', RobustScaler()),('pca',PCA(n_components=9)),('SVR', SVR())]
pipeline = Pipeline(steps)

# evaluate default model with 5 fold cross validation on train set
scores = cross_val_score(pipeline,X_train,y_train, cv=5, scoring='neg_root_mean_squared_error')
mean_score = abs(np.mean(scores))

#fit model on entire train set and predict on test set
pipeline.fit(X_train,y_train)
y_pred_def = pipeline.predict(X_test)

# evaluate predictions made on test set
svr_def_r2_pca = r2_score(y_test,y_pred_def)
svr_def_rmse_pca = np.sqrt(mean_squared_error(y_test, y_pred_def))

print('SVR(default parameters) mean cross validated RMSE on train set after PCA: ' + str(round(mean_score,2)))
print()
print('SVR (default parameters) R2 score on test set after PCA: ' + str(round(svr_def_r2_pca,1)))
print()
print('SVR (default parameters) RMSE score on test set after PCA: ' + str(round(svr_def_rmse_pca,2)))<jupyter_output>SVR(default parameters) mean cross validated RMSE on train set after PCA: 61.45

SVR (default parameters) R2 score on test set after PCA: 0.4

SVR (default parameters) RMSE score on test set after PCA: 58.7<jupyter_text>SVR (hyperparameters tuned) after PCA<jupyter_code>#reduce dimensions using PCA and then perform grid search to tune hyperparameters 
steps = [('scaler', RobustScaler()),('pca',PCA(n_components=9)),('SVR', SVR(kernel='rbf'))]
pipeline = Pipeline(steps)
parameters = {'SVR__C':[1,10,100], 'SVR__gamma':[1.0,0.01,0.001]}
grid_rbf_pca = GridSearchCV(pipeline, param_grid=parameters, cv=5,scoring='neg_root_mean_squared_error')
grid_rbf_pca.fit(X_train, y_train)
print('Best parameters after PCA: ')
print(grid_rbf_pca.best_params_)
grid_pca_rmse = abs(grid_rbf_pca.best_score_)
best_std = grid_rbf_pca.cv_results_['std_test_score'][grid_rbf_pca.best_index_]
print('Mean cross validated RMSE with best estimator after PCA: '+ str(round(grid_pca_rmse,2)) + ', Standard deviation: ' + str(round(best_std,2)))
y_pred_pca = grid_rbf_pca.predict(X_test)
pca_r2 = r2_score(y_test,y_pred_pca)
pca_rmse = mean_squared_error(y_test, y_pred_pca)
pca_rmse = np.sqrt(pca_rmse)
print('R2 score using SVR(rbf) after PCA: ' + str(round(pca_r2,1)))
print()
print('RMSE using SVR(rbf) after PCA: ' + str(round(pca_rmse,2)))<jupyter_output>R2 score using SVR(rbf) after PCA: 0.5

RMSE using SVR(rbf) after PCA: 54.9<jupyter_text>Reducing dimensions with PCA resulted in a slightly lower performance of the default and tuned models on test set, as compared to the respective models' performances without PCA.<jupyter_code>#final summary
print('Summarizing: ')
print()
print('Naive Baseline RMSE on test set: ' + str(round(naive_rmse,2)))
print()
print('SVR (default parameters) RMSE on test set: ' + str(round(svr_def_rmse,2)))
print()
print('SVR(tuned parameters) RMSE on test set: ' + str(round(rbf_rmse,2)))
print()
print('SVR (default parameters) RMSE on test set after PCA: ' + str(round(svr_def_rmse_pca,2)))
print()
print('SVR(tuned parameters) RMSE on test set after PCA: ' + str(round(pca_rmse,2)))<jupyter_output>Summarizing: 

Naive Baseline RMSE on test set: 75.75

SVR (default parameters) RMSE on test set: 58.09

SVR(tuned parameters) RMSE on test set: 53.62

SVR (default parameters) RMSE on test set after PCA: 58.7

SVR(tuned parameters) RMSE on test set after PCA: 54.9
{'fraction_non_alphanumeric': 0.10055587820459637, 'fraction_numerical': 0.018584584750684478, 'mean_word_length': 6.707161125319693, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '364604', 'license': 'Apache-2.0', 'n_tokens_mistral': 4187, 'n_tokens_neox': 3968, 'n_words': 1231, 'path': 'Support_Vector_Regressor.ipynb', 'repo_name': 'vivekpandian08/Airbnb_Prediction_Melbourne', 'size': '12053'}