<jupyter_start><jupyter_text>real-estate-3  AbstractPreviously, a dataset that includes property attributes had been filtered and used in a ridge regression model to estimate the sale prices. This model was subsequently improved by use of an SVR, resulting in a 30% increase in R^2, and 20% decrease in error. However, the model accuracy is still fairly low: ~ 61 %. This notebook will explore the use of autoencoders to extract useful features from the data.  Scope and ObjectivesThis is done outside the scope of the challenge, just for learning purposes. The objectives are as follows:1) create an autoencoder for feature extraction2) test for different feature sizes to see if model accuracy improves at all  Libraries<jupyter_code># mathematical
import numpy as np

# data storage / analysis
import pandas as pd

# plotting
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import probplot

# preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# modelling
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.svm import SVR
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model


# evaluation
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error as MAE
# import data
df = pd.read_csv('Data/data_reduced.csv',index_col = 0)
df.head()<jupyter_output><empty_output><jupyter_text>To use autoencoders well, it may be impossible to combine the categorical variables and the continuous ones, simply because of the way optimisation / loss functions work. As a result, it might be a good idea to split these two data sets, and then build two separate autoencoders.<jupyter_code># extract y
X = df[df.columns[df.columns != 'SalePrice_sqrt']]
y = df['SalePrice_sqrt']

# explore column types
categs = []
for col in X.columns:
    if np.unique(X[col]).shape[0] == 2:
        categs.append(True)
    else:
        categs.append(False)
# prepare model
X_categ = X[X.columns[categs]] 
X_cont = X[X.columns[[not categ for categ in categs]]]


X_categ_train, X_categ_test, X_cont_train, X_cont_test = train_test_split(
    X_categ, X_cont, test_size=0.2)
# define autoencoder

def encoder(X_train, X_test):
    inputs = Input(X_train.shape[1])
    
    x = Dense()(inputs)
    x = Dense()(x)
    encoded = Dense()(x)
    
    
def decoder():
model = SVR(kernel = 'linear')
model.fit(X_train, y_train)
yhat = model.predict(X_test)
sns.regplot(y_test, yhat)
plt.show()
r2_score(y_test,yhat)
MAE(y_test, yhat)<jupyter_output><empty_output><jupyter_text>For an initial use of the SVR, the error / r^2 seems to be comparable to that of the ridge regression method used before. However, using different Kernels may provide us with better results.  Optimise model<jupyter_code>kernels = ['linear', 'poly', 'rbf']
# parameters to care for, polynomial order, C, and epsilon
R2 = []
MAE_val = []
models = []
params = []
for p, kernel in enumerate(kernels):
    params.append([0,0,0])
    models.append(0)
    R2.append(0)
    MAE_val.append(1)
    model = SVR(kernel = kernel)
    for i in range(10,30,1000):
        model.set_params(C = i/100)
        for j in range(0,10,100):
            model.set_params(epsilon = j/100)
            if kernel == 'poly':
                for k in range(2,10):
                    model.set_params(degree = k)
                    model.fit(X_train, y_train)
                    yhat = model.predict(X_test)
                    if r2_score(y_test,yhat) > R2[p] and MAE(y_test, yhat) < MAE_val[p]:
                        R2[p] = r2_score(y_test,yhat)
                        MAE_val[p] = MAE(y_test, yhat)
                        models[p] = yhat
                        params[p] = ['C={}'.format(i/100),
                                     'epsilon={}'.format(j/100),
                                     'poly={}'.format(k)]
            else:
                model.fit(X_train, y_train)
                yhat = model.predict(X_test)
                if r2_score(y_test,yhat) > R2[p] and MAE(y_test, yhat) < MAE_val[p]:
                    R2[p] = r2_score(y_test,yhat)
                    MAE_val[p] = MAE(y_test, yhat)
                    models[p] = yhat
                    params[p] = ['C={}'.format(i/100),
                                 'epsilon={}'.format(j/100),
                                 '{}'.format(kernel)]
        
for y_curr, Rsquared, Merror, parameters in zip(models, R2, MAE_val,params):
    sns.regplot(y_test, y_curr)
    print(Rsquared,Merror)
    plt.title('model of type {2}, with {1} and {0}'.format(*parameters))
    plt.show()<jupyter_output>/Users/yousefnami/python_environments/ICDSS_Challenges/real-estate/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Based on this, one might argue that the best model is indeed the 'rbf' model, with a epsilon  = 0 and C = 0.1.This is, at least, an improvement from the previous ridge regression model, both in terms of the MSE and the R^2. It's interesting however that the epsilon value for all 3 was the 0 one. This seems a bit dubious, so further exploration is necessary.In fact, according to online sources, the value of epsilon affects the overfitting / underfitting capability of the curve [2]. With that in mind, epsilon = 0 is most certainly an overfit, and though it performs well on this particular test set, it may not to so more generally.  Test robustness of model<jupyter_code>model = SVR(kernel = 'rbf', epsilon = 0, C = 0.1)
R2s = []
MAEs = []
for i in range(10):
    X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3)
    model.fit(X_train,y_train)
    yhat = model.predict(X_test)
    R2s.append(r2_score(y_test,yhat))
    MAEs.append(MAE(y_test, yhat))

plt.plot([i for i in range(10)],R2s)
plt.show()
plt.plot([i for i in range(10)],MAEs)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09782082324455206, 'fraction_numerical': 0.01678773204196933, 'mean_word_length': 3.1780175320296697, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '294771', 'license': 'MIT', 'n_tokens_mistral': 2059, 'n_tokens_neox': 1963, 'n_words': 635, 'path': 'real-estate/real-estate-3.ipynb', 'repo_name': 'namiyousef/ICDSS_Challenges2020', 'size': '6195'}