<jupyter_start><jupyter_text>MI-ADM: home assignment 1  * **Deadline**: 15/04/2019, -2 points for a late submission, the hard deadline is the first day of the exam period.  * **What to submit**: Just this notebook with you code and texts, not the dataset!  * **How to submit**: See the instructions at https://courses.fit.cvut.cz/MI-ADM/tutorials/index.html.  Generally speaking, the goal of this assignment is to use **decision trees and related methods (boosted trees, random forests) for the regression problem**.What to do:  * Study the data from `house-prices.csv` (see also `house-prices_description.txt`) and prepare them for application of the algorithms (decision tree, random forest, XGBoost).  * Treat the missing values and nominal/ordinal variables reasonably; experiment with various approaches.  * Try to replace the continuous features with some discrete ones (indicator variables, dummy variables, binning, ...).   * Tune some reasonably selected hyperparameters (using cross-validation or just a validation set) and measure the results using *Root mean squared logarithmic error (RMSLE)*.<jupyter_code>import math
import pandas as pd
import numpy as np
import pydotplus
import xgboost
import matplotlib.pyplot as plt
import sklearn.metrics as metrics
from matplotlib.ticker import FormatStrFormatter
from sklearn.metrics import make_scorer, mean_squared_error
from sklearn import tree
from sklearn.preprocessing import KBinsDiscretizer
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.externals.six import StringIO  
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from IPython.display import Image  
from pandas.api.types import CategoricalDtype
from sklearn.model_selection import cross_validate
data = pd.read_csv('house-prices.csv')
# the logarithm of SalesPrice must be used to evaluate RMSLE properly
# you can comment out the following line and apply the log just before evaluating RMSLE
data.SalePrice = data.SalePrice.apply(math.log)<jupyter_output><empty_output><jupyter_text>Prepare the dataset<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1460 entries, 0 to 1459
Data columns (total 81 columns):
Id               1460 non-null int64
MSSubClass       1460 non-null int64
MSZoning         1460 non-null object
LotFrontage      1201 non-null float64
LotArea          1460 non-null int64
Street           1460 non-null object
Alley            91 non-null object
LotShape         1460 non-null object
LandContour      1460 non-null object
Utilities        1460 non-null object
LotConfig        1460 non-null object
LandSlope        1460 non-null object
Neighborhood     1460 non-null object
Condition1       1460 non-null object
Condition2       1460 non-null object
BldgType         1460 non-null object
HouseStyle       1460 non-null object
OverallQual      1460 non-null int64
OverallCond      1460 non-null int64
YearBuilt        1460 non-null int64
YearRemodAdd     1460 non-null int64
RoofStyle        1460 non-null object
RoofMatl         1460 non-null object
Exterior1st      1460 non-n[...]<jupyter_text>Check nan values in continuous features; here I expect a bigger problem with solutions than with categorical attributes.<jupyter_code>col_con = ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']
for i in col_con:
    display(data[i].unique())<jupyter_output><empty_output><jupyter_text>Check nan values in categorical features; There nan can be proper value -> house don't have the poll for example<jupyter_code>col_cat = ['Alley', 'MasVnrType', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 
       'BsmtFinType2', 'Electrical', 'FireplaceQu', 'GarageType', 'GarageFinish', 'GarageQual', 
       'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']
for i in col_cat:
    display(i + "  " + str(data[i].unique()))<jupyter_output><empty_output><jupyter_text>In the categorical attributes there nan is one of the categories in a lot of attributes. For example alley, but on the other hand, there are some attributes to repair. For example, `MasVnrType` which have None values and nan values, this can be the same values probably.`Electrical` is the second attribute with some problem with nan. Nan is not specific in description. There is probably a house without an electrical connection between the homes being sold. I will count on this and leave it as a separate categorical when converting.  Solve nan values<jupyter_code>data.MasVnrType = data.MasVnrType.replace(np.nan, 'None')
# convert to categorical
data[data.select_dtypes(include=['object']).columns] = \
data.select_dtypes(include=['object']).apply(pd.Series.astype, dtype='category')
# can drop the id column 
data = data.drop(columns=['Id'])<jupyter_output><empty_output><jupyter_text>Nan in continuous features will be replaced by zeros so that the decision tree can work with them, for the year of building the garage it is directly offered because not every house has a garage and 0 can represent that the garage does not. Another option for two remaining attributes to use for example knn, but it would be necessary first to process the dataset so that it can be used.<jupyter_code># Fill all NaN with 0
data.loc[:,data.select_dtypes(include=['float64']).columns] = \
data.loc[:,data.select_dtypes(include=['float64']).columns].fillna(0)<jupyter_output><empty_output><jupyter_text>Convert ordinal/categorical attributesI try the dummy coding and the code of categorical variables.<jupyter_code>data_cat_code = data.copy(deep=True)
data_cat_dummy = data.copy(deep=True)
# convert to code (it use -1 for nan values)
str_cols = data_cat_code.select_dtypes('category').columns
for col in str_cols:
    data_cat_code[col] = data_cat_code[col].cat.codes
data_cat_code.info()
# Convert categorical variables to indicators and create new data 
# This generates a lot of attributes because there is a big amount of categorical attributes.
data_cat_dummy = pd.get_dummies(data_cat_dummy)
data_cat_dummy.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1460 entries, 0 to 1459
Columns: 289 entries, MSSubClass to SaleCondition_Partial
dtypes: float64(4), int64(33), uint8(252)
memory usage: 781.4 KB<jupyter_text>Replace continuous features with some discrete ones<jupyter_code>continous_cols = ['MSSubClass', 'LotFrontage', 'LotArea', 'OverallQual', 'OverallCond',
       'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',
       'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',
       'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath',
       'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces',
       'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF',
       'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal',
       'MoSold', 'YrSold']

def bin_it(bins, data, cols, res = None):
    est = KBinsDiscretizer(n_bins=bins, encode='ordinal', strategy='uniform')
    for i in cols:
        est.fit(data[[str(i)]])
        data[str(i)] = est.transform(data[[str(i)]])
        if res == 'dummy':
            data[str(i)] = data[str(i)].astype('category')
    if res == 'dummy':
        data = pd.get_dummies(data)
    return data
# number of bins for variable
n_bins = 5

# categorical variables in code and continuous in dummy coding using discretization
data_cc_dummy = bin_it(5, data_cat_code.copy(deep=True), continous_cols, 'dummy')
# categorical variables in code and continuous in code of bins
data_cc_code = bin_it(10, data_cat_code.copy(deep=True), continous_cols)

# categorical variables in dummy coding and continuous in dummy coding using discretization
data_cd_dummy = bin_it(5, data_cat_dummy.copy(deep=True), continous_cols, 'dummy')

# categorical variables in dummy coding and continuous in code of bins
data_cd_code = bin_it(10, data_cat_dummy.copy(deep=True), continous_cols)

# Use dummy codding on categorical variables and unprocessed continuously attributes (prepare data)
# data_cat_code

# Use code for categorical variables and unprocessed continuously attributes (prepare data)
# data_cat_dummy<jupyter_output><empty_output><jupyter_text>The bins have their numerical designations, which can be used for the representation for decision trees because they are represented by the number. To keep the bin, I will continue to write it as a code and to process this code into 1 of N as a Dummy code.When testing a different number of bins per partition, I noticed that for the dummy coding method, there are about 5 bins optimum and about 9-10 bins for the code, where they have the best results in these numbers in different methods.As will be seen below, with binaries, the methods are not as accurate as leaving a continuous attribute, but I expected that. However, sometimes discretization is necessary because the algorithm might not be able to work with continuous values.  Evaluating on differently prepared datasets using multiple models (decision tree, random forests, etc...).<jupyter_code># number of cores use to evaluate cross-validation
n_jobs = -1
# numbe of fold in cross-validation
cv = 10
# scorer for crossvalidation -> minimalization RMSLE (log() was applied on the begin)
root_mean_squared_error_scorer = make_scorer(mean_squared_error, greater_is_better=True)
def drange(start, stop, step):
    r = start
    while r < stop:
        yield r
        r += step
# method for test model and iterate over one parameter, can be use only for evaluate one set of model.
def test_model_and_meta_parameter(model, data, s_params, test_param, min_value_p = 1, max_value_p = 20, step_p = 1):
    RMSLE_train = []
    RMSLE_test = []
    param_val = []
    for i in drange(min_value_p, max_value_p, step_p):
        s_params[test_param] = i
        mod = model(**s_params)
        scores = cross_validate(mod, data.drop(columns=['SalePrice']), data.SalePrice, 
                            scoring=root_mean_squared_error_scorer, cv=cv, n_jobs=n_jobs, return_train_score=True)
        param_val.append(i)
        RMSLE_test.append(np.sqrt(scores['test_score']).mean())
        RMSLE_train.append(np.sqrt(scores['train_score']).mean())
    return param_val, RMSLE_train, RMSLE_test<jupyter_output><empty_output><jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables and dummy code of categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cat_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cat_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 2.4405433909236118e-05, test error - 0.19842212380819843
Random forest: train error - 0.06473618869344447, test error - 0.1492925217624662
Gradient boosting tree: train error - 0.07988090764216542, test error - 0.12699471805466941
XGBoost: train error - 0.0857013415832413, test error - 0.12900066326215928<jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables and code for categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cat_code, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cat_code, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_code, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 2.836677465267254e-05, test error - 0.19048474149625616
Random forest: train error - 0.06512957791325254, test error - 0.1493318514499022
Gradient boosting tree: train error - 0.08097129957164763, test error - 0.12806845668989625
XGBoost: train error - 0.08682928734291508, test error - 0.1306802543831242<jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables code in dummy and code for categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cc_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cc_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cc_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cc_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 0.0008802525936586498, test error - 0.21473172732131257
Random forest: train error - 0.07054524070211882, test error - 0.16221735569685347
Gradient boosting tree: train error - 0.10106541815728212, test error - 0.14103006160144832
XGBoost: train error - 0.10780850855887465, test error - 0.14114268742286826<jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables code integer code and code for categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cc_code, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cc_code, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cc_code, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cc_code, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 2.3977193507404993e-05, test error - 0.21778785290966454
Random forest: train error - 0.06721706366185004, test error - 0.1535065031996195
Gradient boosting tree: train error - 0.09060255600289664, test error - 0.1304929368450611
XGBoost: train error - 0.09587270613987596, test error - 0.12998119487228021<jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables code in dummy and dummy coded categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cd_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cd_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cd_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cd_dummy, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 0.0008805623617713472, test error - 0.21854463769433616
Random forest: train error - 0.07064443931408367, test error - 0.1628024176620366
Gradient boosting tree: train error - 0.10012085279340402, test error - 0.1409341118831921
XGBoost: train error - 0.10777927524936144, test error - 0.1415107633670019<jupyter_text>Test model on default parameters of methods (Decision tree, random forest, boosted tree, XGBoost) with continuous variables code integer code and dummy coded categorical attributes<jupyter_code>train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cd_code, {}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cd_code, {}, 'random_state', 1, 2, 1)[1:]
print('Random forest: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cd_code, {}, 'random_state', 1, 2, 1)[1:]
print('Gradient boosting tree: train error - ', train[0], ', test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cd_code, {}, 'random_state', 1, 2, 1)[1:]
print('XGBoost: train error - ', train[0], ', test error - ', test[0], sep='')<jupyter_output>Decision tree: train error - 2.3598280569181105e-05, test error - 0.21100925697096962
Random forest: train error - 0.06752927378103725, test error - 0.15269604843933304
Gradient boosting tree: train error - 0.0903676314776091, test error - 0.12963431421366503
XGBoost: train error - 0.09480186824483995, test error - 0.1304964353792063<jupyter_text>It can be seen that the decision tree without the depth limitation certainly suffers to over-learning and it is certainly necessary to limit the depth. This will be the first model and parameter I will try to improve. Other models certainly have the same problem, but not so much, and the more advanced the model, the lesser the learning curve. XGBoost can already be pretty well set up so that it only suffered minimally.With the basic parameters Gradient boosting achieved the best score. In the next step, I will try to debug single hyperparameters of the models and improve the results.For each model, I will try to achieve the best score within the prepared datasets. Especially in the XGBoost model, there are many parameters that can be tuned in the model.  Tune the hyper parameters of the models<jupyter_code># Function help to plot it
def plot_it(param_val, RMSLE_train, RMSLE_test, test_param):
    fig, ax = plt.subplots()
    fig.set_size_inches(4.5, 2.5)
    ax.plot(param_val, RMSLE_train, 'rs', param_val, RMSLE_test, 'bs')
    ax.legend(['Train', 'Test'])
    ax.yaxis.set_major_formatter(FormatStrFormatter('%.4f'))
    plt.xlabel(test_param)
    plt.ylabel('RMSLE')
    plt.show()<jupyter_output><empty_output><jupyter_text>Decision tree<jupyter_code>plot_it(*test_model_and_meta_parameter(DecisionTreeRegressor, data_cat_code, 
                                       {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1}, 
                                        'max_depth', 1, 25, 1), 'max_depth')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cat_code, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: code for categorical and no for continuous. \n \
              train error - ', train[0], ',       test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cat_dummy, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: categorical in dummy and no for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cc_dummy, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':3}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: code for categorical and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cc_code, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: code for categorical and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cd_dummy, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':3}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: categorical in dummy and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(DecisionTreeRegressor, data_cd_code, {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':2}, 'random_state', 1, 2, 1)[1:]
print('Decision tree: categorical in dummy and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')<jupyter_output>Decision tree: code for categorical and no for continuous. 
               train error - 0.1334527368671097,       test error - 0.1818178334158593
Decision tree: categorical in dummy and no for continuous. 
               train error - 0.13362755561229717,      test error - 0.18054719995189417
Decision tree: code for categorical and continuous in dummy code. 
               train error - 0.1565533336335046,      test error - 0.19781270039228918
Decision tree: code for categorical and code for continuous. 
               train error - 0.14354979928768064,      test error - 0.19478783656162105
Decision tree: categorical in dummy and continuous in dummy code. 
               train error - 0.15566210989421078,      test error - 0.20376837134233644
Decision tree: categorical in dummy and code for continuous. 
               train error - 0.14462254231510213,      test error - 0.19326850488067795<jupyter_text>On the graph, the success of the decision tree depends on the depth of the decision tree.I finally chose the depth 6 according to the cross.I measured the model on all datasets.The best results are if I kept the continuous attributes normal and did not combine them.The depth was finally the main parameter, the other parameters were ideal to leave in the original settings.  Random forrests<jupyter_code>plot_it(*test_model_and_meta_parameter(RandomForestRegressor, data_cat_code, 
                              {'max_depth': 6, 'random_state':0, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'max_depth', 1, 12, 1), 'max_depth')
plot_it(*test_model_and_meta_parameter(RandomForestRegressor, data_cc_dummy, 
                                       {'max_depth': 6, 'random_state':0, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'max_depth', 1, 12, 1), 'max_depth')
plot_it(*test_model_and_meta_parameter(RandomForestRegressor, data_cat_code, 
                                       {'max_depth': 7, 'random_state':0, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'n_estimators', 1, 150, 1), 'n_estimators')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cat_code,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: code for categorical and no for continuous. \n \
              train error - ', train[0], ',       test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cat_dummy,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: categorical in dummy and no for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cc_dummy,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: code for categorical and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cc_code,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: code for categorical and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cd_dummy,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: categorical in dummy and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(RandomForestRegressor, data_cd_code,  {'max_depth': 6, 'random_state':1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':180}, 'random_state', 1, 2, 1)[1:]
print('Random forest: categorical in dummy and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')<jupyter_output>Random forest: code for categorical and no for continuous. 
               train error - 0.10614124166271555,       test error - 0.15130229911145818
Random forest: categorical in dummy and no for continuous. 
               train error - 0.1066302260090732,      test error - 0.15220958853809408
Random forest: code for categorical and continuous in dummy code. 
               train error - 0.12815598381496338,      test error - 0.16554671430277942
Random forest: code for categorical and code for continuous. 
               train error - 0.11837621747927507,      test error - 0.15767939417297042
Random forest: categorical in dummy and continuous in dummy code. 
               train error - 0.1277268953556578,      test error - 0.16536453915027854
Random forest: categorical in dummy and code for continuous. 
               train error - 0.11867407669217933,      test error - 0.1579760537685988<jupyter_text>The random forest parameters behaved as expected. I chose the depth again 6. This method has achieved significantly better results than the tree itself.Here, the datasets, where I left continuous attributes unbound, also came close to a dataset, where I used dummy coding for categorical attributes and continually binned into ten bins and used the generated representation for each bin.  Gradient boosting<jupyter_code>plot_it(*test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_code, 
                                      {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'max_depth', 1, 17, 1), 'max_depth')
plot_it(*test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_code, 
                                       {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'learning_rate', 0.01, 0.2, 0.01), 'learning_rate')
plot_it(*test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_code, 
                                       {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':100}, 
                                        'n_estimators', 1, 75, 1), 'n_estimators')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_code, {'max_depth': 4, 'random_state':0, 'learning_rate':0.1,
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: code for categorical and no for continuous. \n \
              train error - ', train[0], ',       test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cat_dummy,  {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: categorical in dummy and no for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cc_dummy,  {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: code for categorical and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cc_code,  {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: code for categorical and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cd_dummy,  {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: categorical in dummy and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(GradientBoostingRegressor, data_cd_code,  {'max_depth': 4, 'random_state':0, 'learning_rate':0.1, 
                                        'min_samples_split':2, 'min_samples_leaf':1, 'n_estimators':75}, 
                                        'random_state', 1, 2, 1)[1:]
print('Gradient boosting: categorical in dummy and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')<jupyter_output>Gradient boosting: code for categorical and no for continuous. 
               train error - 0.06666079608909234,       test error - 0.12654736625315172
Gradient boosting: categorical in dummy and no for continuous. 
               train error - 0.06632966509081449,      test error - 0.1266040425763933
Gradient boosting: code for categorical and continuous in dummy code. 
               train error - 0.08571531469844,      test error - 0.13986662739376282
Gradient boosting: code for categorical and code for continuous. 
               train error - 0.07774163226752738,      test error - 0.13183123782404613
Gradient boosting: categorical in dummy and continuous in dummy code. 
               train error - 0.08623904116367734,      test error - 0.13840793631859688
Gradient boosting: categorical in dummy and code for continuous. 
               train error - 0.07802866653586708,      test error - 0.13237564440982302<jupyter_text>The boosted tree also had the parameter learning rate. With this method, I chose a smaller depth of trees for over-fitting, which was evident in this method at great depth. Also, the learning rate remained lower to avoid such over-fitting.Again, this method achieved better results than the previous one, and it was best to leave continuous parameters untouched. The difference was even more pronounced than previous methods.  XGBoost<jupyter_code>plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                      {'colsample_bytree': 0.5, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':3, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'max_depth', 1, 20, 1), 'max_depth')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'colsample_bytree', 0.1, 1, 0.1), 'colsample_bytree')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'gamma', 0, 1, 0.1), 'gamma')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'learning_rate', 0.1, 1, 0.1), 'learning_rate')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'min_child_weight', 0.1, 5, 0.2), 'min_child_weight')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'n_estimators', 50, 500, 5), 'n_estimators')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'reg_alpha', 0, 5, 0.1), 'reg_alpha')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'reg_lambda', 0, 5, 0.1), 'reg_lambda')
plot_it(*test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, 
                                       {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 
                                        'subsample', 0.1, 1, 0.01), 'subsample')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_code, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':10000,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: code for categorical and no for continuous. \n \
              train error - ', train[0], ',       test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cat_dummy, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: categorical in dummy and no for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cc_dummy, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: code for categorical and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cc_code, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: code for categorical and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cd_dummy, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: categorical in dummy and continuous in dummy code. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')
train, test = test_model_and_meta_parameter(xgboost.XGBRegressor, data_cd_code, {'colsample_bytree': 0.4, 'gamma':0, 'learning_rate':0.1, 
                                        'max_depth':4, 'min_child_weight':1.5, 'n_estimators':100,
                                        'reg_alpha':0.5, 'reg_lambda':0.45, 'subsample':0.8, 
                                        'seed':42}, 'seed', 42, 43, 1)[1:]
print('XGBoost: categorical in dummy and code for continuous. \n \
              train error - ', train[0], ',      test error - ', test[0], sep='')<jupyter_output>XGBoost: code for categorical and no for continuous. 
               train error - 0.017298075388074676,       test error - 0.12354150601561764
XGBoost: categorical in dummy and no for continuous. 
               train error - 0.08608964233041166,      test error - 0.12557509768761527
XGBoost: code for categorical and continuous in dummy code. 
               train error - 0.10456057348519136,      test error - 0.1381382386315992
XGBoost: code for categorical and code for continuous. 
               train error - 0.09357458097851681,      test error - 0.12855730965738763
XGBoost: categorical in dummy and continuous in dummy code. 
               train error - 0.10565172166973813,      test error - 0.13997466621358123
XGBoost: categorical in dummy and code for continuous. 
               train error - 0.09469943341302785,      test error - 0.12687528975661094
{'fraction_non_alphanumeric': 0.11178700107622479, 'fraction_numerical': 0.06457348743624539, 'mean_word_length': 3.0141810668670175, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 107, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '399242', 'license': 'MIT', 'n_tokens_mistral': 14617, 'n_tokens_neox': 12614, 'n_words': 3561, 'path': 'HW1/MI-ADM_home_assignement_1.ipynb', 'repo_name': 'martilad/MI-ADM2019', 'size': '42742'}