<jupyter_start><jupyter_text>It's ok, we have the same amount of categorical data in train and test both.<jupyter_code>n_unique_train = list(map(lambda col: X_train[col].nunique(), cols_s))
d = dict(zip(cols_s, n_unique_train))
sorted(d.items(), key = lambda x: x[1])
n_unique_test = list(map(lambda col: X_test[col].nunique(), cols_ss))
dd = dict(zip(cols_ss, n_unique_test))
sorted(dd.items(), key = lambda x: x[1])
#First of all let's impute the missing values.

X_train = X_train.fillna(value = {'Alley': 'N','MasVnrType': 'N','BsmtQual': 'TA','BsmtCond': 'TA', 'BsmtExposure': 'No','BsmtFinType1': 'N', 'BsmtFinType2': 'N', 
'Electrical': 'SBrkr','FireplaceQu': 'N', 'GarageType': 'N', 'GarageFinish': 'N','GarageQual': 'N', 'GarageCond': 'N', 'PoolQC': 'N', 'Fence': 'N', 'MiscFeature': 'N', 
'LotFrontage': X_train['LotFrontage'].mean(), 'MasVnrArea': 0, 'GarageYrBlt': X_train['GarageYrBlt'].mean()})

X_test = X_test.fillna(value = {'LotFrontage': X_test['LotFrontage'].mean(), 'MasVnrArea': 0, 'GarageYrBlt': X_test['GarageYrBlt'].mean(), #Test DATA
'BsmtFinSF1' : X_test['LotFrontage'].mean(),'BsmtFinSF2' : X_test['LotFrontage'].mean(),'BsmtUnfSF' : X_test['LotFrontage'].mean(), 'TotalBsmtSF' : X_test['LotFrontage'].mean(),       
'BsmtFullBath' : X_test['LotFrontage'].median(),'BsmtHalfBath' : X_test['LotFrontage'].median(),'GarageCars' : X_test['LotFrontage'].median(),'GarageArea' : X_test['LotFrontage'].mean(),
'MSZoning': 'RL','Alley': 'N','Utilities': 'N','Exterior1st': 'N','Exterior2nd': 'N','MasVnrType': 'N','BsmtQual': 'TA','BsmtCond': 'TA','BsmtExposure': 'No','BsmtFinType1': 'N',
'BsmtFinType2': 'N','KitchenQual': 'TA','Functional': 'Typ','FireplaceQu': 'N','GarageType': 'N','GarageFinish': 'N','GarageQual': 'N','GarageCond': 'N','PoolQC': 'N',
'Fence': 'N','MiscFeature': 'N','SaleType': 'WD',})
X_train.head()
X_test.head()
#Secondly, we need to label features, that: categorical, have cardinality 2 or more than 9. As we found earlier it is:
#in train [('Street', 2), ('Alley', 2), ('Utilities', 2), ('CentralAir', 2), ('Exterior1st', 15), ('Exterior2nd', 16), ('Neighborhood', 25)]
#in test [('Street', 2), ('Alley', 2), ('Utilities', 2), ('CentralAir', 2), ('Exterior1st', 15), ('Exterior2nd', 16), ('Neighborhood', 25)] the same.

#Make copy to avoid changing original data
label_X_train = X_train.copy()
label_X_test = X_test.copy()

#Apply label encoding to each column with a such data:
label_encoder = LabelEncoder()
for col in ['Street','Alley','Utilities','CentralAir','Exterior1st','Exterior2nd','Neighborhood','Utilities','PoolQC','MiscFeature',
            'Electrical','GarageQual','Heating','Condition2','HouseStyle','RoofMatl','Exterior1st','Exterior2nd','MSZoning','KitchenQual','Functional', 'SaleType']:
    label_X_train[col] = label_encoder.fit_transform(X_train[col])
    label_X_test[col] = label_encoder.fit_transform(X_test[col])
    
#As it would found later, we got different cardinaluty in train and test data in the following columns, 
#among those wich we would like to do with OH_encoding:
#[('GarageFinish', 3),('MasVnrType', 4),('Fence', 4),('Electrical', 5),('FireplaceQu', 5),('GarageCond', 5),('BsmtFinType1', 6),
#('BsmtFinType2', 6),('Heating', 6),('GarageType', 6),('Condition2', 8),('HouseStyle', 8),('RoofMatl', 8). 
#So, we need to label them here too.

label_X_train.head()
label_X_test.head()
#Thirdly, we'll use OH_encoding but just with get dummies, now, for all date, because we do what we wanted before, 
#and now there are just categorical features that must be divided by one-hot encoding. 

label_X_train = pd.get_dummies(label_X_train)
label_X_test = pd.get_dummies(label_X_test)
label_X_train.head()
label_X_test.head()<jupyter_output><empty_output><jupyter_text>So, we got the following result:in train data there are 242 columnsin test data there are 228 columnsThe model wouldn't accept it. Well, why we get different amount of columns? There were following columns, that get_dummies:1. ('Alley', 3),2. ('LandSlope', 3),3. ('PavedDrive', 3),4. ('PoolQC', 3),5. ('LotShape', 4),6. ('LandContour', 4),7. ('RoofMatl', 4),8. ('ExterQual', 4),9. ('BsmtQual', 4),10. ('BsmtCond', 4),11. ('BsmtExposure', 4),12. ('Heating', 4),13. ('Electrical', 4),14. ('KitchenQual', 4),15. ('GarageFinish', 4),16. ('MiscFeature', 4),17. ('MSZoning', 5),18. ('LotConfig', 5),19. ('Condition2', 5),20. ('BldgType', 5),21. ('MasVnrType', 5),22. ('ExterCond', 5),23. ('HeatingQC', 5),24. ('GarageQual', 5),25. ('Fence', 5),26. ('RoofStyle', 6),27. ('Foundation', 6),28. ('FireplaceQu', 6),29. ('GarageCond', 6),30. ('SaleCondition', 6),31. ('HouseStyle', 7),32. ('BsmtFinType1', 7),33. ('BsmtFinType2', 7),34. ('Functional', 7),35. ('GarageType', 7),36. ('Condition1', 9),37. ('SaleType', 9).And we have 242 - 228 = 14 different columns, what is it? Well, there are at least 2 problems:1. Some columns has different cardinality in train and test.2. Some of these columns have NONE in test and have no missing values in train and vise versa.What can we do?1. We can find these columns and drop them,2. or just label them too, like others.Ok, done. Now we have in train 173 columns and 170 in test data, still why?That's because there are some NONE that were inpute like 'N' in train data, and in test data there were no empty (in such columns).So, finally we get in both data 179 columns with following label features:['Street','Alley','Utilities','CentralAir','Exterior1st','Exterior2nd','Neighborhood','Utilities','PoolQC','MiscFeature','Electrical','GarageQual','Heating','Condition2','HouseStyle','RoofMatl','Exterior1st','Exterior2nd','MSZoning','KitchenQual','Functional', 'SaleType']<jupyter_code>from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from sklearn.model_selection import cross_val_score

RFR = RandomForestRegressor(n_estimators = 900, random_state=0) 
accuracies_RFR = -1*cross_val_score(estimator= RFR ,X= label_X_train, y=y, cv=5, scoring = 'neg_mean_absolute_error')
print('All scoring_RFR:', accuracies_RFR ,'\n', 'Mean score_RFR:', accuracies_RFR.mean())
XGB = XGBRegressor(n_estimators = 10000, learning_rate = 0.095)
XGB.fit(label_X_train, y, early_stopping_rounds = 55, eval_set = [(label_X_train, y)], verbose = False)
accuracies_XGB = -1*cross_val_score(estimator= XGB ,X = label_X_train, y=y, cv=5, scoring = 'neg_mean_absolute_error')
print('All scoring_XGB:', accuracies_XGB ,'\n', 'Mean score_XGB:', accuracies_XGB.mean())
from xgboost import XGBRegressor
# Testing the model
# Split the data
s_X_train, s_X_test, s_y_train, s_y_test = train_test_split(label_X_train, y, train_size=0.80, test_size=0.20, random_state=0)

# Now, let's test our model!
model = XGBRegressor(n_estimators = 10000, learning_rate = 0.095)
model.fit(s_X_train, s_y_train, early_stopping_rounds = 55, eval_set = [(s_X_test, s_y_test)], verbose = False)

val_predictions = model.predict(s_X_test)
val_mae = mean_absolute_error(val_predictions, s_y_test)
print(val_mae)

# Now, let's submit our model!


#model.fit(label_X_train, y)
test_preds = XGB.predict(label_X_test)

output = pd.DataFrame({'Id': pd.read_csv(iowa_file_test).Id,
                       'SalePrice': test_preds})
output.to_csv('submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18350770925110133, 'fraction_numerical': 0.032351321585903085, 'mean_word_length': 7.126398210290827, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '182559', 'license': 'Apache-2.0', 'n_tokens_mistral': 2920, 'n_tokens_neox': 2662, 'n_words': 655, 'path': 'hpc-my-model.ipynb', 'repo_name': 'Wypyn/Housing-Prices-Competition-for-Kaggle', 'size': '7264'}