<jupyter_start><jupyter_text>Today, in this notebook, we'll try to solve the problem of house prices, first we'll clean the data, then we'll use Machine Learning regression algorithmes to predict the selling price of a house based on 79 characteristics. Let's load the required libraries<jupyter_code>import numpy as np 
import pandas as pd
#import matplotlib.pyplot as plt
#import seaborn as sns

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

import warnings
warnings.filterwarnings("ignore")<jupyter_output>/kaggle/input/sample_submission.csv
/kaggle/input/test.csv
/kaggle/input/data_description.txt
/kaggle/input/train.csv<jupyter_text>Import train and test data sets, display the first 5 lines of the train data set and explore their shapes<jupyter_code>train=pd.read_csv("/kaggle/input/train.csv")
train.head()
#remove outliers
train=train.drop(index=[523,1298],axis=0)
test=pd.read_csv("/kaggle/input/test.csv")
print('th train data has {} rows and {} features'.format(train.shape[0],train.shape[1]))
print('the test data has {} rows and {} features'.format(test.shape[0],test.shape[1]))<jupyter_output>th train data has 1458 rows and 81 features
the test data has 1459 rows and 80 features<jupyter_text>We'll combine these two data sets using the concat attribute<jupyter_code>data=pd.concat([train.iloc[:,:-1],test],axis=0)
print('tha data has {} rows and {} features'.format(data.shape[0],data.shape[1]))<jupyter_output>tha data has 2917 rows and 80 features<jupyter_text>In our data set we have 80 features, check their names and data types using the dataframe attributes: columns and info<jupyter_code>data.columns
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2917 entries, 0 to 1458
Data columns (total 80 columns):
Id               2917 non-null int64
MSSubClass       2917 non-null int64
MSZoning         2913 non-null object
LotFrontage      2431 non-null float64
LotArea          2917 non-null int64
Street           2917 non-null object
Alley            198 non-null object
LotShape         2917 non-null object
LandContour      2917 non-null object
Utilities        2915 non-null object
LotConfig        2917 non-null object
LandSlope        2917 non-null object
Neighborhood     2917 non-null object
Condition1       2917 non-null object
Condition2       2917 non-null object
BldgType         2917 non-null object
HouseStyle       2917 non-null object
OverallQual      2917 non-null int64
OverallCond      2917 non-null int64
YearBuilt        2917 non-null int64
YearRemodAdd     2917 non-null int64
RoofStyle        2917 non-null object
RoofMatl         2917 non-null object
Exterior1st      2916 non-[...]<jupyter_text>It seems we have some features with null values, we'll deal with these values after We'll divide the data into numerical and categorical data and verify their descriptive statistics<jupyter_code>num_features=data.select_dtypes(include=['int64','float64'])
categorical_features=data.select_dtypes(include='object')
num_features.describe()
categorical_features.describe()<jupyter_output><empty_output><jupyter_text>display the columns with the number of missing values<jupyter_code>data.isnull().sum().sort_values(ascending=False)[:34]
#print(categorical_features.isnull().sum().sort_values(ascending=False)[:23])
#num_features.isnull().sum().sort_values(ascending=False)[:11]<jupyter_output><empty_output><jupyter_text>It seems that we have 34 features with NaN values, before processing these values, it is important to know exactly what each feature means by reading the data description file<jupyter_code>f = open("/kaggle/input/data_description.txt", "r")
#print(f.read())<jupyter_output><empty_output><jupyter_text>After reading the description, there is the way we will impute and cleaning the missing values:1. impute the mean value to certain numerical features.2. create a new class 'other' for certain categorical features.3. use the mode value for the other categotical features.4. drop the ID and certain unnecessary features<jupyter_code>data = data.drop(columns=['Id','Street','PoolQC','Utilities'],axis=1)
#data['LotFrontage'].fillna(int(data['LotFrontage'].mean()),inplace=True)
data['LotFrontage'] = data.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))
data['LotFrontage'].isnull().sum()
#create a new class 'other'
features=['Electrical','KitchenQual','SaleType','Exterior2nd','Exterior1st','Alley','Fence', 'MiscFeature','FireplaceQu','GarageCond','GarageQual','GarageFinish','GarageType','BsmtCond','BsmtExposure','BsmtQual','BsmtFinType2','BsmtFinType1','MasVnrType']
for name in features:
    data[name].fillna('Other',inplace=True)
data[features].isnull().sum()
data['MSZoning'] = data.groupby('MSSubClass')['MSZoning'].transform(lambda x: x.fillna(x.mode()[0]))
#data.MSZoning = data.groupby(['MSSubClass'])['MSZoning'].transform(lambda x: x.fillna(x.value_counts()[0]))
data['Functional']=data['Functional'].fillna('typ')
"""mode=['Electrical','KitchenQual','SaleType','Exterior2nd','Exterior1st']
for name in mode:
    data[name].fillna(data[name].mode()[0],inplace=True)"""
zero=['GarageArea','GarageYrBlt','MasVnrArea','BsmtHalfBath','BsmtHalfBath','BsmtFullBath','BsmtFinSF1','BsmtFinSF2','BsmtUnfSF','TotalBsmtSF','GarageCars']
for name in zero:
    data[name].fillna(0,inplace=True)
data.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>That's great, our dataset no longer has any missing values.  Another pre-processing step is to transform categorical features into numerical features<jupyter_code>data.loc[data['MSSubClass']==60, 'MSSubClass']=0
data.loc[(data['MSSubClass']==20)|(data['MSSubClass']==120), 'MSSubClass']=1
data.loc[data['MSSubClass']==75, 'MSSubClass']=2
data.loc[(data['MSSubClass']==40)|(data['MSSubClass']==70)|(data['MSSubClass']==80), 'MSSubClass']=3
data.loc[(data['MSSubClass']==50)|(data['MSSubClass']==85)|(data['MSSubClass']==90)|(data['MSSubClass']==160)|(data['MSSubClass']==190), 'MSSubClass']=4
data.loc[(data['MSSubClass']==30)|(data['MSSubClass']==45)|(data['MSSubClass']==180), 'MSSubClass']=5
data.loc[(data['MSSubClass']==150), 'MSSubClass']=6
object_features = data.select_dtypes(include='object').columns
object_features
def dummies(d):
    dummies_df=pd.DataFrame()
    object_features = d.select_dtypes(include='object').columns
    for name in object_features:
        dummies = pd.get_dummies(d[name], drop_first=False)
        dummies = dummies.add_prefix("{}_".format(name))
        dummies_df=pd.concat([dummies_df,dummies],axis=1)
    return dummies_df
dummies_data=dummies(data)
dummies_data.shape
data=data.drop(columns=object_features,axis=1)
data.columns
final_data=pd.concat([data,dummies_data],axis=1)
final_data.shape<jupyter_output><empty_output><jupyter_text>Let's have fun now with Machine Learning and the Regression algorithms<jupyter_code>#Re-spliting the data into train and test datasets
train_data=final_data.iloc[:1458,:]
test_data=final_data.iloc[1458:,:]
print(train_data.shape)
test_data.shape
# X: independent variables and y: target variable
X=train_data
y=train.loc[:,'SalePrice']
from sklearn.linear_model import Ridge, RidgeCV, LassoCV, ElasticNet
model_las_cv = LassoCV(alphas=(0.0001, 0.0005, 0.001, 0.01, 0.05, 0.1, 0.3, 1, 3, 5, 10))
model_las_cv.fit(X,y)
las_cv_preds=model_las_cv.predict(test_data)
model_ridge_cv = RidgeCV(alphas=(0.01, 0.05, 0.1, 0.3, 1, 3, 5, 10))
model_ridge_cv.fit(X, y)
ridge_cv_preds=model_ridge_cv.predict(test_data)
model_ridge = Ridge(alpha=10, solver='auto')
model_ridge.fit(X, y)
ridge_preds=model_ridge.predict(test_data)
model_en = ElasticNet(random_state=1, alpha=0.00065, max_iter=3000)
model_en.fit(X, y)
en_preds=model_en.predict(test_data)
import xgboost as xgb
model_xgb = xgb.XGBRegressor(learning_rate=0.01,n_estimators=3460,
                                     max_depth=3, min_child_weight=0,
                                     gamma=0, subsample=0.7,
                                     colsample_bytree=0.7,
                                     objective='reg:linear', nthread=-1,
                                     scale_pos_weight=1, seed=27,
                                     reg_alpha=0.00006)
model_xgb.fit(X, y)
xgb_preds=model_xgb.predict(test_data)
from sklearn.ensemble import GradientBoostingRegressor
model_gbr = GradientBoostingRegressor(n_estimators=3000, 
                                learning_rate=0.05, 
                                max_depth=4, 
                                max_features='sqrt', 
                                min_samples_leaf=15, 
                                min_samples_split=10, 
                                loss='huber', 
                                random_state =42)
model_gbr.fit(X, y)
gbr_preds=model_gbr.predict(test_data)
from lightgbm import LGBMRegressor
model_lgbm = LGBMRegressor(objective='regression', 
                                       num_leaves=4,
                                       learning_rate=0.01, 
                                       n_estimators=5000,
                                       max_bin=200, 
                                       bagging_fraction=0.75,
                                       bagging_freq=5, 
                                       bagging_seed=7,
                                       feature_fraction=0.2,
                                       feature_fraction_seed=7,
                                       verbose=-1,
                                       #min_data_in_leaf=2,
                                       #min_sum_hessian_in_leaf=11
                                       )
model_lgbm.fit(X, y)
lgbm_preds=model_lgbm.predict(test_data)
final_predictions = 0.3 * lgbm_preds + 0.3 * gbr_preds + 0.1 * xgb_preds + 0.3 * ridge_cv_preds
#display the first 5 predictions of sale price
final_predictions[:5]
#make the submission data frame
submission = {
    'Id': test.Id.values,
    'SalePrice': final_predictions + 0.007 * final_predictions
}
solution = pd.DataFrame(submission)
solution.head()
#make the submission file
solution.to_csv('submission.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12158590308370044, 'fraction_numerical': 0.03739598629466471, 'mean_word_length': 3.789498359118612, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '379485', 'license': 'MIT', 'n_tokens_mistral': 3597, 'n_tokens_neox': 3278, 'n_words': 674, 'path': 'dataset_0/notebook/house-prices-solution-data-cleaning-ml.ipynb', 'repo_name': 'xuyeliu/JNotebook2csv', 'size': '10215'}