<jupyter_start><jupyter_text>Kaggle House Prices Using ML and DNNTrying out different ML algorithms and constructing a DNN to solve the problem.resource & explanation:    https://www.dataquest.io/blog/kaggle-getting-started/<jupyter_code>import pandas as pd
import numpy as np
# importing the data from csv files
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
# checking the data size
print('Train data shape:',train.shape)
print('Test data shape:',test.shape)
# first few rows from the data
train.head()
train.SalePrice.describe()
# checking the skew of the saleprice (target) data
print('Skew of SalePrice is: ',train.SalePrice.skew())
# plotting the histogram of SalePrice
import matplotlib.pyplot as plt
plt.hist(train.SalePrice)
plt.show()<jupyter_output><empty_output><jupyter_text>Given the skew in the data, the log function can be used to imrove the linearity of the data.Q: why should it be more linear?<jupyter_code># using np.log to improve linearity
lin_target = np.log(train.SalePrice)
print('Skew of SalePrice is: ',lin_target.skew())
# plotting the histogram of the target
plt.hist(lin_target)
plt.show()<jupyter_output><empty_output><jupyter_text>Improved the distribution of the dataQ: How does this compare to normalisation?  Non numerical features can be a hastle to work with, hence only numerical features will be used for nowThis can be done by extracting the numerical features using .select_dtypes()<jupyter_code># selecting numerical data and displaying the results
numeric_features = train.select_dtypes(include=[np.number])
numeric_features.dtypes
# evaluating the correlation between the data
corr = numeric_features.corr()
# most correlated data
print (corr['SalePrice'].sort_values(ascending=False)[:10], '\n')
# least correlated data
print (corr['SalePrice'].sort_values(ascending=False)[-10:])
# given that OverallQual is mostly correlated, lets look into it,
# using a pivot table
quality_pivot = train.pivot_table(index='OverallQual',values='SalePrice',aggfunc=np.median)
quality_pivot<jupyter_output><empty_output><jupyter_text>The values above are the relation between the value of OverallQual and SalePrice, OverallQual takes on values from 1 to 10.This can be shown as following<jupyter_code>train.OverallQual.unique()
# plotting the pivot table
quality_pivot.plot(kind='bar',color='blue')
plt.xlabel('Overall Quality')
plt.ylabel('Median Sale Price')
plt.xticks(rotation=0)# rotate the x labels, purely aesthetic
plt.show()<jupyter_output><empty_output><jupyter_text>Handling Null values is the data<jupyter_code># checking the null values in train data
nulls = pd.DataFrame(numeric_features.isnull().sum().sort_values(ascending=False)[:25])
nulls.columns = ['Null Count']
nulls.index.name = 'Feature'
nulls<jupyter_output><empty_output><jupyter_text>LotFrontage has 259 Null values,GarageYrBlt has 81 Null values,MasVnrArea has 8 Null values.we can full these naively with the mean<jupyter_code>#train.LotFrontage.unique()
train.LotFrontage = train.LotFrontage.fillna(int(np.mean(train.LotFrontage)))
train.GarageYrBlt = train.GarageYrBlt.fillna(int(np.mean(train.GarageYrBlt)))
train.MasVnrArea = train.MasVnrArea.fillna(int(np.mean(train.MasVnrArea)))
print('LotFrontage Nulls: ',train.LotFrontage.isnull().sum())
print('GarageYrBlt Nulls: ',train.GarageYrBlt.isnull().sum())
print('MasVnArea Nulls: ',train.MasVnrArea.isnull().sum())<jupyter_output>LotFrontage Nulls:  0
GarageYrBlt Nulls:  0
MasVnArea Nulls:  0<jupyter_text>What has been done for the training data needs to be done for the test data.- obtain only numerical features- fill in the null values with mean<jupyter_code>numeric_features_test = test.select_dtypes(include=[np.number])

# checking the null values in test data
nulls = pd.DataFrame(numeric_features_test.isnull().sum().sort_values(ascending=False)[:25])
nulls.columns = ['Null Count']
nulls.index.name = 'Feature'
nulls
def myfill(data):
    data = data.fillna(int(np.mean(data)))
    return data
test.LotFrontage = myfill(test.LotFrontage)
test.GarageYrBlt = myfill(test.GarageYrBlt)
test.MasVnrArea = myfill(test.MasVnrArea)
test.BsmtHalfBath = myfill(test.BsmtHalfBath)
test.BsmtFullBath = myfill(test.BsmtFullBath)
test.GarageArea = myfill(test.GarageArea)
test.BsmtFinSF1 = myfill(test.BsmtFinSF1)
test.BsmtFinSF2 = myfill(test.BsmtFinSF2)
test.BsmtUnfSF = myfill(test.BsmtUnfSF)
test.TotalBsmtSF = myfill(test.TotalBsmtSF)
test.GarageCars = myfill(test.GarageCars)
# DataSets end results
Dataset = train.select_dtypes(include=[np.number]).drop(['SalePrice','Id'],axis = 1)
Target = np.log(train.SalePrice)<jupyter_output><empty_output><jupyter_text>Next Phase: LearnersSo far the data has been sorted and processed, although very naivly.i.e, no proper feature engineering has been carried out and non-numeric values have been totaly ignored.Now we can test some ML algorithms, for this we will use SKlearn,Hyperparameter tuning will mostly be ignored.<jupyter_code># splitting the previous train data into train & test sets
from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test = train_test_split(Dataset,Target,random_state= 42,test_size=0.33)
print('Train dataset: ',X_train.shape)
print('Train Target: ',Y_train.shape)
print('Test dataset: ',X_test.shape)
print('Test Target: ',Y_test.shape)<jupyter_output>Train dataset:  (978, 36)
Train Target:  (978,)
Test dataset:  (482, 36)
Test Target:  (482,)<jupyter_text>1. Linear Regression<jupyter_code>from sklearn import linear_model as LM
# creating the model
linR = LM.LinearRegression()
# training the model
model = linR.fit(X_train,Y_train)
# evaluating the model
acc = model.score(X_test,Y_test)
print('Accuracy of LinReg model is: ',acc)
# using RMSE to evaluate the model
from sklearn.metrics import mean_squared_error as mse
predictions = model.predict(X_test)
rmse = mse(Y_test,predictions)
print('RMSE for LinReg model is: ',rmse)<jupyter_output>RMSE for LinReg model is:  0.0233627288955<jupyter_text>Let us visualise the regression using a plot<jupyter_code>plt.scatter(predictions,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('Linear Regression Model')
plt.show()<jupyter_output><empty_output><jupyter_text>2. Ridge Regression<jupyter_code># creating the model
RidgeR = LM.Ridge(alpha = 0.5)
# training the model
ridge_model = RidgeR.fit(X_train,Y_train)
# evaluating the model
acc_ridge = ridge_model.score(X_test,Y_test)
print('Accuracy of RidgeReg model is: ',acc_ridge)

##
ridge_pred = ridge_model.predict(X_test)
ridge_rmse = mse(Y_test,ridge_pred)
print('RMSE for RidgeReg model is: ',ridge_rmse)

##
plt.scatter(ridge_pred,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('Ridge Regression Model')
plt.show()<jupyter_output>Accuracy of RidgeReg model is:  0.86294238267
RMSE for RidgeReg model is:  0.0233589193154<jupyter_text>3. Lasso<jupyter_code># creating the model
Lasso = LM.Lasso(0.001)
#training the model
lasso_model = Lasso.fit(X_train,Y_train)
# evaluating the model
lasso_acc = lasso_model.score(X_test,Y_test)
print('Accuracy of Lasso model is: ',lasso_acc)

##
lasso_pred = lasso_model.predict(X_test)
lasso_rmse = mse(Y_test,lasso_pred)
print('RMSE for Lasso model is: ',lasso_rmse)

##
plt.scatter(lasso_pred,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('Lasso Model')
plt.show()<jupyter_output>Accuracy of Lasso model is:  0.862802720813
RMSE for Lasso model is:  0.0233827220788<jupyter_text>4. Random Forests<jupyter_code>from sklearn.ensemble import RandomForestRegressor as RFR
max_depth = 60

# creating the model
RF = RFR(max_depth = max_depth,random_state=12)
#training the model
rf_model = RF.fit(X_train,Y_train)
# evaluating the model
rf_acc = rf_model.score(X_test,Y_test)
print('Accuracy of RF model is: ',rf_acc)

##
rf_pred = rf_model.predict(X_test)
rf_rmse = mse(Y_test,lasso_pred)
print('RMSE for RF model is: ',rf_rmse)

##
plt.scatter(rf_pred,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('RF Model')
plt.show()<jupyter_output>Accuracy of RF model is:  0.868523344082
RMSE for RF model is:  0.0233827220788<jupyter_text>5. Baysian Ridge Regression<jupyter_code># creating the model
Baye = LM.BayesianRidge()
#training the model
Baye_model = Baye.fit(X_train,Y_train)
# evaluating the model
baye_acc = Baye_model.score(X_test,Y_test)
print('Accuracy of BayeReg model is: ',baye_acc)

##
Baye_pred = lasso_model.predict(X_test)
baye_rmse = mse(Y_test,Baye_pred)
print('RMSE for BayeReg model is: ',baye_rmse)

##
plt.scatter(Baye_pred,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('BayeReg Model')
plt.show()<jupyter_output>Accuracy of BayeReg model is:  0.863599468162
RMSE for BayeReg model is:  0.0233827220788<jupyter_text>6. Neural Netwrok<jupyter_code># Needs a fix for input dtype
'''from sklearn.neural_network import MLPClassifier as NN
# defining the NN strucutre

# creating the model
nn = NN(hidden_layer_sizes = (36,36,36))
# training the model
nn_clf = nn.fit(X_train,Y_train)
# evaluating the model
nn_acc = nn_clf.score(X_test,Y_test)
print('Accuracy of NN model is: ',nn_acc)

##
nn_pred = nn_clf.predict(X_test)
nn_rmse = mse(Y_test,nn_pred)
print('RMSE for NN model is: ',nn_rmse)

##
plt.scatter(nn_pred,Y_test,alpha=0.6)
plt.xlabel('Predictions')
plt.ylabel('Actual value')
plt.title('NN Model')
plt.show()'''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11317763922646096, 'fraction_numerical': 0.021346296100602347, 'mean_word_length': 7.013547840812871, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '606485', 'license': 'MIT', 'n_tokens_mistral': 3647, 'n_tokens_neox': 3401, 'n_words': 819, 'path': 'Kaggle_HousePrices.ipynb', 'repo_name': 'OakLake/Kaggle_House-Prices', 'size': '9463'}