<jupyter_start><jupyter_text>House Price Prediction of King County  Abstract In this assignment, I have applied the machine learning to a real world problem of prediction of house prices of King County. The aim of this assignment was to apply various machine learning algorithms and find out whether this dataset give maximum accuracy if we are treating this dataset as a classification problem or regression a regression problem. In order to do classification, I have added one variable which acts as price quantitative variable. I have applied Random Forest, XGBoost and Logistic regression for classification and Linear Regression, Random Forest regression, XGBoost Regression and Gradient Boosting regression in order to do regression. The XGBoost classification was the best model for this dataset and I was able to achieve 90.73% accurate predictions.  ApproachAlgorithms used.* For Part A:    * Bagging Algorithm : Random Forest Classifier    * Boosting Algorithm : XGBoost Classifier* For Part B:    * Logistic Regression* For Part C:    * Bagging Algorithm : Random Forest Regressor    * Boosting Algorithm : XGBoost Regressor* For Part D:    * Linear Regression * For Part E:    * Gradient Boosting Regression  Importing Libraries<jupyter_code>import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from pprint import pprint
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import GridSearchCV
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.model_selection import StratifiedKFold<jupyter_output><empty_output><jupyter_text>Dataset King County House Price prediction https://www.kaggle.com/harlfoxem/housesalesprediction/home This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015.<jupyter_code>data = pd.read_csv("king_county_cleansed.csv")
data.head(5)<jupyter_output><empty_output><jupyter_text>1. index :- It is the unique numeric number assigned to each house being sold. 2. price:- It is the price of house which we have to predict so this is our target variable and aprat from it are our features. 3. bedrooms :- It determines number of bedrooms in a house. 4. bathrooms :- It determines number of bathrooms in a bedroom of a house. 5. sqft_living :- It is the measurement variable which determines the measurement of house in square foot. 6. sqft_lot : It is also the measurement variable which determines square foot of the lot. 7. floors: It determines total floors means levels of house. 8. waterfront : This feature determines whether a house has a view to waterfront 0 means no 1 means yes. 9. view : This feature determines whether a house has been viewed or not 0 means no 1 means yes. 10. condition : It determines the overall condition of a house on a scale of 1 to 5. 11. grade : It determines the overall grade given to the housing unit, based on King County grading system on a scale of 1 to 12. 13. sqft_above : It determines square footage of house apart from basement. 14. sqft_basement : It determines square footage of the basement of the house. 15. yr_built : It detrmines the date of building of the house. 16. yr_renovated : It detrmines year of renovation of house. 17. zipcode : It determines the zipcode of the location of the house. 18. lat : It determines the latitude of the location of the house. 19. long : It determines the longitude of the location of the house. 20. sqft_living15 : Living room area in 2015(implies-- some renovations) 21. sqft_lot15 : lotSize area in 2015(implies-- some renovations)<jupyter_code>print(data.isnull().any()) #Finding out if there is any null value in our data
data.describe()<jupyter_output><empty_output><jupyter_text>Correlation Analysis<jupyter_code>corr=data.corr()
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

f, ax = plt.subplots(figsize=(20, 20))

cmap = sns.diverging_palette(190, 10, as_cmap=True)
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.8, cbar_kws={"shrink": .5})<jupyter_output><empty_output><jupyter_text>Part A - Classification with Trees We will use Random Forest Classifier (Bagging based tree algorithm) and XGBoost classifier (Boosting tree algorithm) In order to do classification, we need to add catagorical variable which represents house price as HIGH or LOW. We will achieve that by adding one coloumn "Price_Catagory". '0' will represent price below the median value and '1' will represent price above median value.<jupyter_code>median_price = data.loc[:,"price"].median()
print(median_price)
data['Price_Catagory'] = data['price'].apply(lambda x: 1 if x > median_price else 0)
data[['price','Price_Catagory']].head(10)
data.head(5)<jupyter_output><empty_output><jupyter_text>1. Random Forest Classifier  Initially we will find out which features are important.<jupyter_code>data.columns
X_features=['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']

X=data[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040' ]]

y=data['Price_Catagory']


# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3) # 70% training and 30% test
clf=RandomForestClassifier(n_estimators=100)

#Train the model using the training set
clf.fit(X_train,y_train)
y_pred=clf.predict(X_test)
feature_imp = pd.Series(clf.feature_importances_,index=X_features).sort_values(ascending=False)
top_features = feature_imp.head(10)
top_features
sns.barplot(x=top_features, y=top_features.index)
# Add labels to your graph
plt.xlabel('Feature Importance Score')
plt.ylabel('Features')
plt.title("Visualizing Important Features")
plt.legend()
plt.show()<jupyter_output>No handles with labels found to put in legend.<jupyter_text>Above graph represents the importance of features from our test data in decending order.  Now we will estimate the important hyper-parametersTo find out important hyper-prameters, we will use Random Search with Cross Validation and Grid Search with Cross Validation<jupyter_code># Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(clf.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True,
 'class_weight': None,
 'criterion': 'gini',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 100,
 'n_jobs': None,
 'oob_score': False,
 'random_state': None,
 'verbose': 0,
 'warm_start': False}<jupyter_text>(1) Random Search with Cross validation<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 10, stop = 300, num = 5)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(2, 10, num = 5)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [2, 4]
# Method of selecting samples for training each tree
#bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}
pprint(random_grid)
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rfc_base = RandomForestClassifier()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rfc_random = RandomizedSearchCV(estimator=rfc_base, param_distributions=random_grid,
                              n_iter = 100,  
                              cv = 3, verbose=2, random_state=42, n_jobs=-1,
                              return_train_score=True)

# Fit the random search model
rfc_random.fit(X_train, y_train)<jupyter_output>Fitting 3 folds for each of 100 candidates, totalling 300 fits<jupyter_text>Below hyper-parameters are important<jupyter_code>rfc_random.best_params_<jupyter_output><empty_output><jupyter_text>Now, we will eveluate our Random Search by evaluating AUC<jupyter_code>def evaluate(model, X_train,y_train, X_test, y_test):
    model.fit(X_train,y_train)
    predictions = model.predict(X_test)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    #print(roc)
    return roc_auc
base_model = RandomForestClassifier(n_estimators = 10, random_state = 42)
best_random = rfc_random.best_estimator_
print('AUC for best model : ' + str(evaluate(best_random,X_train,y_train, X_test, y_test)))<jupyter_output>AUC for best model : 0.9028382894849601<jupyter_text>(2) Grid Search with Cross Validation<jupyter_code># Create the parameter grid based on the results of random search 
param_grid = {
    'bootstrap': [True],
    'max_features': ['auto'],
    'min_samples_leaf': [1,2,3],
    'min_samples_split': [3,5,7],
    'n_estimators': [100, 200,2000]
}

# Create a base model
clf = RandomForestClassifier()

# Instantiate the grid search model
grid_search = GridSearchCV(estimator = clf, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train, y_train);
grid_search.best_params_
best_grid = grid_search.best_estimator_
print('AUC for best grid search model : ' + str(evaluate(best_grid,X_train,y_train, X_test, y_test)))<jupyter_output>AUC for best grid search model : 0.902194608697308<jupyter_text>2. XGBoost Classifier<jupyter_code>xgBoostClassifierModel = XGBClassifier()
xgBoostClassifierModel.fit(X_train, y_train)
# make predictions for test data
y_pred = xgBoostClassifierModel.predict(X_test)
predictions = [round(value) for value in y_pred]
# evaluate predictions
accuracy = accuracy_score(y_test, predictions)
print("Accuracy: %.2f%%" % (accuracy * 100.0))<jupyter_output>Accuracy: 90.28%<jupyter_text>Estimation the important hyper-parameters of XGBoostWe will implement Grid Search to tune the hyper-parameters of XGBoost. First we will find out the best number of trees "n_estimators".<jupyter_code>model = XGBClassifier()
n_estimators = range(50, 400, 50)
param_grid = dict(n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
# plot
plt.errorbar(n_estimators, means, yerr=stds)
plt.title("XGBoost n_estimators vs Log Loss")
plt.xlabel('n_estimators')
plt.ylabel('Log Loss')<jupyter_output>Best: -0.211984 using {'n_estimators': 350}
-0.255265 (0.009489) with: {'n_estimators': 50}
-0.230343 (0.009820) with: {'n_estimators': 100}
-0.221181 (0.009622) with: {'n_estimators': 150}
-0.217069 (0.008981) with: {'n_estimators': 200}
-0.214744 (0.008929) with: {'n_estimators': 250}
-0.213037 (0.008564) with: {'n_estimators': 300}
-0.211984 (0.008411) with: {'n_estimators': 350}<jupyter_text>It is evident from the above graph that best number of trees was n_estimators=350 resulting in a log loss of -0.211984. Here cross validation log loss are negative and this is because scikit-learn cross validation framework inverted them. he reason is that internally, the framework requires that all metrics that are being optimized are to be maximized, whereas log loss is a minimization metric. It can easily be made maximizing by inverting the scores.  Tuning the size of decision trees in XGBoost<jupyter_code>model = XGBClassifier()
max_depth = range(1, 11, 2)
print(max_depth)
param_grid = dict(max_depth=max_depth)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
#Plotting the graph between max_depth and log loss
plt.errorbar(max_depth, means, yerr=stds)
plt.title("XGBoost max_depth vs Log Loss")
plt.xlabel('max_depth')
plt.ylabel('Log Loss')<jupyter_output><empty_output><jupyter_text>After plotting the graph between max_depth Vs Log Loss, we can see that best value for size of trees is 7.  Tuning Number of Trees and Max Depth in XGBoost<jupyter_code>model = XGBClassifier()
n_estimators = [350, 300, 250, 400]
max_depth = [6, 7, 8, 9]
print(max_depth)
param_grid = dict(max_depth=max_depth, n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))<jupyter_output>[6, 7, 8, 9]
Fitting 10 folds for each of 16 candidates, totalling 160 fits<jupyter_text>Plotting the graph between n_estimators Vs Log Loss<jupyter_code>scores = np.array(means).reshape(len(max_depth), len(n_estimators))
for i, value in enumerate(max_depth):
    plt.plot(n_estimators, scores[i], label='depth: ' + str(value))
plt.legend()
plt.xlabel('n_estimators')
plt.ylabel('Log Loss')<jupyter_output><empty_output><jupyter_text>Model with the best values for Hyper parameters<jupyter_code>model = XGBClassifier(max_depth=6, n_estimators=250)
print(model)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
predictions = [round(value) for value in y_pred]
print('ROC for the XGBoost model is: ' + str(evaluate(model, X_train,y_train, X_test, y_test)))<jupyter_output>ROC for the XGBoost model is: 0.9073363422553585<jupyter_text>Classification Model    ROC         XGBoost Classifier    0.9073           Random Forest Classifier    0.9021    From above table we can see that ROC for XGBoost classifier is higher than that of Random forest Classifier. Hence we can conclude that XGBoost classifier works best for this dataset.  Part B - Classification using Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
import statsmodels.api as sm
data['price_cat_log'] = data['price'].apply(lambda x: 1 if x > median_price else 0)
price_cat_log=data['price_cat_log']
y_log=price_cat_log
X_train_log,X_test_log,y_train_log,y_test_log=train_test_split(data[['bedrooms', 'sqft_living', 'waterfront', 'sqft_basement', 'yr_built', 'bathrooms']]
                                                               ,y_log,test_size=0.3,random_state=0)
logreg=LogisticRegression().fit(X_train_log,y_train_log)
logreg
print("Training set score: {:.3f}".format(logreg.score(X_train_log,y_train_log)))
print("Test set score: {:.3f}".format(logreg.score(X_test_log,y_test_log)))
logit_model=sm.Logit(y_train_log,X_train_log)
result=logit_model.fit()
print(result.summary())<jupyter_output>Optimization terminated successfully.
         Current function value: 0.522835
         Iterations 7
                           Logit Regression Results                           
==============================================================================
Dep. Variable:          price_cat_log   No. Observations:                15129
Model:                          Logit   Df Residuals:                    15123
Method:                           MLE   Df Model:                            5
Date:                Tue, 29 Jan 2019   Pseudo R-squ.:                  0.2457
Time:                        17:46:18   Log-Likelihood:                -7910.0
converged:                       True   LL-Null:                       -10486.
                                        LLR p-value:                     0.000
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
-----------------------[...]<jupyter_text>From above summary, it is evident that most significant values of Z-score is for "sqft_living" and "yr_built".<jupyter_code>from sklearn.metrics import roc_curve
falsepos,truepos,thresholds=roc_curve(y_test_log,logreg.decision_function(X_test_log))

plt.plot(falsepos,truepos,label="ROC")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")

cutoff=np.argmin(np.abs(thresholds))
plt.plot(falsepos[cutoff],truepos[cutoff],'o',markersize=10,label="cutoff",fillstyle="none")
plt.show()
from sklearn import metrics
metrics.auc(falsepos, truepos)<jupyter_output><empty_output><jupyter_text>Classification Model    AUC           XGBoost Classifier    0.9073         Random Forest Classifier    0.9021           Logistics Regression    0.8078     Above table shows that Logistic Regression model gives less ROC value compared with XGBoost and Random Forest. Best model for classification for this dataset is XGBoost.  Part D - Regression with Trees  1. Random Forest Regression<jupyter_code>from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(random_state = 42)
from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(rf.get_params())
X_features_reg = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']

X_reg=data[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']]  # Features

y_reg=data['price']  # Labels
       
# Split dataset into training set and test set
X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(X_reg, y_reg, test_size=0.3) # 70% training and 30% test<jupyter_output><empty_output><jupyter_text>We will estimate the important hyper-parameters There are two methods by which we can find out imporatant hyper-parameters.1. Random search with Cross validation2. Grid Search with Cross valudation  Random search with Cross validation<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 10, stop = 200, num = 5)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(2, 10, num = 5)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [2, 4]
# Method of selecting samples for training each tree
#bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}
pprint(random_grid)
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = RandomForestRegressor()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
# Fit the random search model
rf_random.fit(X_train_reg, y_train_reg)
def evaluate_reg(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100 * np.mean(errors / test_labels)
    accuracy = 100 - mape
    print('Model Performance')
    print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%.'.format(accuracy))
    
    return accuracy
base_model = RandomForestRegressor(n_estimators = 10, random_state = 42)
base_model.fit(X_train_reg, y_train_reg)
base_accuracy = evaluate_reg(base_model, X_test_reg, y_test_reg)
rf_random.best_estimator_
best_random = rf_random.best_estimator_
random_accuracy = evaluate_reg(best_random, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 70579.4860 degrees.
Accuracy = 86.68%.<jupyter_text>Grid search with Cross Validation<jupyter_code>from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {
    'bootstrap': [True, False],
    'max_depth': [None],
    'max_features': [2, 3],
    'min_samples_leaf': [2,3,4],
    'min_samples_split': [3,5,7],
    'n_estimators': [100, 200, 300, 1000]
}
# Create a based model
rf = RandomForestRegressor()

# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
# Fit the grid search to the data
grid_search.fit(X_train_reg, y_train_reg)
grid_search.best_params_
grid_search.best_params_
best_grid = grid_search.best_estimator_
grid_accuracy = evaluate_reg(best_grid, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 91189.4563 degrees.
Accuracy = 81.98%.<jupyter_text>2. XGBoost Regression<jupyter_code>from xgboost import XGBRegressor
base_model = XGBRegressor()
base_model.fit(X_train_reg, y_train_reg)
base_accuracy = evaluate_reg(base_model, X_test_reg, y_test_reg)
base_model<jupyter_output><empty_output><jupyter_text>Now we will find out best hyper-parameters for XGBoost Firstly, we will tune the "n_estimators":<jupyter_code>model = XGBRegressor()
n_estimators = range(50, 400, 50)
param_grid = dict(n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, n_jobs=-1, cv=kfold)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
best_XGBoost = grid_result.best_estimator_
best_XGBoost
best_XGBoost.fit(X_train_reg, y_train_reg)
best_accuracy = evaluate_reg(best_XGBoost, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 72681.2423 degrees.
Accuracy = 86.21%.<jupyter_text>3. Linear Regression<jupyter_code>from sklearn.linear_model import LinearRegression
regression = LinearRegression().fit(X_train_reg,y_train_reg)
evaluate_reg(regression,X_test_reg,y_test_reg)<jupyter_output>Model Performance
Average Error: 120357.4160 degrees.
Accuracy = 76.38%.<jupyter_text>Part E: Gradient Boosting Regression<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor
gradient_boosting = GradientBoostingRegressor()
gradient_boosting.fit(X_train_reg,y_train_reg)
#prediction = gradient_boosting.predict(X_test)
gradient_boosting.score(X_test_reg,y_test_reg)<jupyter_output><empty_output><jupyter_text>We can see that without hyper-parameters tuning, accuracy is 84.81%. Now we will tune the hyper-parameters:<jupyter_code>param_grid={'n_estimators':[100,400,800,1000], 'learning_rate': [0.25,0.35,0.6],# 0.05, 0.02, 0.01],
            'max_depth':[4,6], 
            'min_samples_leaf':[3,5,8], 
            'max_features':[0.5,0.3,1.0]
           }
regressor = GridSearchCV(estimator=gradient_boosting, cv=3, param_grid=param_grid, n_jobs=16,verbose=1)
regressor.fit(X_train_reg, y_train_reg)
regressor.best_estimator_
gradient_boosting_tunned = GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
             learning_rate=0.25, loss='ls', max_depth=4, max_features=1.0,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=3,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=400, n_iter_no_change=None, presort='auto',
             random_state=None, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=0, warm_start=False)
gradient_boosting_tunned.fit(X_train_reg,y_train_reg)
evaluate_reg(gradient_boosting_tunned,X_test_reg,y_test_reg)<jupyter_output>Model Performance
Average Error: 67516.7990 degrees.
Accuracy = 87.04%.<jupyter_text>Below are the best hyper-parameters:<jupyter_code>print(regressor.best_params_)<jupyter_output>{'learning_rate': 0.25, 'max_depth': 4, 'max_features': 0.5, 'min_samples_leaf': 3, 'n_estimators': 400}
{'fraction_non_alphanumeric': 0.11707974328972096, 'fraction_numerical': 0.036946424139275964, 'mean_word_length': 5.079758568657038, 'pattern_counts': {'":': 2, '<': 97, '<?xml version=': 0, '>': 100, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '221778', 'license': 'MIT', 'n_tokens_mistral': 10238, 'n_tokens_neox': 9357, 'n_words': 2534, 'path': 'Assignment_1.ipynb', 'repo_name': 'pratikkadi/House-Price-Prediction-of-King-County', 'size': '28203'}