<jupyter_start><jupyter_text>Assignment 1  Abstract `Aim`In the assignment below, we are trying to run the listed alorithms to find the best metrics of the model. Also, trying to find the best hyperparameters for the fit and fine tune it. We perform our analysis on the basis of the hyperparameter found and the matrices for the respective model fit. In general we compare:* Which hyper-parameters are important?* What hyper-parameter values work best?* Which supervised learner works best on the test data?And have concluded the result at the end with our analysis. `Methods`1. Random forest Classifer2. Random Forest Regression3. XGBoost Classifier & XGBoost Regression4. Logistic Regression5. Redge Regression6. Linear Regression7. k-Nearest Neighbors (NN) Regression `Data Source`I am using the King County Housing Data set. It has 19375 rows and 51 columns. There is a mix of quantitative and qualitative feautres.The datset from - https://www.kaggle.com/harlfoxem/housesalesprediction/home has been cleansed and provided with one hot encoding for the columns - "Floor", "view", "conditions", "zipcode" & "grade". The top 6 best zipcode is choosen and represented by the column.On performing the algorithms, we try ti find the matrices for different value of paramaters and try to tune the parameters to get a good fir for the algorithm.<jupyter_code>import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from pandas import Series
%matplotlib inline
import requests

from sklearn.model_selection import train_test_split

from sklearn import metrics
import warnings
warnings.filterwarnings('ignore')
# url="https://github.com/cs109/2014_data/blob/master/countries.csv"
features=pd.read_csv("./kc_house_data.csv")
features.head()
features.describe()<jupyter_output><empty_output><jupyter_text>Below, we are trying to find the correlation between the columns. We performe this, in order to understand the relationship between the columns.  To give an explanation with example : The columns 'bedrooms' & 'price' are higly corelated with a +ve realtion with each other. Which means, if bedrooms increases, the price of the appartment increases. Similarly, the realtion between the waterfront and bedroom is less corelated, which means they are not dependent on each other. A better intrepretation for a columns bedroom w.r.t grades will be that as grades increase 12345 and so on, the number of bedroom inceases<jupyter_code>corr = features.corr()

mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(20, 20))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(190, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})<jupyter_output><empty_output><jupyter_text>The above correlation has been represented as a matrix, for visualisation purpose in a matrix format<jupyter_code>plt.matshow(features.corr())<jupyter_output><empty_output><jupyter_text>We find the median for the column "price" to further create a new categorical column called "price_cat". The new columns deferetiates the data greater or lessr than the median found for "price"<jupyter_code>median_price = features.loc[:,"price"].median()
print(median_price)<jupyter_output>450000.0<jupyter_text>Adding column to dataframe for holding categorical price value : HIGH/LOWAll values = median are HIGH<jupyter_code>features['price_cat'] = features['price'].apply(lambda x: 1 if x > median_price else -1)
features[['price','price_cat']].head(10)
features.head(5)<jupyter_output><empty_output><jupyter_text>Trying to find null/NaN values in the dataset, and as a result, there is one row consisting null value<jupyter_code>features.isnull().sum()<jupyter_output><empty_output><jupyter_text>We drop the drop the row and proceed ahead. As the null values data can rise it to biasing<jupyter_code>features=features.dropna()
features.isnull().sum()<jupyter_output><empty_output><jupyter_text>`Part A`We shall be performing thr below algorithms for classification model.* Bagging Algo : Random Forest Classifier* Boosting Algo : XGBoost ClassifierWe use Bagging and Boosting algorithm because they give us `LESS` biasing and vairance. They avoid overfitting the model prediction.  Random Forest Classifier<jupyter_code>features.columns<jupyter_output><empty_output><jupyter_text>In the below, we initially store the columns as features for further reference.Then, we split the dataset into training and test data in the ratio of 7:3<jupyter_code>X_features = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']

X=features[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']]  # Features

y=features['price_cat']  # Labels
       
# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3) # 70% training and 30% test<jupyter_output><empty_output><jupyter_text>Further, we perfrom Random Forest Bagging algorithm. We perfrom the fit on the test dataset for the prediction of the model<jupyter_code>#Import Random Forest Model
from sklearn.ensemble import RandomForestClassifier

#Create a Gaussian Classifier
clf=RandomForestClassifier(n_estimators=100)

#Train the model using the training sets y_pred=clf.predict(X_test)
clf.fit(X_train,y_train)

y_pred=clf.predict(X_test)
#Import scikit-learn metrics module for accuracy calculation
from sklearn import metrics
# Model Accuracy, how often is the classifier correct?
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))<jupyter_output>Accuracy: 0.906760708756236<jupyter_text>Now, as we can see the accurancy turned out to be around 0.9. Its a good measure of performace as our model has performed well.Further, we check and select few parameters, to tune the model well.Through random forest meta estimator we check the number of decision tree classifiers on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting.The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if `bootstrap=True` (default).<jupyter_code>RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False)<jupyter_output><empty_output><jupyter_text>Further, we select the imporatant parameters for the tree and store it in a variable to use it. And sort it as per the value obtained.We use it to store in an array using pandas series by indexing it with the columns of the dataset that was stored previously.<jupyter_code>feature_imp = pd.Series(clf.feature_importances_,index=X_features).sort_values(ascending=False)
feature_imp<jupyter_output><empty_output><jupyter_text>Top features<jupyter_code>top_features = feature_imp.head(10)
top_features<jupyter_output><empty_output><jupyter_text>We plot a graph to show the imporatant features and their score of importance.From the Bar graph plotted, we can infer that 'lat' 'sqft_living''sqft_livinf15' 'sqft_above' 'long' 'sqft_lot15' are the top 5 important featues<jupyter_code># Creating a bar plot
sns.barplot(x=top_features.index, y=top_features, orient="v")
# Add labels to your graph
plt.xlabel('Features')
plt.ylabel('Feature Importance Score')
plt.title("Visualizing Important Features")
plt.xticks(rotation=90)
plt.legend()
plt.show()
from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(clf.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True,
 'class_weight': None,
 'criterion': 'gini',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 100,
 'n_jobs': 1,
 'oob_score': False,
 'random_state': None,
 'verbose': 0,
 'warm_start': False}<jupyter_text>According to the documentation, the most important settings are the number of trees in the forest (n_estimators) and the number of features considered for splitting at each leaf node (max_features)We will try adjusting the following set of hyperparameters: Finding important hyper-parametersThe important parameters for a Random Forest Classfier are as follows:  We will use RandomSearchCV and GridSearchCV to find the best values for the same  1. RandomSearchCVWe perform this for Cross Validating our finding. The cross validation helps us in ignoring the biasing and overfitting if in case our model is providinf. We check the peformance by cross validating.<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 10, stop = 200, num = 5)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(2, 10, num = 5)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [2, 4]
# Method of selecting samples for training each tree
#bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}
pprint(random_grid)<jupyter_output>{'max_depth': [2, 4, 6, 8, 10, None],
 'max_features': ['auto', 'sqrt'],
 'min_samples_leaf': [2, 4],
 'min_samples_split': [5, 10],
 'n_estimators': [10, 57, 105, 152, 200]}<jupyter_text>We have listed the parameters and the values that wil be considered with the range of it using `Grid Search`.Further, with those values of parameters we try to fit our model to check its performance, using RandomSearchCV.<jupyter_code># Use the random grid to search for best hyperparameters
# First create the base model to tune
clf_base = RandomForestClassifier()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
clf_random = RandomizedSearchCV(estimator = clf_base, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
# Fit the random search model
clf_random.fit(X_train,y_train)
clf_random.best_params_<jupyter_output><empty_output><jupyter_text>Thus, we listed out the best hyperparameters value for the model for its best accurancy.  Evaluate Random Search We define a function to evaluate the model and perform the prediction on the test dataset and return the `AUC` metric of the model.<jupyter_code>## Common Classification algorithm evaluator. It returns the roc_auc
from sklearn.metrics import roc_curve, auc
def evaluate(model, X_train,y_train, X_test, y_test):
    model.fit(X_train,y_train)
    predictions = model.predict(X_test)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    #print(roc)
    return roc_auc<jupyter_output><empty_output><jupyter_text>As seen the above `best_params_` found, the `n_estimator` is `57`. Applying the same will give us the best fit for the model evaluation. Thus, in the previous code, the `n_estimator` was `10`, changin it to `57` gives a better `ROC` change from `0.80` to `0.9058`<jupyter_code>base_model = RandomForestClassifier(n_estimators = 57, random_state = 42)
best_random = clf_random.best_estimator_
print('ROC for base model : ' + str(evaluate(base_model,X_train,y_train, X_test, y_test)))
print('ROC for best model : ' + str(evaluate(best_random,X_train,y_train, X_test, y_test)))<jupyter_output>ROC for base model : 0.9058417857350242
ROC for best model : 0.9078105168852499<jupyter_text>On evaluation, we found in the above is that `0.9078` ROC value on the best model is a good denotion of its accuracy for a model.  2. Grid Search with Cross Validation Now, we will perfrom a GridSearch Cross Validation, while doing so, we give a paramater range in order to start checking the accurace of the model fit.<jupyter_code>from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {
    'bootstrap': [True],
    'max_features': ['auto'],
    'min_samples_leaf': [1,2,3],
    'min_samples_split': [3,5,7],
    'n_estimators': [100, 200, 2000]
}
# Create a based model
clf = RandomForestClassifier()
# Instantiate the grid search model
grid_search = GridSearchCV(estimator = clf, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train, y_train)
grid_search.best_params_<jupyter_output><empty_output><jupyter_text>Thus, the best paramter with their values for the best fit is listed above.<jupyter_code>best_grid = grid_search.best_estimator_
print('ROC for best grid search model : ' + str(evaluate(best_grid,X_train,y_train, X_test, y_test)))<jupyter_output>ROC for best grid search model : 0.9078105168852499<jupyter_text>Now, using grid search we find the ROC for the model fit to be `0.9078` which matches our RandomForest CV. Now from the RandomForest and XGBoost algorithm model excution below were the parameters we could find from each respectively on perfomring Cross Validation for each respectively:* RandonForest`{'n_estimators': 57, 'min_samples_split': 5, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': None}`  Through GridSearch we got the below best parameters:  `{'bootstrap': True, 'max_features': 'auto', 'min_samples_leaf': 1, 'min_samples_split': 3, 'n_estimators': 2000}`  From the above we can infer the BEST hyper prameters for tuning the model are as below:  * `n_estimators` * `min_samples_leaf` * `max_features` * `min_samples_split`  Now lets implement XGBoost (eXtreme Gradient Boosting) I’ll consider only tree booster here because it always outperforms the linear booster and thus the later is rarely used.<jupyter_code>from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score
xgBoostClassifierModel = XGBClassifier()
xgBoostClassifierModel.fit(X_train, y_train)
# make predictions for test data
y_pred = xgBoostClassifierModel.predict(X_test)
predictions = [round(value) for value in y_pred]

# evaluate predictions
accuracy = accuracy_score(y_test, predictions)
print("Accuracy: %.2f%%" % (accuracy * 100.0))<jupyter_output>Accuracy: 90.61%<jupyter_text>Accuracy 90.61% is a good messure of the model performace. Further we will perform the GridSearch to check for the best parameter fit for the model build<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold<jupyter_output><empty_output><jupyter_text>Wecheck the run for `n_estimators` and plot the values for n_estimators against its log loss<jupyter_code>model = XGBClassifier()
n_estimators = range(50, 400, 50)
param_grid = dict(n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
# plot
plt.errorbar(n_estimators, means, yerr=stds)
plt.title("XGBoost n_estimators vs Log Loss")
plt.xlabel('n_estimators')
plt.ylabel('Log Loss')
plt.savefig('n_estimators.png')<jupyter_output>Best: -0.209829 using {'n_estimators': 350}
-0.257586 (0.007068) with: {'n_estimators': 50}
-0.230445 (0.007295) with: {'n_estimators': 100}
-0.220998 (0.007793) with: {'n_estimators': 150}
-0.215923 (0.007868) with: {'n_estimators': 200}
-0.212878 (0.008351) with: {'n_estimators': 250}
-0.211141 (0.008607) with: {'n_estimators': 300}
-0.209829 (0.008935) with: {'n_estimators': 350}<jupyter_text>We can see that the cross validation log loss scores are negative. This is because the scikit-learn cross validation framework inverted them. The reason is that internally, the framework requires that all metrics that are being optimized are to be maximized, whereas log loss is a minimization metric. It can easily be made maximizing by inverting the scores.The best number of trees was n_estimators=350 resulting in a log loss of -0.213537, but really not a significant difference from n_estimators=200. In fact, there is not a large relative difference in the number of trees between 100 and 350 if we plot the results. Thus, as the number increases, it saturates and doesnt give any significant increase in the log loss  Tune the Size of Decision Trees in XGBoost In gradient boosting, we can control the size of decision trees, also called the number of layers or the depth.Shallow trees are expected to have poor performance because they capture few details of the problem and are generally referred to as weak learners. Deeper trees generally capture too many details of the problem and overfit the training dataset, limiting the ability to make good predictions on new data.`Generally, boosting algorithms are configured with weak learners, decision trees with few layers, sometimes as simple as just a root node, also called a decision stump rather than a decision tree.`The maximum depth can be specified in the XGBClassifier and XGBRegressor wrapper classes for XGBoost in the `max_depth` parameter. This parameter takes an integer value and `defaults to a value of 3`.<jupyter_code>model = XGBClassifier()
max_depth = range(1, 11, 2)
print(max_depth)
param_grid = dict(max_depth=max_depth)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))
plt.errorbar(max_depth, means, yerr=stds)
plt.title("XGBoost max_depth vs Log Loss")
plt.xlabel('max_depth')
plt.ylabel('Log Loss')
plt.savefig('max_depth.png')<jupyter_output><empty_output><jupyter_text>Reviewing the plot of log loss scores, we can see a marked `jump from max_depth=1 to max_depth=3` then pretty even performance for the rest the values of max_depth.Although the best score was observed for max_depth=5, it is interesting to note that there was practically little difference between using max_depth=3 or max_depth=7.Further, if we see, the value of logg loss depriciates as the max_depth value increases. Thus, the max value of log loss is at `max_depth` of around `7`This suggests a point of diminishing returns in max_depth on a problem that you can tease out using grid search. A graph of max_depth values is plotted against (inverted) logarithmic loss below.  Tune The Number of Trees and Max Depth in XGBoost There is a relationship between the number of trees in the model and the depth of each tree.We would expect that deeper trees would result in fewer trees being required in the model, and the inverse where simpler trees (such as decision stumps) require many more trees to achieve similar results.We can investigate this relationship by evaluating a grid of n_estimators and max_depth configuration values. To avoid the evaluation taking too long, we will limit the total number of configuration values evaluated. Parameters were chosen to tease out the relationship rather than optimize the model.We will create a grid of 4 different n_estimators values (350, 300, 250, 400) and 4 different max_depth values (6, 7, 8, 9) and each combination will be evaluated using 10-fold cross validation. A total of 4410 or 160 models will be trained and evaluated.<jupyter_code># grid search
model = XGBClassifier()
n_estimators = [350, 300, 250, 400]
max_depth = [6, 7, 8, 9]
print(max_depth)
param_grid = dict(max_depth=max_depth, n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
    
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))<jupyter_output>[6, 7, 8, 9]
Fitting 10 folds for each of 16 candidates, totalling 160 fits<jupyter_text>Now, as seen above, we try ti find the log loss for a range of n_extimators given. And find that the best(which denotes min log loss) comes around the configuration of `max_depth = 6 & n_estimator = 250` Now based on the 4 different range for n_estimator we plot the log loss to perform our analysis<jupyter_code># plot results
scores = np.array(means).reshape(len(max_depth), len(n_estimators))
for i, value in enumerate(max_depth):
    plt.plot(n_estimators, scores[i], label='depth: ' + str(value))
plt.legend()
plt.xlabel('n_estimators')
plt.ylabel('Log Loss')
plt.savefig('n_estimators_vs_max_depth.png')<jupyter_output><empty_output><jupyter_text>We can see that the best result was achieved with a n_estimators=250 and max_depth=6, similar to the best values found from the previous two rounds of standalone parameter tuning (n_estimators=250, max_depth=5).We can plot the relationship between each series of max_depth values for a given n_estimators.Fewer boosted trees are required with increased tree depth.Further, we would expect the increase complexity provided by deeper individual trees to result in greater overfitting of the training data which would be exacerbated by having more trees, in turn resulting in a lower cross validation score. We don’t see this here as our trees are not that deep nor do we have too many. Exploring this expectation is left as an exercise you could explore yourself.<jupyter_code>model = XGBClassifier(max_depth=6, n_estimators=250)
print(model)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
predictions = [round(value) for value in y_pred]
print('ROC for the XGBoost model is: ' + str(evaluate(model, X_train,y_train, X_test, y_test)))<jupyter_output>ROC for the XGBoost model is: 0.9165764840142064<jupyter_text>`Conclusion` Thus we got a higher ROC for XGBoost model than the bagging models tried earlier  `Part B`  Classification with Logistic Regression Compare logistic regression with the based tree algorithms in Part A on the same data. Answer the following questions for logistic regression:Which independent variables are important?Is there multi-colinearity?How does it compare to the the models in Part A?We need to develop a Nominal Logistic Regression model which will predict whether the house price is HIGH(1) or LOW(-1)<jupyter_code>import statsmodels.api as sm
import statsmodels.formula.api as smf
import pandas as pd
from sklearn.linear_model import LogisticRegression
features['price_cat_log'] = features['price'].apply(lambda x: 1 if x > median_price else 0)
features.columns
price_cat_log=features['price_cat_log']
y_log=price_cat_log
X_train_log,X_test_log,y_train_log,y_test_log=train_test_split(features[['bedrooms', 'sqft_living', 'waterfront', 'sqft_basement', 'yr_built', 'bathrooms']]
                                                               ,y_log,test_size=0.3,random_state=0)
logreg=LogisticRegression().fit(X_train_log,y_train_log)
logreg
print("Training set score: {:.3f}".format(logreg.score(X_train_log,y_train_log)))
print("Test set score: {:.3f}".format(logreg.score(X_test_log,y_test_log)))
logit_model=sm.Logit(y_train_log,X_train_log)
result=logit_model.fit()
print(result.summary())<jupyter_output>Optimization terminated successfully.
         Current function value: 0.520241
         Iterations 8
                           Logit Regression Results                           
==============================================================================
Dep. Variable:          price_cat_log   No. Observations:                13561
Model:                          Logit   Df Residuals:                    13555
Method:                           MLE   Df Model:                            5
Date:                Wed, 30 Jan 2019   Pseudo R-squ.:                  0.2493
Time:                        03:15:43   Log-Likelihood:                -7055.0
converged:                       True   LL-Null:                       -9398.3
                                        LLR p-value:                     0.000
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
-----------------------[...]<jupyter_text>Looking at the z scores, we know that, `sqft_living` is the most significant followed by `yr_built` and then the rest.<jupyter_code>from sklearn.metrics import roc_curve
falsepos,truepos,thresholds=roc_curve(y_test_log,logreg.decision_function(X_test_log))

plt.plot(falsepos,truepos,label="ROC")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")

cutoff=np.argmin(np.abs(thresholds))
plt.plot(falsepos[cutoff],truepos[cutoff],'o',markersize=10,label="cutoff",fillstyle="none")
plt.show()<jupyter_output><empty_output><jupyter_text>Now, from the `AUC-ROC curve` above - at the cordinates `(0.2,0.7)` lies the threshold for the curve.<jupyter_code>from sklearn import metrics
metrics.auc(falsepos, truepos)<jupyter_output><empty_output><jupyter_text>Now as the `AUC = 0.8164` which is closer to 1, gives a moderate value of performance. But, the result of comparison between the regression model between: `Random Forest`, `XGBoost` & `Logistic` is as below. As compared to Models in Part A, the accuracy of the Logistic Regression model is less.  `Part C`  Regression with Trees Compare at least one bagging based tree algorithm (e.g. Random Forest) with a boosting based tree algorithm on a regression problem. Answer the following questions for each supervised learner:Which hyper-parameters are important?What hyper-parameter values work best?Which supervised learner works best on the test data?  Random Forest based RegressionTo look at the available hyperparameters, we can create a random forest and examine the default values.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(random_state = 42)
from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(rf.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True,
 'criterion': 'mse',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 10,
 'n_jobs': 1,
 'oob_score': False,
 'random_state': 42,
 'verbose': 0,
 'warm_start': False}<jupyter_text>* estimators = number of trees in the foreset* max_features = max number of features considered for splitting a node* max_depth = max number of levels in each decision tree* min_samples_split = min number of data points placed in a node before the node is split* min_samples_leaf = min number of data points allowed in a leaf node* bootstrap = method for sampling data points (with or without replacement)<jupyter_code>X_features_reg = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']

X_reg=features[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'waterfront', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated',
       'lat', 'long', 'sqft_living15', 'sqft_lot15', 'basement_present',
       'renovated', 'floors#1.0', 'floors#1.5', 'floors#2.0', 'floors#2.5',
       'floors#3.0', 'floors#3.5', 'view#0', 'view#1', 'view#2', 'view#3',
       'view#4', 'condition#1', 'condition#2', 'condition#3', 'condition#4',
       'condition#5', 'grade#1', 'grade#3', 'grade#4', 'grade#5', 'grade#6',
       'grade#7', 'grade#8', 'grade#9', 'grade#10', 'grade#11', 'grade#12',
       'grade#13', 'zipcode#98004', 'zipcode#98102', 'zipcode#98109',
       'zipcode#98112', 'zipcode#98039', 'zipcode#98040']]  # Features

y_reg=features['price']  # Labels
       
# Split dataset into training set and test set
X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(X_reg, y_reg, test_size=0.3) # 70% training and 30% test<jupyter_output><empty_output><jupyter_text>Random Search Cross Validation in Scikit-Learn To use RandomizedSearchCV, we first need to create a parameter grid to sample from during fitting:<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 10, stop = 200, num = 5)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(2, 10, num = 5)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [2, 4]
# Method of selecting samples for training each tree
#bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}
pprint(random_grid)
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = RandomForestRegressor()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
# Fit the random search model
rf_random.fit(X_train_reg, y_train_reg)
def evaluate_reg(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100 * np.mean(errors / test_labels)
    accuracy = 100 - mape
    print('Model Performance')
    print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%.'.format(accuracy))
    
    return accuracy
base_model = RandomForestRegressor(n_estimators = 10, random_state = 42)
base_model.fit(X_train_reg, y_train_reg)
base_accuracy = evaluate_reg(base_model, X_test_reg, y_test_reg)
rf_random.best_estimator_
best_random = rf_random.best_estimator_
random_accuracy = evaluate_reg(best_random, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 71018.6331 degrees.
Accuracy = 86.97%.<jupyter_text>Grid Search with Cross Validation<jupyter_code>from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {
    'bootstrap': [True, False],
    'max_depth': [None],
    'max_features': [2, 3],
    'min_samples_leaf': [2,3,4],
    'min_samples_split': [3,5,7],
    'n_estimators': [100, 200, 300, 1000]
}
# Create a based model
rf = RandomForestRegressor()

# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
# Fit the grid search to the data
grid_search.fit(X_train_reg, y_train_reg)
grid_search.best_params_
best_grid = grid_search.best_estimator_
grid_accuracy = evaluate_reg(best_grid, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 89737.4405 degrees.
Accuracy = 82.58%.<jupyter_text>Thus, for Random Forest Regression, the best parameters are as below with the values:`'bootstrap': False, 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 2, 'min_samples_split': 3, 'n_estimators': 1000`  Now lets implement XGBoost (eXtreme Gradient Boosting)<jupyter_code>from xgboost import XGBRegressor
base_model = XGBRegressor()
base_model.fit(X_train_reg, y_train_reg)
base_accuracy = evaluate_reg(base_model, X_test_reg, y_test_reg)
base_model
model = XGBRegressor()
n_estimators = range(50, 400, 50)
param_grid = dict(n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, n_jobs=-1, cv=kfold)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
model = XGBRegressor()
max_depth = range(1, 11, 2)
print(max_depth)
param_grid = dict(max_depth=max_depth)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train_reg, y_train_reg)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))<jupyter_output>range(1, 11, 2)
Fitting 10 folds for each of 5 candidates, totalling 50 fits<jupyter_text>Grid Search with Cross Validation for XGBoost<jupyter_code># grid search
model = XGBRegressor()
n_estimators = [300,350,400]
max_depth = [4,5,6,7]
print(max_depth)
param_grid = dict(max_depth=max_depth, n_estimators=n_estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)
grid_search = GridSearchCV(model, param_grid, n_jobs=-1, cv=kfold, verbose=1)
grid_result = grid_search.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
	print("%f (%f) with: %r" % (mean, stdev, param))
grid_result.best_estimator_
# best_XGBoost
grid_search.fit(X_train_reg, y_train_reg)
grid_search.best_params_
# grid_search.best_params_
best_XGBoost.fit(X_train_reg, y_train_reg)
best_accuracy = evaluate_reg(best_XGBoost, X_test_reg, y_test_reg)<jupyter_output>Model Performance
Average Error: 65977.5292 degrees.
Accuracy = 87.79%.<jupyter_text>It turns out to be only 2 important parameters for XGBoost regression problem. And they are as below:    `'max_depth': 5, 'n_estimators': 400`  Evaluating the regression models<jupyter_code>from sklearn.metrics import mean_squared_error
from math import sqrt<jupyter_output><empty_output><jupyter_text>`Conclusion` on important hyperprameters As seen, the end result comibing both Random Forest Regression and XGBoost hyperparamters:* Random Forest hyperparamters`'bootstrap': False,'max_depth': None,'max_features': 3,'min_samples_leaf': 2,'min_samples_split': 3,'n_estimators': 1000`* XGBoost hyperparamters`'max_depth': 5, 'n_estimators': 400`Thus, the best hyperparamters in combining them are `'max_depth', & 'n_estimators'`  `Part D`  Linear Regression Simple Linear regression on predicting `Price`<jupyter_code>from sklearn import cross_validation
# Train a simple linear regression model
regr = linear_model.LinearRegression()
new_data = features[['sqft_living', 'sqft_above', 'sqft_living15','bathrooms','sqft_basement','lat','waterfront','yr_built','bedrooms']]

X = new_data.values
y = features.price.values

# splitting the data into training and test in 8:2 ratio
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y ,test_size=0.2)

regr.fit(X_train, y_train)
regr.predict(X_test)
print("Score on Linear Regression"+" "+str(regr.score(X_test,y_test)))
# Calculate the Root Mean Squared Error
print("RMSE: %.2f" % math.sqrt(np.mean((regr.predict(X_test) - y_test) ** 2)))
#Random Forest best model
print('Random forest best model RMSE is : ' + str(round(sqrt(mean_squared_error(best_random.predict(X_test_reg), y_test_reg)),2)))
#XGBoost Best model
print('XGBoost best model RMSE is : ' + str(round(sqrt(mean_squared_error(best_XGBoost.predict(X_test_reg), y_test_reg)),2)))<jupyter_output>Random forest best model RMSE is : 137073.54
XGBoost best model RMSE is : 124984.51<jupyter_text>As seen the `score` and `RMSE` is high enough to prove that Linear Regression doesnt give a good measure of performace.  XGBoost Algo provides better predictions than Random Forest for the regression problem. And both of them provide a lower RMSE than Linear Regression  Regularization  Ridge Regression Ridge regression aims to avoid overfitting by adding a cost to the Residual Sum of Squares (RSS) term of standard least squares that depends on the 2-norm of the coefficients. The result is penalizing fits with large coefficients. The strength of this penalty, and thus the fit vs. model complexity, is controlled by a parameter lamda.Following the usual notation, suppose our regression equation is written in matrix form as  $$Y = XB + e$$where Y is the dependent variable, X represents the independent variables, B is the regression coefficients to beestimated, and e represents the errors are residuals. In short, in the Ridge Regression, we add a bit of biasing in order to avoiad overfitting of the model and generate a model for better prediction in a long run<jupyter_code>##Splitting the data in train-test split
from sklearn.cross_validation import train_test_split
from sklearn import linear_model
train_data, test_data = train_test_split(features, train_size = 0.8, random_state = 10)
train_data.shape
test_data.shape
test_data.describe()
train_data_2, validation_data = train_test_split(train_data, train_size = 0.75, random_state = 50)<jupyter_output><empty_output><jupyter_text>Let's test two models using alpha equal 1 and 10.<jupyter_code>input_feature = train_data.columns.values.tolist() # list of column name
input_feature.remove('price')
#input_feature


for i in [1,10]:
    ridge = linear_model.Ridge(alpha = i, normalize = True) # initialize the model
    ridge.fit(train_data.as_matrix(columns = input_feature), train_data.as_matrix(columns = ['price'])) # fit the train data

    
    mse = mean_squared_error(test_data.as_matrix(columns = ['price']).astype(np.int), 
                             ridge.predict(test_data.as_matrix(columns = [input_feature]).astype(np.int)))
    print ('test error (RMSE) is:' + str(mse))<jupyter_output>test error (RMSE) is:4.51545192561002e+31
test error (RMSE) is:3.2273681737795096e+30<jupyter_text>Ridge Cross Validation<jupyter_code>ridgeCV = linear_model.RidgeCV(alphas = np.linspace(1.0e-10,1,num = 100), normalize = True, store_cv_values = True) # initialize the model
ridgeCV.fit(train_data.as_matrix(columns = input_feature), train_data.as_matrix(columns = ['price'])) # fit the train data
print ('best alpha is: %s' %ridgeCV.alpha_) # get the best alpha
print ('test error (RMSE) is: %s' %mean_squared_error(test_data.as_matrix(columns = ['price']).astype(int), 
                              ridgeCV.predict(test_data.as_matrix(columns = [input_feature]).astype(int)))**0.5) # predict price and test error<jupyter_output>best alpha is: 1e-10
test error (RMSE) is: 2945413716627902.5<jupyter_text>Finally, we get a score of RMSE : `29e+14` which is too high for Ridge Regression problem  `Part E`Another supervised algorithm k-Nearest Neighbors (NN) Regression The k-NN algorithm is used for estimating continuous variables. One such algorithm uses a weighted average of the k nearest neighbors, weighted by the inverse of their distance. It is the nonparamtric equivalent of ordinary least square regression.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
from sklearn import preprocessing # provides functions and classes to change raw feature vectors
from sklearn import neighbors # provides functionality for unsupervised and supervised neighbors-based learning methods
# normalize the data
train_X = train_data.as_matrix(columns = input_feature)
scaler = preprocessing.StandardScaler().fit(train_X)
train_X_scaled = scaler.transform(train_X)
test_X = test_data.as_matrix(columns = [input_feature]).astype(int)
test_X_scaled = scaler.transform(test_X)

knn = neighbors.KNeighborsRegressor(n_neighbors=10, weights='distance') # initialize the model
knn.fit(train_X_scaled, train_data.as_matrix(columns = ['price']).astype(int)) # fit the train data
print ('test error (RMSE) is: %s' %mean_squared_error(test_data.as_matrix(columns = ['price']).astype(int), 
                              knn.predict(test_X_scaled))**0.5) # predict price and test error<jupyter_output>test error (RMSE) is: 430340.0895801561
{'fraction_non_alphanumeric': 0.10296489833011474, 'fraction_numerical': 0.028308531182551402, 'mean_word_length': 5.498745016979182, 'pattern_counts': {'":': 1, '<': 149, '<?xml version=': 0, '>': 152, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '220972', 'license': 'MIT', 'n_tokens_mistral': 14974, 'n_tokens_neox': 13825, 'n_words': 4511, 'path': '7245_Assignment1.ipynb', 'repo_name': 'prabhuSub/BDIA-Assignment-1', 'size': '44015'}