<jupyter_start><jupyter_text>Ensemble Model- **Course**: Data Mining- **Team Name**: The Awesome- **Team Members:** Enes, Kemal, ErginStep 0: Data Preparation and Cleaning---<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
# Read CSV data into df
df = pd.read_csv('./theAwesome_EnsModel.csv')
# delete id column no need
df.drop('Id',axis=1,inplace=True)
df.head()
# Learn the unique values in diagnosis column
print("Classification labels: ", df.Species.unique() )
# Mapping labels to numerical labels?
df.Species = df.Species.map({'Iris-setosa':0, 'Iris-versicolor':1, 'Iris-virginica':2})<jupyter_output><empty_output><jupyter_text>---Step 1: Data Information and Descriptive Statistics---Generate the information about your dataset: number of columns and rows, names and data types ofthe columns, memory usage of the dataset.> Hint: Pandas data frame info() function.Generate descriptive statistics of all columns (input and output) of your dataset. Descriptive statistics fornumerical columns include: count, mean, std, min, 25 percentile (Q1), 50 percentile (Q2, median), 75percentile (Q3), max values of the columns. For categorical columns, determine distinct values and theirfrequency in each categorical column.> Hint: Pandas, data frame describe() function.<jupyter_code>df.info()
df.describe()
df.Species.value_counts().plot(kind='pie')
plt.show()<jupyter_output><empty_output><jupyter_text>---Step 2: Train Test Split---Split your data into Training and Test data set by randomly selecting; use 70% for training and 30 % fortesting. Generate descriptive statistics of all columns (input and output) of Training and Test datasets.Review the descriptive statistics of input output columns in Train, Test and original Full (before thesplitting operation) datasets and compare them to each other. Are they similar or not? Do you thinkTrain and Test dataset are representative of the Full datasets ? why ?> Hint: Scikit learn, data train_test_split(), stratified function.<jupyter_code>df.Species.value_counts(df.Species.unique()[0]) 
# Splitting train and test data
# .7 and .3 
import numpy as np 
msk = np.random.rand(len(df)) < 0.7
train_df = df[msk]
test_df = df[~msk]
train_df.describe()
test_df.describe()
df.describe()<jupyter_output><empty_output><jupyter_text>>  They are pretty similar to full data when I print the descriptive stats for test and train data. --- Step 3: Analysis of the Output Column---Analyze the output columns in Train and Test dataset. If the output column is numerical then calculatethe IQR (inter quartile range, Q3-Q1) and Range (difference between max and min value). If your outputcolumn is categorical then determine if the column is nominal or ordinal, why?. Is there a classimbalance problem? (check if there is big difference between the number of distinct values in yourcategorical output column)<jupyter_code>print(train_df["Species"].value_counts(train_df["Species"].unique()[0]))
print(len(train_df))
train_df.head()
print(test_df["Species"].value_counts(test_df["Species"].unique()[0]))
print(len(test_df))
test_df.head()<jupyter_output>1    20
2    19
0    13
Name: Species, dtype: int64
52<jupyter_text>> My target/label column is nominal categorical data. This data will  be used for multi-class classification. When I am splitting the test and train data, I was careful to get the similar ratio of the labels for each...--- Step 4: Scale Training and Test dataset---Using one of the scaling method (max, min-max, standard or robust), create a scaler object and scale thenumerical input columns of the Training dataset. Using the same scaler object, scale the numerical inputcolumns of the Test set. Generate the descriptive statistics of the scaled input columns of Training andTest set.If some of the input columns are categorical then convert them to binary columns using one-hotencoder()function (scikit learn) or dummy() function (Pandas data frame).> Hint:- http://scikit-learn.org/stable/modules/preprocessing.htmlpreprocessing<jupyter_code># I am going to apply min-max scaling for my data.
from sklearn import preprocessing
# Fitting the minmax scaled version for training data
minmax_scale = preprocessing.MinMaxScaler().fit(train_df.iloc[:, :4])
# Now actually scale train and test data
train_df.iloc[:, :4] = minmax_scale.transform(train_df.iloc[:, :4])
test_df.iloc[:, :4] = minmax_scale.transform(test_df.iloc[:, :4])
train_df.describe()
test_df.describe()
# Input and Output
inp_train = train_df.iloc[:, :4] 
out_train = train_df["Species"]
inp_test = test_df.iloc[:, :4] 
out_test = test_df["Species"]<jupyter_output><empty_output><jupyter_text>---Step 5: Build Predictive Model---Using one of the methods (Gradient Boosting Machines, Random Forest) build your ensemble predictivemodel using the scaled input columns of Training set. To find the optimum values for the modelparameters, use Grid Search with k-fold cross-validation in building your model. Grid Search is one ofthe method used to perform Hyper Parameter optimization to generate more accurate (bettergeneralized) models.> Hint:- http://scikit-learn.org/stable/supervised_learning.htmlsupervised-learning- http://scikit-learn.org/stable/modules/cross_validation.html- http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html- http://scikit-learn.org/stable/modules/grid_search.html- http://scikit-learn.org/stable/modules/ensemble.htmlrandom-forests- http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html- http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html<jupyter_code>from sklearn.cross_validation import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

# Building a RandomForest with 10 estimators
clf = RandomForestClassifier(n_estimators=10)

# clf = clf.fit(inp_train, out_train)
# 10-fold Cross validation
print("Average of 10 cross validation: ", np.mean(cross_val_score(clf, inp_train, out_train, cv=10)))
param_grid = {
                 'n_estimators': [5, 10, 15, 20],
                 'max_depth': [2, 5, 7, 9]
             }
grid_clf = GridSearchCV(clf, param_grid, cv=10)
grid_clf.fit(inp_train, out_train)
print(grid_clf)
print(grid_clf.best_estimator_)
print(grid_clf.best_params_)
print(grid_clf.best_score_)
# Using optimized parameterss to train my data
# Optimized parameters:
clf = RandomForestClassifier(max_depth=5, n_estimators=5, random_state=None)
clf.fit(inp_train, out_train)
print("Average of 10 cross validation of optimized estimetor: ",
      np.mean(cross_val_score(clf, inp_train, out_train, cv=5)))<jupyter_output>Average of 10 cross validation of optimized estimetor:  0.948370927318<jupyter_text>---Step 6: Model Predictions on Training Dataset---Apply your model to input (scaled) columns of Training dataset to obtain the predicted output forTraining dataset. If your model is regression then plot actual output versus predicted output column ofTraining dataset. If your model is classification then generate confusion matrix on actual and predictedcolumns of Training dataset.> Hint: Matplotlip, Seaborn, Bokeh scatter(), plot() functions- http://scikit-learn.org/0.15/auto_examples/plot_confusion_matrix.html- http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html<jupyter_code># importing libraries for plotting
# Importing library for confusion matrix
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns
import itertools
sns.set(style='darkgrid')
# train prediction for train data
out_train_pred = clf.predict(inp_train)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_train, out_train_pred)
print(cm)

sns.heatmap(cm, center=True)
plt.title('Confusion matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()<jupyter_output>[[37  0  0]
 [ 0 28  2]
 [ 0  2 29]]<jupyter_text>---Step 7: Model Predictions on Test Dataset---Apply your model to input (scaled) columns of Test dataset to obtain the predicted output for Testdataset. If your model is regression then plot actual output versus predicted output column of Testdataset. If your model is classification then generate confusion matrix on actual and predicted columnsof Test dataset.> Hint: Matplotlip, Seaborn, Bokeh scatter(), plot() functions- http://scikit-learn.org/0.15/auto_examples/plot_confusion_matrix.html- http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html<jupyter_code># test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)

sns.heatmap(cm, center=True)
plt.title('Confusion matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()<jupyter_output>[[13  0  0]
 [ 0 18  2]
 [ 0  2 17]]<jupyter_text>---Step 8: Model Performance---Using one of the error (evaluation) metrics (classification or regression), calculate the performance ofthe model on Training set and Test set. Compare the performance of the model on Training and Test set.Which one (Training or Testing performance) is better, is there an overfitting case, why ?.Would you deploy (Productionize) this model for using in your business system? why ?**Classification Metrics: Accuracy, Precision, Recall, F-score, Recall, AUC, ROC etcRegression Metrics: RMSE, MSE, MAE, R2 etc**> Hint:- http://scikit-learn.org/stable/model_selection.htmlmodel-selection- http://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-report<jupyter_code># I would like to use ROC
# Area under ROC Curve (or AUC for short) is 
#  a performance metric for binary classification problems.
from sklearn.metrics import roc_curve
# ROC curve for train data
fpr,tpr,thresholds = roc_curve(out_train, out_train_pred,pos_label=2)
# plot the curve
plt.plot(fpr, tpr, label="Train Data")
# ROC curve for test data
fpr, tpr, thresholds = roc_curve(out_test, out_test_pred, pos_label=2)
# Plotting the curves
plt.plot(fpr, tpr, label="Test Data")
plt.xlim([-0.05,1.05])
plt.ylim([-0.05,1.05])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
plt.show()<jupyter_output><empty_output><jupyter_text>> Train data is performing slightly better than the test data however, I believe it is overfitting, as you can see in the Train data ROC curve, it started very high and hit 1.0 sooner.> This model is not yet deployable because I did not check other models and tweaking the parameters.--- Step 9: Update the Model---Go back to Step5, and choose random values (use default values) of the model parameters and re-trainthe model. Repeat Steps: 6, 7 and 8. Using the same error metric, generate the accuracy of the model onTraining and Test dataset. Did you get a better or worse performance on Training or Test set?Explain why the new model performs better or worse than the former model. What doeshyperparameter optimization (grid search) on model building enable?<jupyter_code># Using default values for RandomForeset Classifier
# Building a RandomForest 
clf = RandomForestClassifier()

clf = clf.fit(inp_train, out_train)
# train prediction for train data
out_train_pred = clf.predict(inp_train)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_train, out_train_pred)
print(cm)

# test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)

# Model trained with default values 
# ROC curve for train data
fpr,tpr,thresholds = roc_curve(out_train, out_train_pred,pos_label=2)
# plot the curve
plt.plot(fpr, tpr, label="Train Data")
# ROC curve for test data
fpr, tpr, thresholds = roc_curve(out_test, out_test_pred, pos_label=2)
# Plotting the curves
plt.plot(fpr, tpr, label="Test Data")
plt.xlim([-0.05,1.05])
plt.ylim([-0.05,1.05])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
plt.show()<jupyter_output><empty_output><jupyter_text>As you can see from the graph train data is clearly overfitted, which means random forest with default parameters did not work for this dataset. Test data is still learning well but model is not really working with default parameters.--- Step 10: Change the Error Metric---Choose another error metric other than you used in Step 8 and evaluate the performance of the model(optimized) on Training and Test dataset by generating the accuracy of the model based on the newmetric. Compare the results and explain which error metric is better for your modeling and why?<jupyter_code># Let's check f1 score on our classification
from sklearn.metrics import f1_score
print("f1_score: ", f1_score(out_test, out_test_pred, average=None))<jupyter_output>f1_score:  [ 1.          0.92682927  0.91891892]
{'fraction_non_alphanumeric': 0.09285277947464875, 'fraction_numerical': 0.014890042761148442, 'mean_word_length': 5.624683864441073, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '124413', 'license': 'MIT', 'n_tokens_mistral': 4115, 'n_tokens_neox': 3870, 'n_words': 1407, 'path': 'Final_Project/theAwesome_EnsModel.ipynb', 'repo_name': 'eneskemalergin/Data_Mining_Spring2017', 'size': '13096'}