<jupyter_start><jupyter_text>FrameThe client bank *XYZ* is running a direct marketing campaign. It wants to identify customers who would potentially be buying their new term deposit plan. AcquireData is obtained from UCI Machine Learning repository. http://mlr.cs.umass.edu/ml/datasets/Bank+MarketingData from direct marketing campaign (phone calls) of a Portuguese Bank is provided.  Attribute Information: bank client data:1. age (numeric)2. job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')3. marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)4. education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')5. default: has credit in default? (categorical: 'no','yes','unknown')6. housing: has housing loan? (categorical: 'no','yes','unknown')7. loan: has personal loan? (categorical: 'no','yes','unknown') related with the last contact of the current campaign:8. contact: contact communication type (categorical: 'cellular','telephone') 9. month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')10. day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')11. duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model. other attributes:12. campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)13. pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)14. previous: number of contacts performed before this campaign and for this client (numeric)15. poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success') social and economic context attributes16. emp.var.rate: employment variation rate - quarterly indicator (numeric)17. cons.price.idx: consumer price index - monthly indicator (numeric) 18. cons.conf.idx: consumer confidence index - monthly indicator (numeric) 19. euribor3m: euribor 3 month rate - daily indicator (numeric)20. nr.employed: number of employees - quarterly indicator (numeric) Output variable (desired target):y - has the client subscribed a term deposit? (binary: 'yes','no')The given data is randomly divided into train and test for the purpose of this workshop. Build the model for train and use it to predict on test.  Explore<jupyter_code>#Import the necessary libraries
import numpy as np
import pandas as pd
#Read the train and test data
train = pd.read_csv("../data/train.csv")
test = pd.read_csv("../data/test.csv")<jupyter_output><empty_output><jupyter_text>**Exercise 1**print the number of rows and columns of train and test **Exercise 2**Print the first 10 rows of train **Exercise 3**Print the column types of train and test. Are they the same in both train and test?<jupyter_code>#train
#test
#Are they the same?
#Combine train and test
frames = [train, test]
input = pd.concat(frames)
#Print first 10 records of input<jupyter_output><empty_output><jupyter_text>**Exercise 4**Find if any column has missing valueThere is a `pd.isnull` function. How to use that?<jupyter_code>#Replace deposit with a numeric column
#First, set all labels to be 0
input.at[:, "depositLabel"] = 0
#Now, set depositLabel to 1 whenever deposit is yes
input.at[input.deposit=="yes", "depositLabel"] = 1<jupyter_output><empty_output><jupyter_text>**Exercise 5**Find % of customers in the input dataset who have purchased the term deposit<jupyter_code>#Create the labels 
labels = 
labels
#Drop the deposit column 
input.drop(["deposit", "depositLabel"], axis=1)<jupyter_output><empty_output><jupyter_text>**Exercise 6**Did it drop? If not, what has to be done? **Exercise 7**Print columnn names of input<jupyter_code>#Get list of columns that are continuous/integer
continuous_variables = input.dtypes[input.dtypes != "object"].index
continuous_variables
#Get list of columns that are categorical
categorical_variables = input.dtypes[input.dtypes=="object"].index
categorical_variables<jupyter_output><empty_output><jupyter_text>**Exercise 8**Create `inputInteger` and `inputCategorical` - two datasets - one having integer variables and another having categorical variables<jupyter_code>inputInteger = 
#print inputInteger
inputInteger.head()
inputCategorical = 
#print inputCategorical
inputCategorical.head()
#Convert categorical variables into Labels using labelEncoder

inputCategorical = np.array(inputCategorical)<jupyter_output><empty_output><jupyter_text>**Exercise 9**Find length of `categorical_variables`<jupyter_code>#Load the preprocessing module
from sklearn import preprocessing
for i in range(len(categorical_variables)):
    lbl = preprocessing.LabelEncoder()
    lbl.fit(list(inputCategorical[:,i]))
    inputCategorical[:, i] = lbl.transform(inputCategorical[:, i])
#print inputCategorical<jupyter_output><empty_output><jupyter_text>**Exercise 10**Convert `inputInteger` to `numpy` array<jupyter_code>inputInteger = 
inputInteger<jupyter_output><empty_output><jupyter_text>**Exercise 11**Now, create the `inputUpdated` array that has both `inputInteger` and `inputCategorical` concatenated*Hint* Check function called `vstack` and `hstack`<jupyter_code>inputUpdated.shape<jupyter_output><empty_output><jupyter_text>Train the model Model 1: Decision Tree<jupyter_code>from sklearn import tree
from sklearn.externals.six import StringIO
import pydot
bankModelDT = tree.DecisionTreeClassifier(max_depth=2)
bankModelDT.fit(inputUpdated[:train.shape[0],:], labels[:train.shape[0]])
dot_data = StringIO() 
tree.export_graphviz(bankModelDT, out_file=dot_data) 
graph = pydot.graph_from_dot_data(dot_data.getvalue()) 
graph.write_pdf("bankDT.pdf") 
#Check the pdf<jupyter_output><empty_output><jupyter_text>**Exercise 12**Now, change the max_depth = 6 and check the results.Then, change the max_depth= None and check the results<jupyter_code># Prediction
prediction_DT = bankModelDT.predict(inputUpdated[train.shape[0]:,:])
#Compute the error metrics
import sklearn.metrics
sklearn.metrics.auc(labels[train.shape[0]:], prediction_DT)
#What does that tell?
#What's the error AUC for the other Decision Tree Models<jupyter_output><empty_output><jupyter_text>**Exercise 13**Instead of predicting classes directly, predict the probability and check the `auc`<jupyter_code>sklearn.metrics.auc(labels[train.shape[0]:], prediction_DT[:,0])<jupyter_output><empty_output><jupyter_text>Accuracy Metrics* AUC* ROC* Misclassification Rate* Confusion Matrix* Precision & Recall Confusion Matrix Calculate True Positive Rate    TPR = TP / (TP+FN) Calculate False Positive Rate    FPR = FP / (FP+TN)     Precision Recall<jupyter_code>#Precision and Recall
sklearn.metrics.precision_score(labels[train.shape[0]:], prediction_DT)
sklearn.metrics.recall_score(labels[train.shape[0]:], prediction_DT)<jupyter_output><empty_output><jupyter_text>Ensemble Trees*src*: http://www.slideshare.net/hustwj/scaling-up-machine-learning-the-tutorial-kdd-2011-part-iia-tree-ensembles  Random Forest *src*: http://www.slideshare.net/0xdata/jan-vitek-distributedrandomforest522013<jupyter_code>from sklearn.ensemble import RandomForestClassifier
bankModelRF = RandomForestClassifier(n_jobs=-1, oob_score=True)
bankModelRF.fit(inputUpdated[:train.shape[0],:], labels[:train.shape[0]])
bankModelRF.oob_score_<jupyter_output><empty_output><jupyter_text>**Exercise 14**Do the following1. Predict on test2. Find accuracy metrics: AUC, Precision, Recall 3. How does it compare against Decision Tree  Gradient Boosting Machines *src*: http://www.slideshare.net/hustwj/scaling-up-machine-learning-the-tutorial-kdd-2011-part-iia-tree-ensembles<jupyter_code>import xgboost as xgb
params = {}
params["min_child_weight"] = 3
params["subsample"] = 0.7
params["colsample_bytree"] = 0.7
params["scale_pos_weight"] = 1
params["silent"] = 0
params["max_depth"] = 4
params["nthread"] = 6
params["gamma"] = 1
params["objective"] = "binary:logistic"
params["eta"] = 0.005
params["base_score"] = 0.1
params["eval_metric"] = "auc"
params["seed"] = 123
plst = list(params.items())
num_rounds = 120
xgtrain_pv = xgb.DMatrix(inputUpdated[:train.shape[0],:], label=labels[:train.shape[0]])
watchlist = [(xgtrain_pv, 'train')]
bankModelXGB = xgb.train(plst, xgtrain_pv, num_rounds)
prediction_XGB = bankModelXGB.predict(xgb.DMatrix(inputUpdated[train.shape[0]:,:]))
sklearn.metrics.auc(labels[train.shape[0]:], prediction_XGB)<jupyter_output><empty_output><jupyter_text>Another way of encoding  One Hot EncodingWhiteboard !<jupyter_code>inputOneHot = pd.get_dummies(input)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11568263669104005, 'fraction_numerical': 0.013750954927425516, 'mean_word_length': 7.226211849192101, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '314452', 'license': 'MIT', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2776, 'n_words': 817, 'path': 'RF_GBM/notebook/Bank Marketing.ipynb', 'repo_name': 'jithurjacob/machine-learning', 'size': '9163'}