<jupyter_start><jupyter_text>Project Title : Predicting the effectiveness of bank marketing campaigns    Problem Description  The data is related with direct marketing campaigns (phone calls) of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed. The classification goal is to predict if the client will subscribe a term deposit (variable y).   Data Description  Input variables:   Bank Client data: *  age (numeric)*  job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')*  marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)*  education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')*  default: has credit in default? (categorical: 'no','yes','unknown')*  housing: has housing loan? (categorical: 'no','yes','unknown')*  loan: has personal loan? (categorical: 'no','yes','unknown')  Related with the last contact of the current campaign:*  contact: contact communication type (categorical: 'cellular','telephone')*  month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')*  day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')*  duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model. Other attributes: *  campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)*  pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)*  previous: number of contacts performed before this campaign and for this client (numeric)*  poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success') Output variable (desired target):*  y - has the client subscribed a term deposit? (binary: 'yes','no')<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from google.colab import drive
drive.mount('/content/drive')
dataset =pd.read_excel('/content/drive/MyDrive/Bank Marketing Effectiveness prediction/Bankfull.xlsx')
dataset.columns
dataset.head()
dataset.tail()
dataset.dtypes
dataset.describe()
dataset.shape
dataset.isna().sum()<jupyter_output><empty_output><jupyter_text>**NO NULL VALUES IN THE DATASET**<jupyter_code>dataset.info()
dataset=dataset.drop_duplicates()
dataset.shape<jupyter_output><empty_output><jupyter_text>**NO DUPLICATES IN THE DATASET**<jupyter_code>dataset.boxplot(column='age',by='y')<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>**THE MEDIAN, MAXIMUM AND MINIMUM DIFFERS SLIGHTLY**<jupyter_code>dataset.boxplot(column='balance',by='y')<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>**BOTH CLASS HAVE SIMILAR DISTRIBUTION**<jupyter_code>dataset.boxplot(column='day',by='y')<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>**THE MEDIAN, MAXIMUM AND MINIMUM DIFFERS SLIGHTLY**<jupyter_code>dataset.boxplot(column='duration',by='y')<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>**HIGH DURATION HAVE A BIG EFFECT OF THE DEPENDENT VARIABLE**<jupyter_code>dataset.boxplot(column='campaign',by='y')<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>**THE MEDIAN, MAXIMUM AND MINIMUM DIFFERS SLIGHTLY**<jupyter_code>dataset['job'].unique()
dataset['job'].value_counts()
dataset['marital'].unique()
dataset['marital'].value_counts()
dataset['education'].unique()
dataset['education'].value_counts()
dataset['default'].unique()
dataset['default'].value_counts()
dataset['housing'].unique()
dataset['housing'].value_counts()
dataset['loan'].unique()
dataset['loan'].value_counts()
dataset['contact'].unique()
dataset['contact'].value_counts()
dataset['month'].unique()
dataset['month'].value_counts()
dataset['poutcome'].unique()
dataset['poutcome'].value_counts()
dataset['y'].unique()
dataset['y'].value_counts()
plt.figure(figsize=(20, 6), dpi=80)
sns.countplot(data=dataset, x='job', hue='y')<jupyter_output><empty_output><jupyter_text>**BLUE COLLAR HAVE HIGHER DISTRIBUTION IN NO CLASS****MANAGEMENT HAVE HIGHER DISTRIBUTION IN YES CLASS**<jupyter_code>sns.countplot(data=dataset, x='marital', hue='y')<jupyter_output><empty_output><jupyter_text>**MARRIED HAVE HIGH DISTRIBUTION IN BOTH CLASS**<jupyter_code>sns.countplot(data=dataset, x='education', hue='y')<jupyter_output><empty_output><jupyter_text>**SECONDARY HAVE HIGH DISTRIBUTION IN BOTH CLASS**<jupyter_code>sns.countplot(data=dataset, x='default', hue='y')<jupyter_output><empty_output><jupyter_text>**NO(NOT HAVING CREDIT CARD) HAVE HIGH DISTRIBUTION IN BOTH CLASS**<jupyter_code>sns.countplot(data=dataset, x='housing', hue='y')<jupyter_output><empty_output><jupyter_text>**YES(TAKEN HOUSING LOAN) HAVE HIGHER DISTRIBUTION IN YES CLASS****NO(NOT TAKEN HOUSING LOAN) HAVE HIGHER DISTRIBUTION IN NO CLASS**<jupyter_code>sns.countplot(data=dataset, x='loan', hue='y')<jupyter_output><empty_output><jupyter_text>**NO(NOT TAKEN LOAN) HAVE HIGHER DISTRIBUTION IN BOTH CLASS**<jupyter_code>sns.countplot(data=dataset, x='contact', hue='y')<jupyter_output><empty_output><jupyter_text>**CELLULAR HAVE HIGHER DISTRIBUTION IN BOTH CLASS**<jupyter_code>sns.countplot(data=dataset, x='month', hue='y')<jupyter_output><empty_output><jupyter_text>**MAY MONTH HAVE HIGHER DISTRIBUTION IN BOTH CLASS**<jupyter_code>from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()
dataset['default']= label_encoder.fit_transform(dataset['default'])
dataset['housing']= label_encoder.fit_transform(dataset['housing'])
dataset['loan']= label_encoder.fit_transform(dataset['loan'])
dataset['y']= label_encoder.fit_transform(dataset['y'])
dataset['default'].value_counts()
dataset['housing'].value_counts()
dataset['loan'].value_counts()
dataset['y'].value_counts()
dataset.head()
plt.figure(figsize=(12,12))
sns.heatmap(dataset.corr(), cmap='BuPu', annot=True)<jupyter_output><empty_output><jupyter_text>**NO PARAMETER HAVE HIGH CORRELATION**<jupyter_code>onehotencoder_list=['job','marital','education','contact','month','poutcome'] 
for j in onehotencoder_list:
  onehotencoder = preprocessing.OneHotEncoder()
  X = onehotencoder.fit_transform(dataset[j].values.reshape(-1,1)).toarray()
  datasetOneHot = pd.DataFrame(X, columns = [str(j)+'_'+str(int(i)) for i in range(X.shape[1])]) 
  dataset = pd.concat([dataset, datasetOneHot], axis=1)
  dataset.drop([j],axis=1,inplace=True)
dataset.head()
dataset.info()
X = dataset.drop(['y','duration'],axis=1)
 Y = dataset['y']
X
Y
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, random_state=50,stratify=Y)
from sklearn.preprocessing import StandardScaler
scalerTrain = preprocessing.StandardScaler().fit(X_train)
scalerTest = preprocessing.StandardScaler().fit(X_test)
X_train_Scaled=scalerTrain.transform(X_train)
X_test_Scaled=scalerTest.transform(X_test)
Train_model_score = {}
Test_model_score = {}
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
logisticmodel = LogisticRegression(class_weight='balanced')
logisticmodel.fit(X_train_Scaled, Y_train)
logistic_pred_train = logisticmodel.predict_proba(X_train_Scaled)
logistic_pred_test = logisticmodel.predict_proba(X_test_Scaled)
logistic_auc_score_train = roc_auc_score(Y_train, logistic_pred_train[:,1])
logistic_auc_score_test = roc_auc_score(Y_test, logistic_pred_test[:,1])
print("Logistic regression, AUC score for train",logistic_auc_score_train )
print("Logistic regression, AUC score for test",logistic_auc_score_test)
Train_model_score['Logistic regression'] = logistic_auc_score_train
Test_model_score['Logistic regression'] = logistic_auc_score_test
from sklearn.neighbors import KNeighborsClassifier
from sklearn.calibration import CalibratedClassifierCV
knn_neighbors = [2,4,6,8,11,13]
for h,i in enumerate(knn_neighbors):
    knn = KNeighborsClassifier(n_neighbors=i,n_jobs=-1)
    knn.fit(X_train_Scaled,Y_train)
    calibrated_knn = CalibratedClassifierCV(knn, method="sigmoid")
    calibrated_knn.fit(X_train_Scaled, Y_train)
    knn_pred_train = calibrated_knn.predict_proba(X_train_Scaled)
    knn_pred_test = calibrated_knn.predict_proba(X_test_Scaled)
    knn_auc_score_train = (roc_auc_score(Y_train,knn_pred_train[:,1]))
    knn_auc_score_test = (roc_auc_score(Y_test,knn_pred_test[:,1]))
    print ('AUC for number of neighbors = ',knn_neighbors[h],'for train',knn_auc_score_train,'and for test',knn_auc_score_test)
best_knn = KNeighborsClassifier(n_neighbors=13,n_jobs=-1)
best_knn.fit(X_train_Scaled,Y_train)
best_calibrated_knn = CalibratedClassifierCV(best_knn)
best_calibrated_knn.fit(X_train_Scaled, Y_train)
best_knn_pred_train = best_calibrated_knn.predict_proba(X_train_Scaled)
best_knn_pred_test = best_calibrated_knn.predict_proba(X_test_Scaled)
best_knn_auc_score_train = (roc_auc_score(Y_train,best_knn_pred_train[:,1]))
best_knn_auc_score_test = (roc_auc_score(Y_test,best_knn_pred_test[:,1]))
print ('The best KNN AUC for train',best_knn_auc_score_train,'and for test',best_knn_auc_score_test)
Train_model_score['KNN Classification'] = best_knn_auc_score_train
Test_model_score['KNN Classification'] = best_knn_auc_score_test
from sklearn.tree import DecisionTreeClassifier
decisiontree_depth = [1,4,5,6,7,8,10,15,20]
for h,i in enumerate(decisiontree_depth):
    decisiontree =DecisionTreeClassifier(max_depth=i,random_state=50)
    decisiontree.fit(X_train_Scaled,Y_train)
    calibrated_decisiontree = CalibratedClassifierCV(decisiontree)
    calibrated_decisiontree.fit(X_train_Scaled, Y_train)
    decisiontree_pred_train = calibrated_decisiontree.predict_proba(X_train_Scaled)
    decisiontree_pred_test = calibrated_decisiontree.predict_proba(X_test_Scaled)
    decisiontree_auc_score_train = (roc_auc_score(Y_train,decisiontree_pred_train[:,1]))
    decisiontree_auc_score_test = (roc_auc_score(Y_test,decisiontree_pred_test[:,1]))
    print ('AUC for depth = ',decisiontree_depth[h],'for train',decisiontree_auc_score_train,'and for test',decisiontree_auc_score_test)
best_decisiontree = DecisionTreeClassifier(max_depth=15,random_state=50)
best_decisiontree.fit(X_train_Scaled,Y_train)
best_calibrated_decisiontree = CalibratedClassifierCV(best_decisiontree)
best_calibrated_decisiontree.fit(X_train_Scaled, Y_train)
best_decisiontree_pred_train = best_calibrated_decisiontree.predict_proba(X_train_Scaled)
best_decisiontree_pred_test = best_calibrated_decisiontree.predict_proba(X_test_Scaled)
best_decisiontree_auc_score_train = (roc_auc_score(Y_train,best_decisiontree_pred_train[:,1]))
best_decisiontree_auc_score_test = (roc_auc_score(Y_test,best_decisiontree_pred_test[:,1]))
print ('The best decsiontree AUC for train',best_decisiontree_auc_score_train,'and for test',best_decisiontree_auc_score_test)
Train_model_score['DecisionTree Classification'] = best_decisiontree_auc_score_train
Test_model_score['DecisionTree Classification'] = best_decisiontree_auc_score_test
from sklearn.ensemble import RandomForestClassifier
randomforest_estimator=[2,3,4,5,6,7,8,9,10]
for h,i in enumerate(randomforest_estimator):
    randomforest = RandomForestClassifier(n_estimators=i,random_state=50,n_jobs=-1)
    randomforest.fit(X_train_Scaled,Y_train)
    calibrated_randomforest = CalibratedClassifierCV(randomforest)
    calibrated_randomforest.fit(X_train_Scaled, Y_train)
    randomforest_pred_train = calibrated_randomforest.predict_proba(X_train_Scaled)
    randomforest_pred_test = calibrated_randomforest.predict_proba(X_test_Scaled)
    randomforest_auc_score_train = (roc_auc_score(Y_train,randomforest_pred_train[:,1]))
    randomforest_auc_score_test = (roc_auc_score(Y_test,randomforest_pred_test[:,1]))
    print ('AUC for number of estimators = ',randomforest_estimator[h],'for train',randomforest_auc_score_train,'and for test',randomforest_auc_score_test)
best_randomforest = RandomForestClassifier(n_estimators=10,random_state=50,n_jobs=-1)
best_randomforest.fit(X_train_Scaled,Y_train)
best_calibrated_randomforest = CalibratedClassifierCV(best_randomforest)
best_calibrated_randomforest.fit(X_train_Scaled, Y_train)
best_randomforest_pred_train = best_calibrated_randomforest.predict_proba(X_train_Scaled)
best_randomforest_pred_test = best_calibrated_randomforest.predict_proba(X_test_Scaled)
best_randomforest_auc_score_train = (roc_auc_score(Y_train,best_randomforest_pred_train[:,1]))
best_randomforest_auc_score_test = (roc_auc_score(Y_test,best_randomforest_pred_test[:,1]))
print ('The best randomforest AUC for train',best_randomforest_auc_score_train,'and for test',best_randomforest_auc_score_test)
Train_model_score['RandomForest Classification'] = best_randomforest_auc_score_train
Test_model_score['RandomForest Classification'] = best_randomforest_auc_score_test
from xgboost import XGBClassifier
xgb_estimator=[10,50,100,500,1000,2000]
for h,i in enumerate(xgb_estimator):
    xgb=XGBClassifier(n_estimators=i)
    xgb.fit(X_train_Scaled,Y_train)
    calibrated_xgb = CalibratedClassifierCV(xgb, method="sigmoid")
    calibrated_xgb.fit(X_train_Scaled, Y_train)
    xgb_pred_train = calibrated_xgb.predict_proba(X_train_Scaled)
    xgb_pred_test = calibrated_xgb.predict_proba(X_test_Scaled)
    xgb_auc_score_train = (roc_auc_score(Y_train,xgb_pred_train[:,1]))
    xgb_auc_score_test = (roc_auc_score(Y_test,xgb_pred_test[:,1]))
    print ('AUC for number of estimators = ',xgb_estimator[h],'for train',xgb_auc_score_train,'and for test',xgb_auc_score_test)
from sklearn.model_selection import GridSearchCV
param = {'n_estimators' : [100,500,1000]}
best_xgb=XGBClassifier()
gridSearch_best_xgb=GridSearchCV(best_xgb,param,cv=5,n_jobs = -1)
best_calibrated_xgb=CalibratedClassifierCV(gridSearch_best_xgb)
best_calibrated_xgb.fit(X_test_Scaled,Y_test)
best_xgb_pred_train = best_calibrated_xgb.predict_proba(X_train_Scaled)
best_xgb_pred_test = best_calibrated_xgb.predict_proba(X_test_Scaled)
best_xgb_auc_score_train = (roc_auc_score(Y_train,best_xgb_pred_train[:,1]))
best_xgb_auc_score_test = (roc_auc_score(Y_test,best_xgb_pred_test[:,1]))
print ('The best xgb AUC for train',best_xgb_auc_score_train,'and for test',best_xgb_auc_score_test)
Train_model_score['XGB Classification'] = best_xgb_auc_score_train
Test_model_score['XGB Classification'] = best_xgb_auc_score_test
Train_model_score
Test_model_score
plt.figure(figsize=(18, 6), dpi=80)
plt.bar(*zip(*Train_model_score.items()))
plt.title('Train Score')
plt.ylabel("ROC_AUC score")
plt.xlabel('Model')
plt.show()
plt.figure(figsize=(18, 6), dpi=80)
plt.bar(*zip(*Test_model_score.items()))
plt.title('Test Score')
plt.ylabel("ROC_AUC score")
plt.xlabel('Model')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11455847255369929, 'fraction_numerical': 0.00890622271377845, 'mean_word_length': 9.412121212121212, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '576509', 'license': 'MIT', 'n_tokens_mistral': 6103, 'n_tokens_neox': 5785, 'n_words': 1105, 'path': 'Bank_Marketing_Effectiveness_Prediction_Capstone_Project.ipynb', 'repo_name': 'sanjeevm4788/Bank-Marketing-Effectiveness-Prediction', 'size': '17179'}