<jupyter_start><jupyter_text>The Pipeline1. Question & required data2. Acquire the data3. Data Analysis4. Prepare the data for the Deep Learning Model5. Building & training the model6. Testing the model7. Interpreting the model8. Improving the model  Problem introduction: predicting whether a bank customer will leave the bank Question & required dataThe dataset consists of a bunch of features describing the customers of a bank. The question that is posed is that, given these features could we detect some patterns that allow us to predict if a customer is about to leave a bank?It would then be possible to direct marketing efforts to the customers if it would be beneficial to retain them. ML problem characteristics1. Task: **classification**2. Attribute Types: **Categorical, continuous and binary**3. **14** attributes4. **10k** instances5. ML model: **Deep neural net**6. Deep learning library: **Keras (Tensorflow)**7. Hyperparameter tuning: **Talos** The feature vector1. *RowNumber* 2. *CustomerId:* unique id to identify a customer3. *Surname*4. *CreditScore:* a creditscore compiled by the bank for each customer5. *Geography*6. *Gender*7. *Age*8. *Tenure:* how long has the customer been with the bank9. *Balance*10. *NumOfProducts:* how many products of the bank (credit/debit cards etc.) does the customer have11. *HasCrCard:* does the customer have a credit card12. *IsActiveMember:* is the customer an active member of the bank13. *EstimatedSalary:* the customer's estimated salary<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from collections import Counter

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dropout, Dense
import talos

sns.set()
tf.__version__
data = pd.read_csv('dataset/exit_prediction.csv')
print('Our dataset holds {} records and {} attributes'.format(*data.shape))
data.head()<jupyter_output>Our dataset holds 10000 records and 14 attributes<jupyter_text>*RowNumber*, *CustomerId* & *Surname* won't actually be useful for the deep learning model that we will build in the end so we can just go ahead and drop them.<jupyter_code>data.drop(labels=['RowNumber', 'CustomerId', 'Surname'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Data Analysis1. [Tidy](https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html) Data: by default2. **Acquaintance with the data**3. Outliers4. Missing Data5. Class-Imbalance problem6. Feature correlation  Acquaintance with the dataLets visualize a subset of the feature vector for this purpose<jupyter_code>plt.figure(figsize=(15,13))
selected_features = data[['CreditScore', 'Geography', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 
                                                                      'EstimatedSalary', 'HasCrCard', 'Exited']]

for i in range(1, selected_features.shape[1]+1):
    plt.subplot(3,3,i)
    figure=plt.gca()
    figure.set_title(selected_features.columns[i-1])
    
    plt.hist(selected_features.iloc[:, i-1])
    if selected_features.columns[i-1] in ['EstimatedSalary', 'HasCrCard', 'Exited']:
        plt.xlabel('Measure')
    if selected_features.columns[i-1] in ['CreditScore', 'Tenure', 'EstimatedSalary']:
        plt.ylabel('Count')
    
plt.tight_layout(pad=2)<jupyter_output><empty_output><jupyter_text>OutliersIt appears that we have no outliers. This observation is also confirmed by the above plot<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Missing DataOur dataset seems to have no missing values. We cannot call the values of the *Balance = 0* as missing values since this can be a value that the balance can indeed take.For an approach of taking care of missing values see the other project on Autism Predictor<jupyter_code>print('Does our dataset contain missing values?', data.isnull().any().any())<jupyter_output>Does our dataset contain missing values? False<jupyter_text>Class-Imbalance problemOur data seems indeed to have imbalanced classes. Obviously, most of the people did not leave the bank.<jupyter_code>print(data['Exited'].value_counts()/data.shape[0])

sns.countplot(x='Exited', data=data, palette='hls'); plt.show()<jupyter_output>0    0.7963
1    0.2037
Name: Exited, dtype: float64<jupyter_text>**Are the classes well separated?**If our classes are well separated, then the class-imbalance problem is not really a problem. We can simply visualize a couple of the attributes for this purpose.<jupyter_code>fig, ((ax1, ax2, ax3)) = plt.subplots(nrows=1, ncols=3, figsize=(15,5))

att1, att2, att3 = np.random.choice(data.columns.delete([13]), 3, replace=False)

att1_data = list(data[att1].groupby(data['Exited']))
ax1.plot(att1_data[0][1], 'g^', label=att1_data[0][0])
ax1.plot(att1_data[1][1], 'ro', label=att1_data[1][0])
ax1.set_xlabel('Count'); ax1.set_ylabel('Measure'); ax1.set_title(att1_data[0][1].name)
ax1.legend()

att2_data = list(data[att2].groupby(data['Exited']))
ax2.plot(att2_data[0][1], 'g^', label=att2_data[0][0])
ax2.plot(att2_data[1][1], 'ro', label=att2_data[1][0])
ax2.set_xlabel('Count'); ax2.set_ylabel(''); ax2.set_title(att2_data[0][1].name)
ax2.legend()

att3_data = list(data[att3].groupby(data['Exited']))
ax3.plot(att3_data[1][1], 'ro', label=att3_data[1][0])
ax3.plot(att3_data[0][1], 'g^', label=att3_data[0][0])
ax3.set_xlabel('Count'); ax3.set_ylabel(''); ax3.set_title(att3_data[0][1].name)
ax3.legend(); plt.show()<jupyter_output><empty_output><jupyter_text>It seems that our classes are not well separated. There are techniques to fight the class-imbalance problem but in this case we are just going to continue like this.  One-Hot EncodingConvert categorical data to numerical. We have two categorical features in our case: *geography* & *gender*<jupyter_code>data = pd.get_dummies(data)
data.head(2)<jupyter_output><empty_output><jupyter_text>Feature correlationWe want the features to be independent of each-other. Most ML algorithms work on this assumption. Correlation of features with the class labelThis will also give us an impression into which variable is the more important for predicting the class variable.None of the attributes seems to be significantly correlated with the class.<jupyter_code>data.drop('Exited', axis=1).corrwith(data['Exited']).plot.bar(figsize=(12,5), 
                                            title='Correlation with the class variable', rot=45, fontsize=11)
plt.show()<jupyter_output><empty_output><jupyter_text>We can also check the influence of the attributes on the class by aggregating the mean values after grouping by the class label. The below result confirms the correlation plot above.<jupyter_code>class_group = data.groupby('Exited')
print(class_group.agg(np.mean))<jupyter_output>CreditScore        Age    Tenure       Balance  NumOfProducts  \
Exited                                                                  
0        651.853196  37.408389  5.033279  72745.296779       1.544267   
1        645.351497  44.837997  4.932744  91108.539337       1.475209   

        HasCrCard  IsActiveMember  EstimatedSalary  Geography_France  \
Exited                                                                 
0        0.707146        0.554565     99738.391772          0.527942   
1        0.699067        0.360825    101465.677531          0.397644   

        Geography_Germany  Geography_Spain  Gender_Female  Gender_Male  
Exited                                                                  
0                0.212859         0.259199       0.427477     0.572523  
1                0.399607         0.202749       0.559156     0.440844<jupyter_text>Correlation of features with each-other<jupyter_code>sns.set(style='white', font_scale=1.5)
corr_matrix = data.corr()

# create a mask for the upper triangle so that we can ignore it when later building the heatmap
mask = np.zeros_like(corr_matrix, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# color map for the different values of the correlation matrix
cmap = sns.diverging_palette(220, 10, as_cmap=True)

plt.figure(figsize=(11,8))
plt.title('Correlation matrix for Features')
sns.heatmap(corr_matrix, square=True, mask=mask, cmap=cmap, center=0, linewidths=1.0, cbar_kws={'shrink':0.8})
plt.show()<jupyter_output><empty_output><jupyter_text>We can see some correlations, like for instance the negative correlation between the number of products and the balance but they are less than 0.3 and so do not pose any problems for the algorithm. We may further proceed like this.  Prepare the data for the Neural Net Model1. Class & Features separation2. Train/Test split3. Scaling/Standardization if needed  Class & Feature separation<jupyter_code>labels = data.pop('Exited')
features = data
print('Shape of features', features.shape)
print('Shape of labels', labels.shape)
features.head(2)<jupyter_output>Shape of features (10000, 13)
Shape of labels (10000,)<jupyter_text>Train/Test split<jupyter_code>train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.3, 
                                                                            random_state=1, stratify=labels)
print('Shape of training data', train_features.shape)
print('Shape of testing data', test_features.shape)<jupyter_output>Shape of training data (7000, 13)
Shape of testing data (3000, 13)<jupyter_text>Feature Scaling<jupyter_code>scaler = StandardScaler()
to_scale = ['CreditScore', 'Age', 'Balance', 'EstimatedSalary']

train_features_sc = pd.DataFrame(scaler.fit_transform(train_features[to_scale]))
train_features_sc.columns = train_features[to_scale].columns
train_features_sc.index = train_features[to_scale].index.values
train_features[to_scale] = train_features_sc

test_features_sc = pd.DataFrame(scaler.fit_transform(test_features[to_scale]))
test_features_sc.columns = test_features[to_scale].columns
test_features_sc.index = test_features[to_scale].index.values
test_features[to_scale] = test_features_sc<jupyter_output><empty_output><jupyter_text>Build & train the Neural Net modelWe shall first build a basic Neural Net model using Keras. On the next chapter we shall see how to tune its parameters using talos in a bid to improve the model's performance. Build the Deep Neural NetWe can instantiate different aspects of the neural net either by passing a string identifier (and in this case the default parameters for the optimizer can be used) or by building an object from the class. The latter should be the used way as it gives us the possibility to tweak stuff.[Docs](https://keras.io/api/layers/core_layers/dense/) on the dense layers[Docs](https://keras.io/api/models/model_training_apis/) for the compile method[Docs](https://keras.io/api/optimizers/) on optimizers[Docs](https://keras.io/api/losses/) on the loss function- *from_logits=True*: will significantly boost testing accuracy at a cost of training- *label_smoothing=1*: can be played with in the range of [0,1]- *reduction*: different options include, 'sum', keras.losses.Reduction.NONE[Docs](https://keras.io/api/models/model_training_apis/) on the fit method1. *class_weight*: can be useful to tell the model to "pay more attention" to samples from an under-represented class<jupyter_code>ann = Sequential()
ann.add(Dense(units=100, activation='relu', use_bias=True))
ann.add(Dense(units=64, activation='relu', use_bias=True))
ann.add(Dense(units=32, activation='relu', use_bias=True))
ann.add(Dense(units=32, activation='relu', use_bias=True))

# Use sigmoid in the final layer since we have a binary classification problem
ann.add(Dense(units=1, activation='sigmoid', use_bias=False))

optimizer = keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)

# binary cross entropy as loss since we have a binary classification problem
loss = keras.losses.BinaryCrossentropy(name='binary_crossentropy', from_logits=False,
                                                                            label_smoothing=0.05, reduction='auto')

ann.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
def get_class_weights(labels):
    counter = Counter(labels)
    majority = max(counter.values())
    return {cls: float(majority/count) for cls, count in counter.items()}

get_class_weights(labels)
c_weight = {0:1, 1:4}
ann.fit(x=train_features.values, y=train_labels.values, epochs=100, verbose=2, shuffle=True, batch_size=70,
                 use_multiprocessing=True, )<jupyter_output>Epoch 1/100
 - 1s - loss: 0.5168 - accuracy: 0.7741
Epoch 2/100
 - 0s - loss: 0.4520 - accuracy: 0.8164
Epoch 3/100
 - 0s - loss: 0.4298 - accuracy: 0.8299
Epoch 4/100
 - 0s - loss: 0.4040 - accuracy: 0.8513
Epoch 5/100
 - 0s - loss: 0.3889 - accuracy: 0.8579
Epoch 6/100
 - 0s - loss: 0.3819 - accuracy: 0.8610
Epoch 7/100
 - 0s - loss: 0.3766 - accuracy: 0.8643
Epoch 8/100
 - 0s - loss: 0.3750 - accuracy: 0.8671
Epoch 9/100
 - 0s - loss: 0.3711 - accuracy: 0.8664
Epoch 10/100
 - 0s - loss: 0.3688 - accuracy: 0.8680
Epoch 11/100
 - 0s - loss: 0.3676 - accuracy: 0.8661
Epoch 12/100
 - 0s - loss: 0.3657 - accuracy: 0.8681
Epoch 13/100
 - 0s - loss: 0.3623 - accuracy: 0.8730
Epoch 14/100
 - 0s - loss: 0.3612 - accuracy: 0.8734
Epoch 15/100
 - 0s - loss: 0.3586 - accuracy: 0.8714
Epoch 16/100
 - 0s - loss: 0.3589 - accuracy: 0.8724
Epoch 17/100
 - 0s - loss: 0.3562 - accuracy: 0.8726
Epoch 18/100
 - 0s - loss: 0.3546 - accuracy: 0.8736
Epoch 19/100
 - 0s - loss: 0.3549 - accuracy: 0.8721
Ep[...]<jupyter_text>Test the Model<jupyter_code>print('Shape of testing dataset', test_features.shape)
predictions = ((ann.predict(test_features, batch_size=70)) > 0.5)
print('Our models accuracy on test data: ', round(100*accuracy_score(test_labels, predictions), 2))<jupyter_output>Shape of testing dataset (3000, 13)
Our models accuracy on test data:  83.6
{'fraction_non_alphanumeric': 0.1070332024277044, 'fraction_numerical': 0.05583720099964298, 'mean_word_length': 4.269375470278405, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 58, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '39231', 'license': 'MIT', 'n_tokens_mistral': 4872, 'n_tokens_neox': 4385, 'n_words': 1329, 'path': 'Customer Exit Predictor/customer_exit_predictor.ipynb', 'repo_name': 'rlleshi/MachineLearning', 'size': '14005'}