<jupyter_start><jupyter_text>Deep learning A-Z : Building an ANN This notebook is my response to the first homework of the course called *Deep Learning A-Z™: Hands-On Artificial Neural Networks* accessible here : https://www.udemy.com/deeplearning/In this notebook, we are going to build an ANN using keras by following instructions given on the course. This neurals network will predict, for a customer of a bank, if this customer is going to leave the bank or not. We are going to train our ann with a dataset containing data about approximately 10000 clients, which also includes a response column in which we can see whether the client stayed or not in the bank.  Imports<jupyter_code>import os
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier

# Ignore warnings
import warnings
warnings.filterwarnings('ignore')<jupyter_output>Using TensorFlow backend.<jupyter_text>1. Data preprocessing<jupyter_code>path_train = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath('__file__')))), 'ressources/Artificial_Neural_Networks/Churn_Modelling.csv')
dataset = pd.read_csv(path_train)
dataset.head()
dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
RowNumber          10000 non-null int64
CustomerId         10000 non-null int64
Surname            10000 non-null object
CreditScore        10000 non-null int64
Geography          10000 non-null object
Gender             10000 non-null object
Age                10000 non-null int64
Tenure             10000 non-null int64
Balance            10000 non-null float64
NumOfProducts      10000 non-null int64
HasCrCard          10000 non-null int64
IsActiveMember     10000 non-null int64
EstimatedSalary    10000 non-null float64
Exited             10000 non-null int64
dtypes: float64(2), int64(9), object(3)
memory usage: 1.1+ MB<jupyter_text>"Exited" is our response column, its the 14th column, we are going to need this information to modify the template given by the course to make it works for this case. The three first columns have no impact on the response so we will not include it in out training table. As shown in the course, we are going to preprocess our data using the template given in the course:<jupyter_code>X = dataset.iloc[:, 3:13].values # we modify indexes according to what we saw with the info() method of the dataset
y = dataset.iloc[:, 13].values # idem
X[1]<jupyter_output><empty_output><jupyter_text>Now we have to encode our categorical variables. We are going to do it using the template of the course. Here we have two categorical columns (Geography and Gender) so we have to create to encoders:<jupyter_code># Encoding categorical data
labelencoder_X_geo = LabelEncoder()
X[:, 1] = labelencoder_X_geo.fit_transform(X[:, 1])
labelencoder_X_gender = LabelEncoder()
X[:, 2] = labelencoder_X_gender.fit_transform(X[:, 2])
onehotencoder = OneHotEncoder(categorical_features = [1])
X = onehotencoder.fit_transform(X).toarray()
X = X[:, 1:]
X[0]
# Splitting the dataset into the Training set and Test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Feature Scaling
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
X_train.shape<jupyter_output><empty_output><jupyter_text>Our data are now preprocessed ! We can start building our model:  2. Let's build our ANN<jupyter_code># Initializing our ANN
classifier = Sequential()<jupyter_output><empty_output><jupyter_text>As a reminder, here are all the steps we must follow for training an ANN with stochastic gradient descent method. Dense function will be used for step 1. From step 2, we know that each features is attributed to one node, so we have to create 11 input nodes in our input layer. We also have to choose an activation function (step 3) as we saw in the course, we will use the best one for our hidden layers (based on experiment) : the rectifier function. The sigmoid function is a very good option for our output layer because it will gives us probabilities for each classes.After that, we will use learning rate to choose how weights are updated and we will also think about how many epochs we are going to do. Let's go ! ![title](../images/steps_ann.png)<jupyter_code># Adding the input layer and the first hidden layer of our ANN
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu', input_shape = (11,)))<jupyter_output><empty_output><jupyter_text>- *units* param corresponds to the number of node of the layer:**tip :** choose the number of nodes in the hidden layers as the average of the number of nodes in the input layer and the number of nodes in the output layerHere we have 11 nodes in input layer and 1 node in output layer (because binary output) so we choose 6 nodes in hidden layers- *kernel_initializer* param corresponds to the way we are initializing our weights:As we saw during the course, weights must be initialized as small numbers close to zero. The random uniform function allows us to initialize our weights in this way.- *activation* param corresponds to the activation function for hidden layers:We choose 'relu' for rectifier function.- *input_shape* is the number of nodes in the input layer Now, let’s create our second hidden layer, which will be the same as the first one. Here, we don’t have to specify the input shape because it can deduce this with the previous hidden layer. For the first one, there was no hidden layer yet created so we had to specify it.<jupyter_code># Adding the second hidden layer
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))<jupyter_output><empty_output><jupyter_text>Finally, we have to add our output layer. We have to change the activation function because we said that the activation function of our output layer will be the sigmoid function. We also want a single binary output so the units (output dimension) will be set to 1.<jupyter_code># Adding the output layer
classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>Now, we are going to apply stochastic gradient method on the whole neural network by compiling our model. Indeed, we have built our ANN but the weigths are still initialized, so now we need to find the best weights that will make our NN the most powerful.<jupyter_code># Compilling the ANN
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>- *optimizer* is the algorithm which will be used to find the optimal set of weights in the ANN, here we will use (as we seen in the course) the stochastic gradient descent method. There are several types of this method, a very efficient one is called 'adam', we will use this one.- *loss* is the loss function to be optimized within the SGD method. We will use here a log loss because the activation function of our output layer is the sigmoid function (this is the same as for logistic regression model).**tip**: for more than two categories, the log loss function is called "categorical_crossentropy"- *metrics* is the criterion that we use to evaluate our model. Here we are going to choose 'accuracy'. Now we are going to train our NN with the fit method<jupyter_code># Training the ANN
classifier.fit(X_train, y_train, batch_size=10, epochs=100)<jupyter_output>Epoch 1/100
8000/8000 [==============================] - 2s 227us/step - loss: 0.4929 - acc: 0.7952
Epoch 2/100
8000/8000 [==============================] - 1s 178us/step - loss: 0.4190 - acc: 0.7960
Epoch 3/100
8000/8000 [==============================] - 2s 192us/step - loss: 0.4102 - acc: 0.7960
Epoch 4/100
8000/8000 [==============================] - 2s 226us/step - loss: 0.4016 - acc: 0.8229
Epoch 5/100
8000/8000 [==============================] - 2s 198us/step - loss: 0.3941 - acc: 0.8295
Epoch 6/100
8000/8000 [==============================] - 2s 202us/step - loss: 0.3872 - acc: 0.8309
Epoch 7/100
8000/8000 [==============================] - 2s 209us/step - loss: 0.3816 - acc: 0.8365
Epoch 8/100
8000/8000 [==============================] - 2s 216us/step - loss: 0.3775 - acc: 0.8431
Epoch 9/100
8000/8000 [==============================] - 1s 182us/step - loss: 0.3734 - acc: 0.8447
Epoch 10/100
8000/8000 [==============================] - 1s 168us/step - loss: 0.3706 - acc: 0.8474[...]<jupyter_text>- *batch_size* is the number of observations after which you want to update the weights.- *epochs*, an epoch is an iteration over the entire x and y data provided. We need to repeat step one to six several time (number of epoch) on our dataset to train the model.For this two arguments, there is no rule and the experimentation is the best method to find out the best number to input.After running this cell, we can see how stochastic gradient descent if performing on our dataset and how accuracy is improving as the number of epochs realised increases.  3. Making predictions and evaluating model Finally, we can make our prediction:<jupyter_code># Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred = (y_pred > 0.5) # convert probabilities to binary output

# Making the Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
cm<jupyter_output><empty_output><jupyter_text>We can do a better plot of the confusion matrix:<jupyter_code>fig = plt.subplot()
sns.heatmap(cm, annot=True, fmt=".3f", linewidths=.5, square=True, cmap='Blues_r')
plt.ylabel('Actual exited label')
plt.xlabel('Predicted exited label')
fig.xaxis.set_ticklabels(['Stay', 'Leave'])
fig.yaxis.set_ticklabels(['Stay', 'Leave']);
plt.show()<jupyter_output><empty_output><jupyter_text>We have 1539 + 193 = 1732 correct predictions and 268 incorrect predictions. We can calculate the accuracy on test dataset (so this is the accuracy on new observations):<jupyter_code>accuracy = 1732/2000
accuracy<jupyter_output><empty_output><jupyter_text>4. Homework Use our ANN model to predict if the customer with the following informations will leave the bank: - Geography: France- Credit Score: 600- Gender: Male- Age: 40 years old- Tenure: 3 years- Balance: 60000 dollars- Number of Products: 2- Does this customer have a credit card ? Yes- Is this customer an Active Member: Yes- Estimated Salary: 50000 dollarsSo should we say goodbye to that customer ? Let's begin by creating a dataframe containing the observation:<jupyter_code>columns = dataset.columns.tolist()
data = [0,0,0, 600, 'France', 'Male', 40, 3, 60000.00, 2, 1, 1, 50000.00, 0]
table = {columns[n]:[data[n]] for n in range(14)}
homework_customer = pd.DataFrame(table)
homework_customer.head()<jupyter_output><empty_output><jupyter_text>We can now add our observation to our dataset:<jupyter_code>dataset_homework = dataset.copy()
dataset_homework = dataset_homework.append(homework_customer)
X_homework = dataset_homework.iloc[:, 3:13].values
y_homework = dataset_homework.iloc[:, 13].values # idem
X_homework[10000] # here is our observation<jupyter_output><empty_output><jupyter_text>Now, we can standardize the data like above (encode cathegorical features and feature scaling) and make the prediction:<jupyter_code>X_homework[:, 1] = labelencoder_X_geo.fit_transform(X_homework[:, 1])
X_homework[:, 2] = labelencoder_X_gender.fit_transform(X_homework[:, 2])
onehotencoder = OneHotEncoder(categorical_features = [1])
X_homework = onehotencoder.fit_transform(X_homework).toarray()
X_homework = X_homework[:, 1:]
X_homework = sc.transform(X_homework)
homework_observation = X_homework[10000:]
homework_prediction = classifier.predict(homework_observation)
homework_prediction<jupyter_output><empty_output><jupyter_text>The model predict a probability of 0.047 for this customer to leave the bank, so this customer is going to **stay** in the bank.  5. Solution There was an easiest way to do it. The fact is that the StandarScaler object will apply the same scale than previously when it was fitted. Futhermore, We could just look at the dummies variables of a similar column (France and Male obs) to create an array, there was no need to reapply encoders:<jupyter_code>new_prediction = classifier.predict(sc.transform(np.array([[0, 0, 600, 1, 40, 3, 60000, 2, 1, 1, 50000]])))
new_prediction<jupyter_output><empty_output><jupyter_text>Same result than found above !  6. Evaluating the ANN It’s important to evaluate the model to ensure that accuracy is not very different from one training to another, in other words we want to ensure that our model has a low variance. To evaluate our model, we will use the K-fold cross validation method.<jupyter_code>def build_ann():
    classifier = Sequential()
    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu', input_shape = (11,)))
    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))
    classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))
    classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    return classifier

classifier = KerasClassifier(build_fn=build_ann, batch_size=10, epochs=100, verbose=0)
accuracies = cross_val_score(estimator=classifier, X=X_train, y=y_train, cv=10, n_jobs=1)
accuracies<jupyter_output><empty_output><jupyter_text>Now, let's calculate the true accuracy of our model and its variance:<jupyter_code>mean = accuracies.mean()
variance = accuracies.std()
print('Mean accuracy:', mean, '\nVariance:', variance)<jupyter_output>Mean accuracy: 0.8486249951273204 
Variance: 0.01912499840888716<jupyter_text>The accuracy of 86% obtained in our first training of the model was not relevant ! But now, let's try to reach it by tuning our model !  7. Dropout regularization Let's add some dropout regularization to our two hidden layers to reduce overfitting.At each iteration of the training some neurons of your artificial neural network are randomly disabled to prevent them from being too dependent on each other when they learn the correlations and therefore, by overwriting these neurons the artificial neural network learns several independent correlations in the data because each time there is not the same configuration of the neurons. And the fact that we get these independent correlations of the data, thanks to the fact that the neurons work more independently, that prevents the neurons from learning too much and therefore that prevents overfitting.- *p* is the fraction of the neurons we want to drop. In general, begin with 10% and increase this value (by 10% steps) if there is still overfitting. Don't try to go over 50% to avoid underfitting.<jupyter_code># Initializing our ANN
classifier = Sequential()
# Adding the input layer and the first hidden layer of our ANN with dropout
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu', input_shape = (11,)))
classifier.add(Dropout(p=0.1)) 
# Adding the second hidden layer with dropout
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))
classifier.add(Dropout(p=0.1))
# Adding the output layer
classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))
# Compilling the ANN
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>8. Model tuning (homework : get over 86% accuracy)<jupyter_code>def build_ann(optimizer):
    classifier = Sequential()
    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu', input_shape = (11,)))
    classifier.add(Dropout(p=0.1)) 
    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))
    classifier.add(Dropout(p=0.1)) 
    classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))
    classifier.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
    return classifier

classifier = KerasClassifier(build_fn=build_ann, verbose=1)
parameters = {
    'batch_size' : [25, 38],
    'epochs' : [100, 510],
    'optimizer' : ['adam', 'rmsprop']
}
grid_search = GridSearchCV(estimator=classifier, param_grid=parameters, scoring='accuracy', cv=5)
grid_search.fit(X_train, y_train)
best_params = grid_search.best_params_
best_accuracy = grid_search.best_score_
print('Best accuracy:', best_accuracy, '\nBest parameters:', best_params)<jupyter_output>Epoch 1/10
6400/6400 [==============================] - 9s 1ms/step - loss: 0.5805 - acc: 0.7959
Epoch 2/10
6400/6400 [==============================] - 2s 342us/step - loss: 0.4458 - acc: 0.7980
Epoch 3/10
6400/6400 [==============================] - 2s 311us/step - loss: 0.4412 - acc: 0.7980
Epoch 4/10
6400/6400 [==============================] - 2s 299us/step - loss: 0.4347 - acc: 0.7980
Epoch 5/10
6400/6400 [==============================] - 2s 305us/step - loss: 0.4321 - acc: 0.7980
Epoch 6/10
6400/6400 [==============================] - 2s 301us/step - loss: 0.4314 - acc: 0.7980
Epoch 7/10
6400/6400 [==============================] - 2s 322us/step - loss: 0.4304 - acc: 0.7980
Epoch 8/10
6400/6400 [==============================] - 2s 308us/step - loss: 0.4315 - acc: 0.7980
Epoch 9/10
6400/6400 [==============================] - 2s 308us/step - loss: 0.4269 - acc: 0.7980
Epoch 10/10
6400/6400 [==============================] - 2s 311us/step - loss: 0.4282 - acc: 0.7980
1600/1600 [[...]
{'fraction_non_alphanumeric': 0.11914631413847188, 'fraction_numerical': 0.049854358055119875, 'mean_word_length': 5.373795073188147, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '112335', 'license': 'MIT', 'n_tokens_mistral': 5960, 'n_tokens_neox': 5187, 'n_words': 2034, 'path': 'Udemy/Deep_Learning_A-Z/notebooks/ann_homework_nb.ipynb', 'repo_name': 'nicohlr/courses', 'size': '17852'}