<jupyter_start><jupyter_text>Artificial Neural Network in Python with KerasThis notebook is a simple demonstration to how to build an Artificial Neuron Network using TensorFlow to a classification problem.**Problem Description**The goal of program is to build a classification model to predict if a certain client will leave the bank service in the next six months.**Dataset Description**The dataset is composed by 10000 instances (rows) and 14 features (columns).   The features considered to build the model are:-  RowNumber (This is not important to the model)-  CustomerId (This is not important to the model)-  Surname (This is not important to the model)-  CreditScore (numerical variable)-  Geography (categorical variable)-  Gender (categorical variable)-  Age (numerical variable)-  Tenure (categorical variable)-  Balance (numerical variable)-  NumOfProducts (categorical variable)-  HasCrCard (categorical variable)-  EstimatedSalary (numerical variable)-  Exited (target)  Data preprocessing  Importing Libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import keras # To build our ANN model
from keras.models import Sequential # To initialize the ANN
from keras.layers import Dense # To creat the hidden layers<jupyter_output><empty_output><jupyter_text>Importing the Dataset<jupyter_code>dataset = pd.read_csv('Churn_Modelling.csv')<jupyter_output><empty_output><jupyter_text>Visualizing some informations from the dataset<jupyter_code>dataset.shape
dataset.dtypes
dataset.isna().sum()
dataset.head(10)
dataset.describe()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis (EDA)  Visualizing the categorical variables  Target variable<jupyter_code>dataset['Exited'].value_counts()
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
sns.countplot(x=dataset['Exited'])
plt.subplot(1, 2, 2)
values = dataset.iloc[:, - 1].value_counts(normalize = True).values # to show the binirie values in parcentage
index = dataset.iloc[:, -1].value_counts(normalize = True).index
plt.pie(values, labels= index, autopct='%1.1f%%', colors=['b', 'tab:orange'])
plt.show()<jupyter_output><empty_output><jupyter_text>Others categorical variables<jupyter_code>categorical_list = ['Geography', 'Gender', 'Tenure', 'NumOfProducts', 'HasCrCard','IsActiveMember', 'Exited']
data_cat = dataset[categorical_list]
fig = plt.figure(figsize=(15, 15))
plt.suptitle('Pie Chart Distribution', fontsize = 20)
for i in range(1, data_cat.shape[1]):
    plt.subplot(2, 3, i)
    f = plt.gca()
    f.axes.get_yaxis().set_visible(False)
    f.set_title(data_cat.columns.values[i - 1])
  # Setting the biniries values
    values = data_cat.iloc[:, i - 1].value_counts(normalize = True).values # to show the binirie values in parcentage
    index = data_cat.iloc[:, i -1].value_counts(normalize = True).index
    plt.pie(values, labels= index, autopct='%1.1f%%')
    plt.axis('equal')
#fig.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()
plt.figure(figsize=(15, 10))
for i in range(1, data_cat.shape[1]):
    plt.subplot(3, 3, i)
    sns.countplot(x=data_cat.iloc[: , i-1], hue=data_cat['Exited'])
plt.show()<jupyter_output><empty_output><jupyter_text>Visualizing the numerical variables<jupyter_code>numerical_list = ['CreditScore', 'Age', 'Balance', 'EstimatedSalary', 'Exited']
data_num  = dataset[numerical_list]<jupyter_output><empty_output><jupyter_text>Distribution of numerical variables<jupyter_code>plt.figure(figsize=(25,15))
plt.suptitle('Histograms of numerical variables (mean values)', fontsize = 20)
for i in range(1, data_num.shape[1]):
    plt.subplot(2, 2, i)
    f = plt.gca()
    sns.histplot(data_num.iloc[:, i-1], color = '#3F5D7D', kde= True)
plt.show()
plt.figure(figsize=(25,15))
plt.suptitle('Histograms of numerical variables (mean values)', fontsize = 20)
for i in range(1, data_num.shape[1]):
    plt.subplot(2, 2, i)
    f = plt.gca()
    sns.histplot(data=data_num, x=data_num.iloc[:, i-1], hue='Exited', kde = True)
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation and PairPlot (scatter)  Correlation with the response variable<jupyter_code>column_drop = ['RowNumber', 'CustomerId', 'Surname', 'Exited']
dataset.drop(columns=column_drop).corrwith(dataset.Exited).plot.bar(
        figsize = (20, 10), title = "Correlation with Exited", fontsize = 15,
        rot = 45, grid = True)<jupyter_output><empty_output><jupyter_text>Correlation Between the Variables<jupyter_code>column_drop = ['RowNumber', 'CustomerId', 'Surname']
## Correlation Matrix
sns.set(style="white")

# Compute the correlation matrix
corr = dataset.drop(columns=column_drop).corr()

# Generate a mask for the upper triangle
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(10, 20))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=1, vmin=-1, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5}, annot = True)<jupyter_output><empty_output><jupyter_text>Pair plot for the numerical variables<jupyter_code>sns.pairplot(data_num, hue = 'Exited', kind = 'scatter', corner=True, diag_kind='None')<jupyter_output><empty_output><jupyter_text>EDA conclusion**Target variable**Represented by 0 or 1 (stay/leave) shows a frequency of $79.6 \%$ for the customers which decided to stay in the bank against $20.4\%$ of customers that decided to leave the bank.**Categorical variables** The analysis of these variables shows that the most clients are French. The majority of clients are males. The frequency of these variables are approximately equals without a fact that requires attention and manipulation, it means feature engineering.**Numerical variables**The distribution of these variables are normal except for the salary. One interesting aspect is the age distribution for the clients which leaved the bank, the distribution shows that the almost clients have age among 40 and 50 years. This fact can be well visualized in the scatter plots.**Correlation**The correlation between the target and the independent variables are not so big, but considerable to build the model. The correlation among the independent variables are satisfactory, the almost of correlation show a low values.  Building the Artificial Neuron Network  Data Preprocessing  Excluding not important columns<jupyter_code>dataset = dataset.drop(columns=['RowNumber', 'CustomerId', 'Surname'])<jupyter_output><empty_output><jupyter_text>Encoding Categorical Data<jupyter_code>dataset['Gender'] = dataset['Gender'].astype('category').cat.codes<jupyter_output><empty_output><jupyter_text>One Hot Econding<jupyter_code>dataset = pd.get_dummies(dataset)
dataset.head()<jupyter_output><empty_output><jupyter_text>Defining the independent and target variable<jupyter_code>response = dataset['Exited']
i_var = dataset.drop(columns=['Exited'])<jupyter_output><empty_output><jupyter_text>Splitting the Dataset into train and test set<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(i_var, response, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Feature ScalingIn almost ANN models we must to apply feature scale.<jupyter_code>from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train_bckp = pd.DataFrame(sc_X.fit_transform(X_train))
X_test_bckp = pd.DataFrame(sc_X.transform(X_test))
X_train_bckp.columns = X_train.columns.values
X_test_bckp.columns = X_test.columns.values
X_train_bckp.index = X_train.index.values
X_test_bckp.index = X_test.index.values
X_train = X_train_bckp
X_test = X_test_bckp<jupyter_output><empty_output><jupyter_text>The Artificial Neuron Network - ANNAn Artificial Neuron Network is a technique that tries to reproduce brain functions.  To build an ANN, we consider a set of input information about something, each of these information will be considered as a neuron, we call this set of neuron input layer. As we know, in the brain we have many neurons and the communication among them is made by synapses process. The neurons from the input layer will communicate with other set of neurons in a hidden layer by the synapses process. Once the communication between the neurons of input and hidden layer is made, the initial information is changed, in this moment, the neurons of the hidden layer will communicate with a others neurons, these neurons is considered as a output layer and it gives a response about something. For example, in a classification model, if we consider five independent variables, the input layer will be composed by five neurons, these neurons communicate with the hidden layer (the number of neurons must be chosen), the hidden layer communicate with the output layer, that provides the final response 0 or 1, if we have two class.**How do synapses work in an ANN?**The synapses process in an ANN is made by an activation function, this function transforms the input information according with an associated weight (this might be interpreted as an importance degree to each input variable). We have some kind of activation function as- Threshold function- Sigmoid function- Rectifier function- Hyperbolic tangent function.For this notebook we are interested in the rectifier function and sigmoid function.The rectifier function is defined as $\phi(x) = \max(x,0)$, if a certain values is less than 0 the function returns 0, otherwise the function returns the maximum value. The synapses or the communication is made according $\sum_{i=1}^{m}w_{i}x_{i}$, where $m$ is the number of input variable. We consider the rectifier function between the input layer and the hidden layers. The activation function that we consider to make the communication among the hidden and output layer is the sigmoid function. The sigmoid function returns the probability of occurrence to certain class, this function is defined as $\phi(x) = \frac{1}{1 + e^{-x}}$.**How do an ANN learn?**The learning process of an ANN starts with the input parameters in a input layer, this layer communicate with the hidden layer by synapses. The next step is the communication among the hidden layer with the output layer, to obtain a response. In this stage, the learning process is not ended. After the response, the ANN must calculate the loss function to measure the precision of the prediction. Once the loss function was calculated there is the retro-propagation process, this process tries to find the minimal of the loss function changing the associated weights to each neuron. After this step, the ANN remake the previous process of synapses among the layers. The number of retro-propagation is defined as epochs. **About loss function**One of the most used loss function is the cross entropy. To more details [see](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e).**About the retro-propagation**Here, we consider the stochastic gradient descent. This method tries to minimize the loss function changing the weights of each neuron. In this process, if we have 128 instances, we choose a batch number as 32, it means that we calculate the response for the associated batch number and we returns to the begin to remake the same process to other 32 instances. To more details [see](https://en.wikipedia.org/wiki/Stochastic_gradient_descentAdam).**Summarizing the ANN process** 1. Initializing the ANN 2. Build the input layer 3. Build the hidden layers 4. Build the output layer 5. Training the ANN    1. Compile   2. Train.  Initializing the ANNThe first step is to create an object to build a sequence layer. This sequence layer takes account the input layer (the parameters that we initialize the ANN), the hidden layers and output layer. To do it, we utilize the Keras library and the object Sequencial.<jupyter_code>classifier = Sequential() # To initiate the ANN<jupyter_output><empty_output><jupyter_text>Input layerIn this step, we create the input layer, it means, we set all independent variables (considering that these variables as input neurons). To communicate the input layer with the first hidden layer, we must to choose an activation function. For this case, we choose rectifier function (the weights to each input parameters are chosen randomly by the ann object). Units is the number of input neurons, relu is the activation function.<jupyter_code>classifier.add(Dense(units=X_train.shape[1], activation='relu',
                    kernel_initializer= 'uniform',
                    input_dim=X_train.shape[1])) # Creating the hidden layer<jupyter_output><empty_output><jupyter_text>Hidden layersHere, we set the hidden layers. We can put how much we want. For this problem, we consider just one hidden layer. The object to create the hidden layer is the same to the input layer, but we can change it according with the problem. The parameters are the same of the input layer.<jupyter_code>layers.Dropout(0.2) # To drop 20% of input neurons, to avoid overfit.
classifier.add(Dense(units=X_train.shape[1], activation='relu',
                    kernel_initializer= 'uniform')) # Creating the hidden layer<jupyter_output><empty_output><jupyter_text>Output layerThe last layer is the output layer. We use the same object the we used to build the preceding layers, but here, we make some changes in the parameters. Like this problem has a binary response yes or no, the number of neuron corresponds to 1, but if the response gives more then two results (0, 1, 2, for example), we must to consider the correspondent number of responses. The second change is on the activation function. Here, we consider sigmoid activation function, for one simple reason, this gives to us the probability which will be interpreted as 0 or 1 according with the values. Likelihood less than $0.5$ is considered as 0, otherwise 1.<jupyter_code>layers.Dropout(0.2) 
classifier.add(Dense(units=1, activation='sigmoid',
                    kernel_initializer= 'uniform')) # Creating the hidden layer<jupyter_output><empty_output><jupyter_text>Learning process of the ANN  Compiling the ANNCompile the ANN is one of most important step. We select a method to optimize our ANN, stochastic gradient descent, represented by adam. The lost function is also very import, because from this function we are able to improve the accuracy, precision and other relevant parameters. The lost function that we are going to use is Binary Cross Entropy (due to have a binary response). Finally, the metric which we choose is the accuracy. Beyond this metric, we have others important metrics as F1, precision and recall.<jupyter_code>classifier.compile(optimizer='adam', loss='binary_crossentropy', 
                   metrics=['accuracy', metrics.Recall(), metrics.Precision()])<jupyter_output><empty_output><jupyter_text>Training the ANNNow we train the ANN. Here we have two important hype parameters. Batch size determines how many instances the stochastic gradient descent will consider to work in the minimizing process. Epochs indicates the number of retro-propagation we want to train the model.<jupyter_code>classifier.fit(X_train, y_train, batch_size=32, epochs=100)<jupyter_output>Epoch 1/100
250/250 [==============================] - 0s 660us/step - loss: 0.5654 - accuracy: 0.7958
Epoch 2/100
250/250 [==============================] - 0s 636us/step - loss: 0.4385 - accuracy: 0.7960
Epoch 3/100
250/250 [==============================] - 0s 664us/step - loss: 0.4317 - accuracy: 0.7960
Epoch 4/100
250/250 [==============================] - 0s 655us/step - loss: 0.4286 - accuracy: 0.7960
Epoch 5/100
250/250 [==============================] - 0s 610us/step - loss: 0.4261 - accuracy: 0.7960
Epoch 6/100
250/250 [==============================] - 0s 646us/step - loss: 0.4236 - accuracy: 0.7960
Epoch 7/100
250/250 [==============================] - 0s 632us/step - loss: 0.4211 - accuracy: 0.8016
Epoch 8/100
250/250 [==============================] - 0s 622us/step - loss: 0.4192 - accuracy: 0.8175
Epoch 9/100
250/250 [==============================] - 0s 632us/step - loss: 0.4176 - accuracy: 0.8206
Epoch 10/100
250/250 [==============================] - 0s 647us/step - l[...]<jupyter_text>Making a single predictionHere, we make a simple prediction. Remember, Geography and Gender was changed. We must take it account.<jupyter_code>print(classifier.predict(sc_X.transform([[1, 0, 0, 600, 1, 1, 40, 60000, 2, 1, 1, 50000]])) > 0.5)<jupyter_output><empty_output><jupyter_text>Predicting the test results<jupyter_code>y_pred = classifier.predict(X_test)
y_pred = (y_pred > 0.5) # Here we must put it, because we have as outcome the probability, but we want a binary response.<jupyter_output><empty_output><jupyter_text>Making the confusion matrix and metrics scores<jupyter_code>from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
plt.figure()
sns.heatmap(cm, annot=True)
plt.show()
# Here, we consider the metrics class from TensorFlow library.
m1 = metrics.Accuracy() # Object
m2 = metrics.Recall()
m3 = metrics.Precision()

m1.update_state(y_test, y_pred) # Calculating the metric
m2.update_state(y_test, y_pred)
m3.update_state(y_test, y_pred)

print('Metric results for the test set\n')

print('Accuracy {:.2f}%'.format(m1.result().numpy()*100))
print('Recall {:.2f}%'.format(m2.result().numpy()*100))
print('Precision {:.2f}%'.format(m3.result().numpy()*100))
score_train = classifier.evaluate(X_train, y_train)
print('\n')
print('Metric results for the training set\n')

print('Accuracy {:.2f}%'.format(score_train[1]*100))
print('Recall {:.2f}%'.format(score_train[2]*100))
print('Precision {:.2f}%'.format(score_train[3]*100))<jupyter_output><empty_output><jupyter_text>Metrics with cross validate<jupyter_code>def build_classifier(optimizer='adam'):
    classifier = Sequential()
    classifier.add(Dense(units = X.shape[1], kernel_initializer = 'uniform', activation = 'relu', input_dim = X.shape[1]))
    classifier.add(Dense(units = X.shape[1], kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = X.shape[1], kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = optimizer, loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
scoring = ['accuracy', 'recall', 'precision'] # List of metrics

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_validate
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 32, epochs = 100)
accuracies = cross_validate(estimator = classifier, X = X_train, y = y_train, 
                             cv = 10, n_jobs = -1, scoring = scoring)
print('Metric results with cross validate cv=10')
print('\n')
print("Accuracy: {:.2f} %".format(accuracies['test_accuracy'].mean()*100))
print("Recall: {:.2f} %".format(accuracies['test_recall'].mean()*100))
print("Precision: {:.2f} %".format(accuracies['test_precision'].mean()*100))<jupyter_output><empty_output><jupyter_text>Boosting the model with GridSearchCV<jupyter_code>scoring = {'ACC' : 'accuracy', 'REC' : 'recall', 'PC' : 'precision'} # Dictionary of metrics
from sklearn.model_selection import GridSearchCV

classifier = KerasClassifier(build_fn = build_classifier)

parameters = {'batch_size': [8, 16, 32],
              'epochs': [50, 100, 500],
              'optimizer': ['adam', 'rmsprop']}

grid_search = GridSearchCV(estimator = classifier,
                           param_grid = parameters,
                           scoring = scoring,
                           refit = 'ACC',
                           cv = 10, n_jobs = -1)


grid_search = grid_search.fit(X_train, y_train)
best_parameters = grid_search.best_params_
best_accuracy = grid_search.best_score_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11088254477948924, 'fraction_numerical': 0.02543776027294165, 'mean_word_length': 5.3761996161228405, 'pattern_counts': {'":': 1, '<': 108, '<?xml version=': 0, '>': 110, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '184551', 'license': 'Apache-2.0', 'n_tokens_mistral': 6319, 'n_tokens_neox': 5754, 'n_words': 2104, 'path': '1 - Artifical Neural Networks/.ipynb_checkpoints/Artificial_Neural_Network_Keras-checkpoint.ipynb', 'repo_name': 'IgorMeloS/Deep-Learning', 'size': '19931'}