<jupyter_start><jupyter_text>Feedforward Neural Network - Bank Churn Prediction Training  Part 0: Import librariesThe default import for Keras will use Tensorflow backend.<jupyter_code>import keras
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from keras.layers import Dense
from keras.models import Sequential
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler<jupyter_output>Using TensorFlow backend.<jupyter_text>Part 1: Data ProcessingThe first step of data processing is to load data with pandas csv function and split the dataset into independent variables and dependent variable. Independent variable is also the properties to be used for training. Dependent variable is also the outcome that we want to predict.<jupyter_code># Importing the dataset

dataset = pd.read_csv('Churn_Modelling.csv')

# print out the dataset to inspect
print(dataset)
# independent variables
X = dataset.iloc[:, 3:13].values

# dependent variables: outcome
y = dataset.iloc[:, 13].values<jupyter_output><empty_output><jupyter_text>Encoding propertiesSome variables are designed by string or enums, so we need to encode into numerical variable in order to feed into our neural network. Everything is all calculated in a math way!<jupyter_code># Encoding for countries
labelencoder_X_1 = LabelEncoder()
X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])

# Encoding for gender
labelencoder_X_2 = LabelEncoder()
X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])<jupyter_output><empty_output><jupyter_text>Handle dummy variablesThe countries variable is not `ordinal` after encoding, France, German,Spain has no ordinal relationship. If we just simply use ordinal number to represent countries, the result will be affected in calculation. Hence, we need to create dummy variable.gender has no need to do it due to it only has two category (binary state).Moreover, please be care of dummy variable trap problem. There is a great example of dummy variable trap probelm in [Chinese](http://belleaya.pixnet.net/blog/post/30877354-%5B%E6%95%99%E5%AD%B8%5D-%5B%E7%B5%B1%E8%A8%88%5D-%E8%99%9B%E6%93%AC%E8%AE%8A%E6%95%B8-dummy-variable) and [English](http://www.algosome.com/articles/dummy-variable-trap-regression.html)<jupyter_code># add dummy variable for country code
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()
# avoid dummy variable trap for country code by removing one column
X = X[:, 1:]<jupyter_output><empty_output><jupyter_text>Splitting the dataset into the Training set and Test setWhen we have some dataset for supervised machine learning, we can separate into two datasets. One for training and one for test the prediction.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Feature ScalingWe definitely need to do [Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling) to standardize our input properties. Feature scaling can facilitate intensive computation. Also, feature scaling will affect the calculation speed for gradient descent. We use standard scaling here.<jupyter_code>sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)<jupyter_output><empty_output><jupyter_text>Part 2: Build Neural NetworkThere are two way to implement neural network in Keras, by `graph` or `sequential`. This example is a classification problem to determine a given customer will leave or stay. Also, we can use sigmoid function as output layer's activation function to output the probability of leaving.TODO: add a figure here to visualize the neural network layers.TODO: also need to take notes on each parameter.TODO: a lot of math stuff bellow need to be figured out.<jupyter_code># Initialising the ANN
classifier = Sequential()

# we can use the average number of inputs and outputs as output_dim variable by experience.
# Adding the input layer and the first hidden layer, units parameter is also called output_dim
input_layer = Dense(units=6, kernel_initializer='uniform', activation='relu', input_dim=11)
classifier.add(input_layer)

# Adding the second hidden layer
hidden_layer = Dense(units=6, kernel_initializer='uniform', activation='relu')
classifier.add(hidden_layer)
# Adding the output layer
# if we want to output as categories for countries, we need to change units to 3,
# and the activation function must use Softmax. Use Softmax when we need to deal
# with more than two categorical dependent variables.

output_layer = Dense(units=1, kernel_initializer='uniform', activation='sigmoid')
classifier.add(output_layer)
# Compile the ANN
# loss function also need to change to categorical_crossentropy for categorical results
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Part3: Fitting the ANN to the Training set Decide the training parameters1. update weights each observation (reinforcement learning). This is also a stochastic gradient dependent).2. update weights after a batch of observation (batch learning)3. when the whole training set passed through the ANN, that makes an epoch.<jupyter_code>classifier.fit(X_train, y_train, batch_size=10, epochs=100)<jupyter_output>Epoch 1/100
8000/8000 [==============================] - 2s - loss: 0.4736 - acc: 0.7960     
Epoch 2/100
8000/8000 [==============================] - 2s - loss: 0.4298 - acc: 0.7960     
Epoch 3/100
8000/8000 [==============================] - 2s - loss: 0.4240 - acc: 0.7961     
Epoch 4/100
8000/8000 [==============================] - 2s - loss: 0.4198 - acc: 0.8186     
Epoch 5/100
8000/8000 [==============================] - 2s - loss: 0.4168 - acc: 0.8275     
Epoch 6/100
8000/8000 [==============================] - 2s - loss: 0.4149 - acc: 0.8306     
Epoch 7/100
8000/8000 [==============================] - 2s - loss: 0.4137 - acc: 0.8320     
Epoch 8/100
8000/8000 [==============================] - 2s - loss: 0.4123 - acc: 0.8326     
Epoch 9/100
8000/8000 [==============================] - 2s - loss: 0.4110 - acc: 0.8334     
Epoch 10/100
8000/8000 [==============================] - 2s - loss: 0.4102 - acc: 0.8317     
Epoch 11/100
8000/8000 [==============================] - 2[...]<jupyter_text>Part 4: Making predictions and evaluating the model<jupyter_code># Predicting the Test set results
y_pred = classifier.predict(X_test)

# numpy can convert all array into conditional result
y_pred = (y_pred > 0.5)

print(y_pred)<jupyter_output>[[False]
 [False]
 [False]
 ..., 
 [False]
 [False]
 [False]]<jupyter_text>Making the Confusion MatrixConfusion matrix is to evaluate the accurracy of your prediction. Please read the following article to understand more. It acutally simply display the prediction result and the test dataset to see how many result is the same in its category.- [Simple guide](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/)- [Wiki](https://en.wikipedia.org/wiki/Confusion_matrix)- [scikit-learning reference](http://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.metrics)<jupyter_code>cm = confusion_matrix(y_test, y_pred)
print(cm)

# you can calculate the accuracy on your own
accuracy = (cm[0][0] + cm[1][1]) / len(y_pred)

# or simply use scikit learn function
accuracy = accuracy_score(y_test, y_pred)

print(f'accuracy: {accuracy}')<jupyter_output>[[1554   41]
 [ 281  124]]
accuracy: 0.839<jupyter_text>Predicting a single new observationWe can try to predict single customer's result by generate a new test array. Let's try to predict the customer with the following informations will leave the bank:```Geography: FranceCredit Score: 600Gender: MaleAge: 40Tenure: 3Balance: 60000Number of Products: 2Has Credit Card: YesIs Active Member: YesEstimated Salary: 50000```<jupyter_code># add 0.0 to avoid warning: 
# DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
new_test = np.array([[0.0, 0, 600, 1, 40, 3, 60000, 2, 1, 1, 50000]])
new_test = sc.transform(new_test)
new_prediction = classifier.predict(new_test)
new_prediction = (new_prediction > 0.5)
print(new_prediction)<jupyter_output>[[False]]
{'fraction_non_alphanumeric': 0.12642164491799354, 'fraction_numerical': 0.04680952951035556, 'mean_word_length': 5.6618819776714515, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '232441', 'license': 'MIT', 'n_tokens_mistral': 2763, 'n_tokens_neox': 2438, 'n_words': 875, 'path': 'ANN/concept/concept.ipynb', 'repo_name': 'memorysaver/deep-learning-journey', 'size': '8353'}