<jupyter_start><jupyter_text>XOR Prediction Neural Network A simple neural network which will learn the XOR logic gate.I will provide you with any links necessary so that you can read about the different aspects of this NN(Neural Network).  Neural Network Info All information regarding the neural network:- Input Layer Units = 2 (Can be modified)- Hidden Layer Units = 2 (Can be modified)- Output Layer Units = 1 (Since this is problem specific, it can't be modified)- No. of hidden layers = 1- Learning Algorithm = Backpropagation![nn_pic.png](attachment:nn_pic.png)Feel free to mess around with it and try out different things.<jupyter_code>import numpy as np # For matrix math
import matplotlib.pyplot as plt # For plotting

import sys # For printing<jupyter_output><empty_output><jupyter_text>Neural Network ImplementationInitially, I was going to approach this in an Object Oriented manner but I think that it would be much easier to read and implement, functionally. So, let's get started.  Training DataThe XOR logic gate returns true when the number of inputs given is odd and false when they're even. Here is the simple training dataset.<jupyter_code># The training data.
X = np.array([
    [0, 1],
    [1, 0],
    [1, 1],
    [0, 0]
])

# The labels for the training data.
y = np.array([
    [1],
    [1],
    [0],
    [0]
])

outNN =np.zeros([4])
X
y<jupyter_output><empty_output><jupyter_text>Additional ParametersThese are just additional parameters which are required by the weights for their dimensions.<jupyter_code>num_i_units = 2 # Number of Input units
num_h_units = 2 # Number of Hidden units
num_o_units = 1 # Number of Output units<jupyter_output><empty_output><jupyter_text>Neural Network ParametersThese are the parameters required directly by the NN. Comments should describe the variables.<jupyter_code># The learning rate for Gradient Descent.
learning_rate = 0.01

# The parameter to help with overfitting.
reg_param = 0

# Maximum iterations for Gradient Descent.
max_iter = 100

# Number of training examples
m = 4<jupyter_output><empty_output><jupyter_text>Weights and BiasesThese are the numbers the NN needs to learn to make accurate predictions.For the connections being made from the input layer to the hidden layer, the weights and biases are arranged in the following order: **each row contains the weights for each hidden unit**. Then, the shape of these set of weights is: *(number of hidden units X number of input units)* and the shape of the biases for this connection will be: *(number of hidden units X 1)*.So, the overall shape of the weights and biases are:**Weights1(Connection from input to hidden layers)**: num_h_units X num_i_units**Biases1(Connection from input to hidden layers)**: num_h_units X 1**Weights2(Connection from hidden to output layers)**: num_o_units X num_h_units**Biases2(Connection from hidden to output layers)**: num_o_units X 1 Generating the WeightsThe weights here are going to be generated using a [Normal Distribution(Gaussian Distribution)](http://mathworld.wolfram.com/NormalDistribution.html). They will also be seeded so that the outcome always comes out the same.<jupyter_code>np.random.seed(1)
W1 = np.random.normal(0, 1, (num_h_units, num_i_units)) # 2x2
W2 = np.random.normal(0, 1, (num_o_units, num_h_units)) # 1x2

B1 = np.random.random((num_h_units, 1)) # 2x1
B2 = np.random.random((num_o_units, 1)) # 1x1
W1
W2
B1
B2<jupyter_output><empty_output><jupyter_text>Sigmoid Function[This](http://mathworld.wolfram.com/SigmoidFunction.html) function maps any input to a value between 0 and 1.![sigmoid-curve.png](attachment:sigmoid-curve.png)In my implementation, I have added a boolean which if set to true, will return [Sigmoid Prime(the derivative of the sigmoid function)](http://www.ai.mit.edu/courses/6.892/lecture8-html/sld015.htm) of the input value. This will be used in backpropagation later on.<jupyter_code>def sigmoid(z, derv=False):
    if derv: return z * (1 - z)
    return 1 / (1 + np.exp(-z))<jupyter_output><empty_output><jupyter_text>Forward Propagation[This](https://en.wikipedia.org/wiki/Feedforward_neural_network) is how predictions are made. Propagating the input through the NN to get the output.In my implementation, the forward function only accepts a feature vector as row vector which is then converted to a column vector. Also, the predict boolean, if set to true, only returns the output. Otherwise, it returns a tuple of the outputs of all the layers.<jupyter_code>def forward(x, predict=False):
    a1 = x.reshape(x.shape[0], 1) # Getting the training example as a column vector.

    z2=  W1.dot(a1) + B1 # 2x2 * 2x1 + 2x1 = 2x1
    a2 = sigmoid(z2) # 2x1

    z3 = W2.dot(a2) + B2 # 1x2 * 2x1 + 1x1 = 1x1
    a3 = sigmoid(z3)

    if predict: return a3
    return (a1, a2, a3)<jupyter_output><empty_output><jupyter_text>Gradients for the Weights and BiasesThese variables will contain the gradients for the weights and biases which will be used by gradient descent to update the weights and biases.Also, creating the vector which will be storing the cost values for each gradient descent iteration to help visualize the cost as the weights and biases are updated.<jupyter_code>dW1 = 0 # Gradient for W1
dW2 = 0 # Gradient for W2

dB1 = 0 # Gradient for B1
dB2 = 0 # Gradient for B2

cost = np.zeros((max_iter, 1)) # Column vector to record the cost of the NN after each Gradient Descent iteration.<jupyter_output><empty_output><jupyter_text>TrainingThis is the training function which contains the meat of NN. This contains forward propagation and [Backpropagation](http://neuralnetworksanddeeplearning.com/chap2.html). BackpropagationThe process of propagating the error in the output layer, backwards through the NN to calculate the error in each layer. Intuition: It's like forward propagation, but backwards.Steps(for this NN):1. Calculate the error in the output layer(dz2).2. Calculate the error in the weights connecting the hidden layer to the output layer using dz2 (dW2).3. Calculate the error in the hidden layer(dz1).4. Calculate the error in the weights connecting the input layer to the hidden layer using dz1 (dW1).5. The errors in the biases are just the errors in the respective layers.Afterwards, the gradients(errors) of the weights and biases are used to update the corresponding weights and biases by multiplying them with the negative of the learning rate and scaling it by divinding it by the number of training examples.While iterating over all the training examples, the cost is also being calculated simultaneously for each example. Then, a regurlization parameter is added, although for such a small dataset, regularization is unnecessary since to perform well, the NN will have to over fit to the training data.<jupyter_code>def train(_W1, _W2, _B1, _B2): # The arguments are to bypass UnboundLocalError error
    for i in range(max_iter):
        c = 0
        
        dW1 = 0
        dW2 = 0

        dB1 = 0
        dB2 = 0
        
        for j in range(m):
            sys.stdout.write("\rIteration: {} and {}".format(i + 1, j + 1))

            # Forward Prop.
            a0 = X[j].reshape(X[j].shape[0], 1) # 2x1

            z1 = _W1.dot(a0) + _B1 # 2x2 * 2x1 + 2x1 = 2x1
            a1 = sigmoid(z1) # 2x1

            z2 = _W2.dot(a1) + _B2 # 1x2 * 2x1 + 1x1 = 1x1
            a2 = sigmoid(z2) # 1x1

            # Back prop.
            dz2 = a2 - y[j] # 1x1
            dW2 += dz2 * a1.T # 1x1 .* 1x2 = 1x2

            dz1 = np.multiply((_W2.T * dz2), sigmoid(a1, derv=True)) # (2x1 * 1x1) .* 2x1 = 2x1
            dW1 += dz1.dot(a0.T) # 2x1 * 1x2 = 2x2

            dB1 += dz1 # 2x1
            dB2 += dz2 # 1x1

            c = c + (-(y[j] * np.log(a2)) - ((1 - y[j]) * np.log(1 - a2)))
            sys.stdout.flush() # Updating the text.
        
        _W1 = _W1 - learning_rate * (dW1 / m) + ( (reg_param / m) * _W1)
        _W2 = _W2 - learning_rate * (dW2 / m) + ( (reg_param / m) * _W2)

        _B1 = _B1 - learning_rate * (dB1 / m)
        _B2 = _B2 - learning_rate * (dB2 / m)
        cost[i] = (c / m) + ( 
            (reg_param / (2 * m)) * 
            (
                np.sum(np.power(_W1, 2)) + 
                np.sum(np.power(_W2, 2))
            )
        )
    return (_W1, _W2, _B1, _B2)<jupyter_output><empty_output><jupyter_text>RunningNow, let's try out the NN. Here, I have called the train() function. You can make any changes you like and then run all the kernels again. I have also plotted the cost function to visual how the NN performed.The console printing might be off.The weights and biases are then shown.<jupyter_code>W1, W2, B1, B2 = train(W1, W2, B1, B2)
W1
W2
B1
B2<jupyter_output><empty_output><jupyter_text>PlottingNow, let's plot a simple plot showing the cost function with respect to the number of iterations of gradient descent.<jupyter_code># Assigning the axes to the different elements.
plt.plot(range(max_iter), cost)

# Labelling the x axis as the iterations axis.
plt.xlabel("Iterations")

# Labelling the y axis as the cost axis.
plt.ylabel("Cost")

# Showing the plot.
plt.show()<jupyter_output><empty_output><jupyter_text>ObservationWith the initial parameters, the cost function doesn't look that good. It is decreasing which is a good sign but it isn't flattening out. I have tried, multiple different values but this some seems like the best fit.Try out your own values, run the notebook again and see what you get.<jupyter_code>coba = np.array([
    [0, 1],
    [1, 0],
    [1, 1],
    [0, 0]
])
coba
    # Forward Prop.
    for j in range(4):
            a0 = coba[j].reshape(coba[j].shape[0], 1) # 2x1

            z1 = W1.dot(a0) + B1 # 2x2 * 2x1 + 2x1 = 2x1
            a1 = sigmoid(z1) # 2x1

            z2 = W2.dot(a1) + B2 # 1x2 * 2x1 + 1x1 = 1x1
            outNN[j] = sigmoid(z2) # 1x1
outNN
plt.plot(y, 'bo', linewidth=2, markersize=12)

for j in range(4):
    plt.plot(j,outNN[j], 'r+', linewidth=2, markersize=12)
for j in range(4):
    plt.plot(y, 'bo', j,outNN[j], 'r+', linewidth=2, markersize=12)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09239940387481371, 'fraction_numerical': 0.03199205166418281, 'mean_word_length': 3.5568130375735625, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '18945', 'license': 'MIT', 'n_tokens_mistral': 3427, 'n_tokens_neox': 3206, 'n_words': 1251, 'path': 'ANN_PID_Tuning/.ipynb_checkpoints/XOR-Net-Notebook - oprek2-checkpoint.ipynb', 'repo_name': 'bernardusrendy/ssicon', 'size': '10065'}