<jupyter_start><jupyter_text>Neural Networks consist of the following components* An input layer, x* An arbitrary amount of hidden layers* An output layer, ŷ* A set of weights and biases between each layer, W and b* A choice of activation function for each hidden layer, σ. In this tutorial, we’ll use a Sigmoid activation function.* The diagram below shows the architecture of a 2-layer Neural Network (note that the input layer is typically excluded when counting the number of layers in a Neural Network)![](https://cdn-images-1.medium.com/max/800/1*sX6T0Y4aa3ARh7IBS_sdqw.png)  Training the Neural NetworkThe output ŷ of a simple 2-layer Neural Network is:$$ y = \sigma{(W_2\sigma{(W_1 x + b_1)} + b_2)}  $$You might notice that in the equation above, the weights W and the biases b are the only variables that affects the output ŷ.Naturally, the right values for the weights and biases determines the strength of the predictions. The process of fine-tuning the weights and biases from the input data is known as training the Neural Network.Each iteration of the training process consists of the following steps:* Calculating the predicted output ŷ, known as feedforward* Updating the weights and biases, known as backpropagation.The sequential graph below illustrates the process.![](https://cdn-images-1.medium.com/max/800/1*CEtt0h8Rss_qPu7CyqMTdQ.png)  Backpropagation![](https://cdn-images-1.medium.com/max/1800/1*7zxb2lfWWKaVxnmq2o69Mw.png)<jupyter_code>class NeuralNetwork:
    def __init__(self, x, y):
        self.input = x
        self.weights1 = np.random.rand(self.input.shape[1], 4)
        self.weights2 = np.random.rand(4, 1)
        self.y = y
        self.output = np.zeros(self.y.shape)
        
        
    def feedforward(self):
        self.layer1 = sigmoid(np.dot(self.input , self.weights1))
        self.layer2 = sigmoid(np.dot(self.layer1 , self.weights2))
        return self.layer2

    def backprop(self):
        d_weights2 = np.dot(self.layer1.T , (2*(self.y - self.output) * sigmoid_derivative(self.output)))
        d_weights1 = np.dot(self.input.T , (np.dot(2*(self.y - self.output) * sigmoid_derivative(self.output), self.weights2.T) *  sigmoid_derivative(self.layer1)))
        
        #update the weights with the derivative (slope)  of the loss function                   
        self.weights1 += d_weights1
        self.weights2 += d_weights2
        
    def train(self):
        self.output = self.feedforward()
        self.backprop()<jupyter_output><empty_output><jupyter_text>Loss FunctionThere are many available loss functions, and the nature of our problem should dictate our choice of loss function. In this tutorial, we’ll use a simple sum-of-sqaures error as our loss function. sum - of -squares  Error = sum(y-y)^2<jupyter_code># Define dataset
X = np.array(([1, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]), dtype = float)
y = np.array(([0], [1], [1], [0]), dtype = float)

# Train
NN = NeuralNetwork(X , y)
for i in range(1000):
    if i % 100 == 0 :
        print(f'''For iteration {i} \n Input:{X} \n Actual Output : {y} \n   
        Predicted Output : {NN.feedforward()} \n Loss: {np.mean(np.square(y)- NN.feedforward())} ''')
        
    NN.train()<jupyter_output>For iteration 0 
 Input:[[1. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [1. 1. 1.]] 
 Actual Output : [[0.]
 [1.]
 [1.]
 [0.]] 
   
        Predicted Output : [[0.7883838 ]
 [0.79467201]
 [0.7883838 ]
 [0.81517954]] 
 Loss: -0.29665478594769135 
For iteration 100 
 Input:[[1. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [1. 1. 1.]] 
 Actual Output : [[0.]
 [1.]
 [1.]
 [0.]] 
   
        Predicted Output : [[0.47645885]
 [0.87972053]
 [0.47645885]
 [0.1739196 ]] 
 Loss: -0.001639456886734128 
For iteration 200 
 Input:[[1. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [1. 1. 1.]] 
 Actual Output : [[0.]
 [1.]
 [1.]
 [0.]] 
   
        Predicted Output : [[0.49578437]
 [0.94018359]
 [0.49578437]
 [0.07817594]] 
 Loss: -0.002482069623474758 
For iteration 300 
 Input:[[1. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [1. 1. 1.]] 
 Actual Output : [[0.]
 [1.]
 [1.]
 [0.]] 
   
        Predicted Output : [[0.49801179]
 [0.95720114]
 [0.49801179]
 [0.05467904]] 
 Loss: -0.0019759418771202446 
For iteration 400 
 Input:[[1. 0. 1.]
 [0. 1. 1.]
[...]
{'fraction_non_alphanumeric': 0.14235043745566328, 'fraction_numerical': 0.09056514542445022, 'mean_word_length': 2.9385474860335195, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '646433', 'license': 'Unlicense', 'n_tokens_mistral': 1749, 'n_tokens_neox': 1496, 'n_words': 465, 'path': 'Neural_Network.ipynb', 'repo_name': 'klampropoulos/ML_from_scratch', 'size': '4229'}