<jupyter_start><jupyter_text>Backpropagation Practice *Data Science Unit 4 Sprint 2 Assignment 2*Implement a 3 input, 4 node hidden-layer, 1 output node Multilayer Perceptron on the following dataset:| x1 | x2 | x3 | y ||----|----|----|---|| 0  | 0  | 1  | 0 || 0  | 1  | 1  | 1 || 1  | 0  | 1  | 1 || 0  | 1  | 0  | 1 || 1  | 0  | 0  | 1 || 1  | 1  | 1  | 0 || 0  | 0  | 0  | 0 |If you look at the data you'll notice that the first two columns behave like an XOR gate while the last column is mostly just noise. Remember that creating an XOR gate was what the perceptron was criticized for not being able to learn.<jupyter_code>import numpy as np

np.random.seed(42)
# 3 input nodes
X = np.array(([0,0,1], [0,1,1], [1,0,1], [0,1,0], [1,0,0], [1,1,1], [0,0,0]), dtype=float)
# 1 output node
y = np.array(([0], [1], [1], [1], [1], [0], [0]), dtype=float)

print("3 inputs \n", X)
print("1 output \n", y)
class NeuralNetwork:
    def __init__(self):
        # Set up Architecture of Neural Network
        self.inputs = 3
        self.hiddenNodes = 4
        self.outputNodes = 1
        
        # Initial Weights
        # 3x4 Matrix Array for the First Layer
        self.weights1 = np.random.randn(self.inputs, self.hiddenNodes)
        # 4x1 Matrix Array for Hidden to Output
        self.weights2 = np.random.rand(self.hiddenNodes, self.outputNodes)
        
    def sigmoid(self, s):
        return 1 / (1 + np.exp(-s))
    
    def sigmoidPrime(self, s):
        return s * (1 - s)
    
    def feed_forward(self, X):
        """
        Calculate the NN inference using feed forward.
        aka "predict"
        """
        
        # Weighted sum of inputs => hidden layer
        self.hidden_sum = np.dot(X, self.weights1)
        
        # Activations of weighted sum
        self.activated_hidden = self.sigmoid(self.hidden_sum)
        
        # Weight sum between hidden and output
        self.output_sum = np.dot(self.activated_hidden, self.weights2)
        
        # Final activation of output
        self.activated_output = self.sigmoid(self.output_sum)
        
        return self.activated_output
    def backward(self, X, y, o):
        """
        Backward propagate through the network
        """
        
        # Error in output
        self.o_error = y - o
        
        # Apply derivative of sigmoid to error
        # How far off are we in relation to the Sigmoid f(x) of the output
        # ^- hidden => output
        self.o_delta = self.o_error * self.sigmoidPrime(o)
        
        # z2 error
        self.z2_error = self.o_delta.dot(self.weights2.T)
        # How much of that "far off" can be explained by the inputs => hidden layer
        self.z2_delta = self.z2_error * self.sigmoidPrime(self.activated_hidden)
        
        # Adjustment to first set of weights (input => hidden)
        self.weights1 += X.T.dot(self.z2_delta)
        
        # Adjustment to second set of weights (hidden => output)
        self.weights2 += self.activated_hidden.T.dot(self.o_delta)
        
    def train(self, X, y):
        o = self.feed_forward(X)
        self.backward(X, y, o)
# Train the NN
nn = NeuralNetwork()

# number of epochs / iterations
for i in range(1000):
    if (i+1 in [1,2,3]) or ((i+1) % 50 ==0):
        print('+' + '---' * 3 + f'EPOCH {i+1}' + '---'*3 + '+')
#         print('Input: \n', X)
#         print('Actual Output: \n', y)
#         print('Predicted Output: \n', str(nn.feed_forward(X)))
        loss = np.mean(np.square(y - nn.feed_forward(X)))
        if loss < .01:
            print('Input: \n', X)
            print('Actual Output: \n', y)
            print('Predicted Output: \n', str(nn.feed_forward(X)))
            print("Loss: \n", str(loss))
            break
        print("Loss: \n", str(loss))
    nn.train(X,y)<jupyter_output>+---------EPOCH 1---------+
Loss: 
 0.2561291968116666
+---------EPOCH 2---------+
Loss: 
 0.2466265808421276
+---------EPOCH 3---------+
Loss: 
 0.24289794519365643
+---------EPOCH 50---------+
Loss: 
 0.20281930763902123
+---------EPOCH 100---------+
Loss: 
 0.1553469025730411
+---------EPOCH 150---------+
Loss: 
 0.12041913432654865
+---------EPOCH 200---------+
Loss: 
 0.08156330551821807
+---------EPOCH 250---------+
Loss: 
 0.035407244157710314
+---------EPOCH 300---------+
Loss: 
 0.017704995669713793
+---------EPOCH 350---------+
Loss: 
 0.011616157918231248
+---------EPOCH 400---------+
Input: 
 [[0. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [0. 1. 0.]
 [1. 0. 0.]
 [1. 1. 1.]
 [0. 0. 0.]]
Actual Output: 
 [[0.]
 [1.]
 [1.]
 [1.]
 [1.]
 [0.]
 [0.]]
Predicted Output: 
 [[0.02969411]
 [0.91491324]
 [0.90452497]
 [0.92326236]
 [0.94066262]
 [0.09990983]
 [0.15520948]]
Loss: 
 0.008674074935548483<jupyter_text>Try building/training a more complex MLP on a bigger dataset.Use the [MNIST dataset](http://yann.lecun.com/exdb/mnist/) to build the cannonical handwriting digit recognizer and see what kind of accuracy you can achieve. If you need inspiration, the internet is chalk-full of tutorials, but I want you to see how far you can get on your own first. I've linked to the original MNIST dataset above but it will probably be easier to download data through a neural network library. If you reference outside resources make sure you understand every line of code that you're using from other sources, and share with your fellow students helpful resources that you find. Parts1. Gathering & Transforming the Data2. Making MNIST a Binary Problem3. Estimating your Neural Network (the part you focus on)  Gathering the Data `keras` has a handy method to pull the mnist dataset for you. You'll notice that each observation is a 28x28 arrary which represents an image. Although most Neural Network frameworks can handle higher dimensional data, that is more overhead than necessary for us. We need to flatten the image to one long row which will be 784 values (28X28). Basically, you will be appending each row to one another to make on really long row.<jupyter_code>import numpy as np
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
# input image dimensions
img_rows, img_cols = 28, 28
# the data, split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(x_train.shape[0], img_rows * img_cols)
X_test = X_test.reshape(x_test.shape[0], img_rows * img_cols)

# Normalize Our Data
X_train = x_train / 255
X_test = x_test / 255
# Now the data should be in a format you're more familiar with
X_train.shape<jupyter_output><empty_output><jupyter_text>Making MNIST a Binary Problem MNIST is multiclass classification problem; however we haven't covered all the necessary techniques to handle this yet. You would need to one-hot encode the target, use a different loss metric, and use softmax activations for the last layer. This is all stuff we'll cover later this week, but let us simply the problem for now: Zero or all else.<jupyter_code>import numpy as np

y_temp = np.zeros(y_train.shape)
y_temp[np.where(y_train == 0.0)[0]] = 1
y_train = y_temp

y_temp = np.zeros(y_test.shape)
y_temp[np.where(y_test == 0.0)[0]] = 1
y_test = y_temp
# A Nice Binary target for ya to work with
y_train.shape<jupyter_output><empty_output><jupyter_text>Estimating Your `net<jupyter_code>for i in range(2000):
    if (i+1 in [1,2,3]) or ((i+1) % 50 ==0):
        print('+' + '---' * 3 + f'EPOCH {i+1}' + '---'*3 + '+')
#         print('Input: \n', X)
#         print('Actual Output: \n', y)
#         print('Predicted Output: \n', str(nn.feed_forward(X)))
        loss = np.mean(np.square(y - nn.feed_forward(X)))
        if loss < .001:
            print('Input: \n', X)
            print('Actual Output: \n', y)
            print('Predicted Output: \n', str(nn.feed_forward(X)))
            print("Loss: \n", str(loss))
            break
        print("Loss: \n", str(loss))
    nn.train(X_train,y_train)<jupyter_output>+---------EPOCH 1---------+
Input: 
 [[0. 0. 1.]
 [0. 1. 1.]
 [1. 0. 1.]
 [0. 1. 0.]
 [1. 0. 0.]
 [1. 1. 1.]
 [0. 0. 0.]]
Actual Output: 
 [[0.]
 [1.]
 [1.]
 [1.]
 [1.]
 [0.]
 [0.]]
Predicted Output: 
 [[0.00898785]
 [0.97474989]
 [0.96667986]
 [0.97955847]
 [0.97092287]
 [0.03860541]
 [0.04826252]]
Loss: 
 0.0009873664795659706
{'fraction_non_alphanumeric': 0.1499515503875969, 'fraction_numerical': 0.07146317829457365, 'mean_word_length': 2.4780960404380794, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '510980', 'license': 'MIT', 'n_tokens_mistral': 3235, 'n_tokens_neox': 2813, 'n_words': 919, 'path': 'curriculum/unit-4-machine-learning/sprint-2-intro-nn/module2-backpropagation/LS_DS_422_Backprop_Assignment.ipynb', 'repo_name': 'BrianThomasRoss/lambda-school', 'size': '8256'}