<jupyter_start><jupyter_text>Neural Network BasicsWe can try to build a neural network from scratch to see the details of how it works. We won't be using these homemade networks for real purposes, but it is good to get a bit of a look at the mechanics to help make things make sense. Scroll down to the diabetes section to look at what we are going to create to solve a problem that we are familiar with - logistic regression.   Helpers - Base Layer and Support FunctionsFirst we need a few helper functions - separating these out will make our code much easier to read. Below we have: Base Layer Class: we are going to have two types of layers - the "normal" fully connected one, and an activation one to apply the activation function. In the future the activation is wrapped into the normal layer, but this will help us see the parts more clearly.  Activation Functions and Derivitives: we have two activation functions, and the derivitive of each. We'll talk about different activation functions in more detail later on.  Loss Function and Derivitive: we have the loss function and its derivitive. We'll keep it simple and use MSE. We'll also talk more about different loss functions later on.  Convert to Bool: this just translates probability predictions [0,1] into a binary classification. We just need it to calculate predicted accuracy of our test data predictions.<jupyter_code>class Layer:
    def __init__(self):
        self.input = None
        self.output = None

    # computes the output Y of a layer for a given input X
    def forward_propagation(self, input):
        raise NotImplementedError

    # computes dE/dX for a given dE/dY (and update parameters if any)
    def backward_propagation(self, output_error, learning_rate):
        raise NotImplementedError
# activation function and its derivative
def tanh(x):
    return np.tanh(x)

def tanh_prime(x):
    return 1-np.tanh(x)**2
# activation function and its derivative
def sigmoid(z):
    s = 1 / (1 + np.exp(-z))
    return s

def sigmoid_prime(z):
  fz = sigmoid(z)
  return fz * (1 - fz)
# loss function and its derivative
def mse(y_true, y_pred):
    return np.mean(np.power(y_true-y_pred, 2))

def mse_prime(y_true, y_pred):
    return 2*(y_pred-y_true)/y_true.size
# Convert a decimal between 0 and 1 to either 0 or 1, based on if it is
# above or below the cutoff. 
def conv_to_bool(float_list, cutoff=.5):
    new_list = []
    for i in float_list:
        if i < cutoff:
            new_list.append(0)
        else:
            new_list.append(1)
    return new_list<jupyter_output><empty_output><jupyter_text>Usable LayersWe can create the layers that we'll actually use now. Normally the FClayer and the activation layer we are creating would be combined, but this makes it easier to explore. Fully Connected Layer (Dense)The main component of the neural network is the fully connected, or dense, layer. The fully connected (dense) part just means that each neuron is connected to each neuron in the next layer. The three functions each have simple jobs: Initialization: set up the matrix of weights and a vector of bias values. Here they are initialized to a random bumber, in practice these are initialized using some smart method, configurable with a parameter. Each neuron connects to each other neuron, so there is a wight for each of those connections - a matrix of input size by output size. Each output has one bias value.  Forward Propagation: in forward propagation this layer generates predictions by multiplying weights * values and adding the bias. The calculation itself is a dot product - which multiplies each input by its corresponding weight in the matrix automatically. This could be done by lots of loops, but this is more compact and efficient. Recall that each connection has a weight, so we end up with this large matrix.  Backward Propagation: this layer calculates the impact of the error stemming from each input X. This is done by calculating the gradient of the loss with respect to each input - thus allowing us to attribute portions of the output error to the different inputs. The calculation is done using the derivitive of the activation function and some math called the chain rule. Once we have this, we update all the weights and the bias so that we shrink our loss. ![Weights](images/weights.png "Weights" )<jupyter_code># inherit from base class Layer
class FCLayer(Layer):
    # input_size = number of input neurons
    # output_size = number of output neurons
    def __init__(self, input_size, output_size):
        self.weights = np.random.rand(input_size, output_size) - 0.5
        self.bias = np.random.rand(1, output_size) - 0.5

    # returns output for a given input
    def forward_propagation(self, input_data):
        self.input = input_data
        self.output = np.dot(self.input, self.weights) + self.bias
        return self.output

    # computes dE/dW, dE/dB for a given output_error=dE/dY. Returns input_error=dE/dX.
    def backward_propagation(self, output_error, learning_rate):
        input_error = np.dot(output_error, self.weights.T)
        weights_error = np.dot(self.input.T, output_error)
        
        # update parameters
        # dBias = output_error
        self.weights -= learning_rate * weights_error
        self.bias -= learning_rate * output_error

        return input_error<jupyter_output><empty_output><jupyter_text>Activation LayerThe activation layer is more simple, it just applies the activation function. Normally this is built into the other layer, but this is a little more simple to build by hand.  Initialization: set the activation function and derivitive to use going forward.  Forward Propagation: in forward propagation the activation layer just takes the input that it gets and applies the activation function to it.   Backward Propagation: in backward propagation the activation layer translates the error of the predictions back up by multiplying the error by the derivitive of the activation function. This has the effect of translating the error we got with respect to the output of the activation function into error with respect to the input of the activation function.<jupyter_code># inherit from base class Layer
class ActivationLayer(Layer):
    def __init__(self, activation, activation_prime):
        self.activation = activation
        self.activation_prime = activation_prime

    # returns the activated input
    def forward_propagation(self, input_data):
        self.input = input_data
        self.output = self.activation(self.input)
        return self.output

    # Returns input_error=dE/dX for a given output_error=dE/dY.
    # learning_rate is not used because there is no "learnable" parameters.
    def backward_propagation(self, output_error, learning_rate):
        return self.activation_prime(self.input) * output_error<jupyter_output><empty_output><jupyter_text>Full NetworkWe can now build our layers into an actual model. Our code here is a little long, but most of it is pretty simple - the majority of the extra stuff is to make the model flexible.  Add - call this to add layers to the model.  Use - provide a loss function, and the derivitive of a loss function for the model to use.  Predict - generate prediction. This is just one execution of a forward propagation for the new inputs.  Fit - train the model.          Each iteration is called an epoch. We loop through the process until we've hit the desired number of epochs.      Run each sample through a forward propagation.      Get the prediction, and calculate the error.      Pass the error to the back propagation. Repeat    Another way to phrase the fitting and optimizing of the model is to think of the errors on each epoch. First we generate a prediction (FP), and find the error of that prediction by simply comparing it to the true value - this is the error with respect to y, the output. Next we do the back propagation, this takes that error with respect to y that we figured out, and "breaks" that error down into error with respect to each term from the previous layer. Recall that one neuron creates a prediction (during FP) that is equal to w1*x1 + w2*x2 +... b; this does a sort of reverse execution of that - starting with the overall error and mapping it to each term. The error is effectively split into parts and each of the m*x terms and the b term is labeled as being accountable for some of it. Since the x values are the inputs, they can't change, so we modify the weights and bias to lessen the error - this is the gradient descent part. This gets repeated back through each layer until the first layer, then another FP begins with the new weights. The learning rate controls the size of the adjustments. This back propagation part is the key to the high accuracy ceiling that neural networks have when the data is large - we can tune each of many neurons specifically to their contribution of error. If we have enough data, we can get very accurate predictions.<jupyter_code>class Network:
    def __init__(self):
        self.layers = []
        self.loss = None
        self.loss_prime = None

    # add layer to network
    def add(self, layer):
        self.layers.append(layer)

    # set loss to use
    def use(self, loss, loss_prime):
        self.loss = loss
        self.loss_prime = loss_prime

    # predict output for given input
    def predict(self, input_data):
        # sample dimension first
        samples = len(input_data)
        result = []

        # run network over all samples
        for i in range(samples):
            # forward propagation
            output = input_data[i]
            for layer in self.layers:
                output = layer.forward_propagation(output)
            result.append(output)

        return result

    # train the network
    def fit(self, x_train, y_train, epochs, learning_rate):
        # sample dimension first
        samples = len(x_train)

        # training loop
        errors = []
        for i in range(epochs):
            err = 0
            for j in range(samples):
                # forward propagation
                output = x_train[j]
                for layer in self.layers:
                    output = layer.forward_propagation(output)

                # compute loss (for display purpose only)
                #print(self.loss)
                err += self.loss(y_train[j], output)

                # backward propagation
                error = self.loss_prime(y_train[j], output)
                for layer in reversed(self.layers):
                    error = layer.backward_propagation(error, learning_rate)

            # calculate average error on all samples
            if i % 100 == 0:
                err /= samples
                errors.append(err)
                print('epoch %d/%d   error=%f' % (i+1, epochs, err))<jupyter_output><empty_output><jupyter_text>Use NetworkWe have a neural network now - we need to use it! Since we were smart and made the framework able to handle input data of an arbitrary size, we can use our model for pretty much any application!  Diabetes DataOne weird thing that we will do to the data here is that we will reshape the Xs into a 3 dimensional array. This is due to the structure of the model - it is expecting data to be in that shape. When we start using Keras (the package we use for neural network models) next week this can generally be handled automatically, but since we've made this from scratch, we need to do it here. What we end up with for the data is: 8 numerical features. We will have 8 neurons / Xs in the input of our network.  1 binary categorical output. We will have one neuron for the output of the network.  Activation function applied to our output, to ensure we get categorization. ![SimpleNN](images/simple_nn.png "SimpleNN" )<jupyter_code>df = pd.read_csv("data/diabetes.csv")
#df.head()

y = np.array(df["Outcome"]).reshape(-1,1)
X = df.drop(columns={"Outcome"})

X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = MinMaxScaler()
X_train = scaler.fit_transform(X_train).reshape(-1,1,8)
X_test = scaler.transform(X_test).reshape(-1,1,8)

#Look at some details
print ("train_set_x shape: " + str(X_train.shape))
print ("train_set_y shape: " + str(y_train.shape))
print ("test_set_x shape: " + str(X_test.shape))
print ("test_set_y shape: " + str(y_test.shape))
print(X_test[0])
print(y_test[0])
print(X_train[0])
print(y_train[0])
df.head()<jupyter_output><empty_output><jupyter_text>Implement Simple NNWe can create the network that we defined above.<jupyter_code># Network
net = Network()
net.add(FCLayer(8, 1))
net.add(ActivationLayer(sigmoid, sigmoid_prime))

# Train
net.use(mse, mse_prime)
net.fit(X_train, y_train, epochs=1000, learning_rate=0.01)

# Evaluate on test set
out = net.predict(X_test)
pred_labels = conv_to_bool(out)
print(accuracy_score(y_test, pred_labels))<jupyter_output>epoch 1/1000   error=0.265747
epoch 101/1000   error=0.169417
epoch 201/1000   error=0.160810
epoch 301/1000   error=0.157499
epoch 401/1000   error=0.155846
epoch 501/1000   error=0.154910
epoch 601/1000   error=0.154338
epoch 701/1000   error=0.153969
epoch 801/1000   error=0.153722
epoch 901/1000   error=0.153551
0.75<jupyter_text>Try a Larger NetworkWhat if we try adding some layers? Outside of the input and output the rest of the stuff inside our network is configurable. We can add whatever we want in terms of layer, and each layer can be any size, as long as the shape matches the previous and next layer. We can also try that other activation function if we want. What's the "right" size? That question doesn't have a direct answer, we'll look at some things we can use to estimate it ~ 2 workbooks from now. We can play around a bit with it now and see what we get.<jupyter_code># Network
net = Network()
net.add(FCLayer(8, 50))            
net.add(ActivationLayer(sigmoid, sigmoid_prime))
net.add(FCLayer(50, 25))               
net.add(ActivationLayer(sigmoid, sigmoid_prime))
net.add(FCLayer(25, 1))  
net.add(ActivationLayer(sigmoid, sigmoid_prime))

# Train
net.use(mse, mse_prime)
net.fit(X_train, y_train, epochs=1000, learning_rate=0.1)

# test on 3 samples
out = net.predict(X_test)
pred_labels = conv_to_bool(out)

print(accuracy_score(y_test, pred_labels))<jupyter_output>epoch 1/1000   error=0.237428
epoch 101/1000   error=0.153672
epoch 201/1000   error=0.147243
epoch 301/1000   error=0.144862
epoch 401/1000   error=0.143339
epoch 501/1000   error=0.141828
epoch 601/1000   error=0.140173
epoch 701/1000   error=0.138757
epoch 801/1000   error=0.137398
epoch 901/1000   error=0.135733
0.734375<jupyter_text>XORWe can test and see what our network can do with a very simple trial. XOR is a logical operation - exclusive or. It is 1 only if exactly one input is 1, or else it is false. We have that data as the training X and y, we can see if our network can learn this very simple relationship before we test it on some real data.<jupyter_code># training data
x_train = np.array([[[0,0]], [[0,1]], [[1,0]], [[1,1]]])
y_train = np.array([[[0]], [[1]], [[1]], [[0]]])

# network
net = Network()
net.add(FCLayer(2, 3))
net.add(ActivationLayer(tanh, tanh_prime))
net.add(FCLayer(3, 1))
net.add(ActivationLayer(tanh, tanh_prime))

# train
net.use(mse, mse_prime)
net.fit(x_train, y_train, epochs=1000, learning_rate=0.1)

# test
out = net.predict(x_train)
print(out)<jupyter_output>epoch 1/1000   error=0.786096
epoch 101/1000   error=0.282364
epoch 201/1000   error=0.280512
epoch 301/1000   error=0.279865
epoch 401/1000   error=0.279410
epoch 501/1000   error=0.279194
epoch 601/1000   error=0.279084
epoch 701/1000   error=0.279020
epoch 801/1000   error=0.278976
epoch 901/1000   error=0.278944
[array([[0.52469271]]), array([[0.52743519]]), array([[0.51453628]]), array([[0.51732794]])]<jupyter_text>MNIST ImagesNow that we are at least a little comfortable with the usage of neural networks, we can start to apply it to more useful applications, such as our old friends, the MNIST digits. Here we will make a larger model and we can also try a differnet activation function. One other difference to note is the "to_categorical" applied to the target y values. This works with the final layer of 10 output neurons - we get a softmax type of output where there is a probability for being in each class.<jupyter_code># load MNIST from server
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# training data : 60000 samples
# reshape and normalize input data
x_train = x_train.reshape(x_train.shape[0], 1, 28*28)
x_train = x_train.astype('float32')
x_train /= 255
# encode output which is a number in range [0,9] into a vector of size 10
# e.g. number 3 will become [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
y_train = np_utils.to_categorical(y_train)

# same for test data : 10000 samples
x_test = x_test.reshape(x_test.shape[0], 1, 28*28)
x_test = x_test.astype('float32')
x_test /= 255
y_test = np_utils.to_categorical(y_test)

print(x_train.shape)

# Network
net = Network()
net.add(FCLayer(28*28, 100))                # input_shape=(1, 28*28)    ;   output_shape=(1, 100)
net.add(ActivationLayer(tanh, tanh_prime))
net.add(FCLayer(100, 50))                   # input_shape=(1, 100)      ;   output_shape=(1, 50)
net.add(ActivationLayer(tanh, tanh_prime))
net.add(FCLayer(50, 10))                    # input_shape=(1, 50)       ;   output_shape=(1, 10)
net.add(ActivationLayer(tanh, tanh_prime))

# train on 1000 samples
# we didn't use batches, which we'll look at more next time, so we can't use too much data or it will be slow. 
net.use(mse, mse_prime)
net.fit(x_train[0:1000], y_train[0:1000], epochs=1000, learning_rate=0.01)

# test on 3 samples
out = net.predict(x_test[0:3])
print("\n")
print("predicted values : ")
print(out, end="\n")
print("true values : ")
print(y_test[0:3])<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
11501568/11490434 [==============================] - 0s 0us/step
(60000, 1, 784)
epoch 1/1000   error=0.469412
epoch 101/1000   error=0.016236
epoch 201/1000   error=0.009168
epoch 301/1000   error=0.006269
epoch 401/1000   error=0.004776
epoch 501/1000   error=0.003887
epoch 601/1000   error=0.003340
epoch 701/1000   error=0.002738
epoch 801/1000   error=0.002470
epoch 901/1000   error=0.002225


predicted values : 
[array([[ 0.00206888, -0.009769  , -0.01856826, -0.00982643,  0.00306083,
        -0.38077274,  0.00235521,  0.95269863,  0.44518058,  0.1162439 ]]), array([[ 0.95436756,  0.05620731,  0.20887065,  0.09166356,  0.05059499,
        -0.59853293,  0.05668743, -0.15779277,  0.01203375,  0.21089159]]), array([[ 5.02228215e-03,  9.87173158e-01,  5.34966353e-02,
        -4.15743061e-04, -2.94559515e-03,  1.00020798e-01,
     [...]
{'fraction_non_alphanumeric': 0.08256295824035703, 'fraction_numerical': 0.0582828604824142, 'mean_word_length': 3.645360315893386, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '50332', 'license': 'MIT', 'n_tokens_mistral': 6378, 'n_tokens_neox': 5602, 'n_words': 2291, 'path': '021_nn_scratch_merged_sol.ipynb', 'repo_name': 'allyryry/3950_Workbooks', 'size': '18822'}