<jupyter_start><jupyter_text>Backpropagation InstructionsWe will train a neural network to draw a curve.The curve takes one input variable, the amount travelled along the curve from 0 to 1, and returns 2 outputs, the 2D coordinates of the position of points on the curve.To help capture the complexity of the curve, we shall use two hidden layers in our network with 6 and 7 neurons respectively.![nn](img/nn_example_graph.png)We will calculate the Jacobian of the cost function, with respect to the weights and biases of the network. It will form part of a stochastic steepest descent algorithm that will train the network.  Feed forwardIn the following cell, we will define functions to set up our neural network.Namely an activation function, $\sigma(z)$, it's derivative, $\sigma'(z)$, a function to initialise weights and biases, and a function that calculates each activation of the network using feed-forward.Recall the feed-forward equations,$$ \mathbf{a}^{(n)} = \sigma(\mathbf{z}^{(n)}) $$$$ \mathbf{z}^{(n)} = \mathbf{W}^{(n)}\mathbf{a}^{(n-1)} + \mathbf{b}^{(n)} $$In this worksheet we will use the *logistic function* as our activation function, rather than the more familiar $\tanh$.$$ \sigma(\mathbf{z}) = \frac{1}{1 + \exp(-\mathbf{z})} $$<jupyter_code>import numpy as np
import matplotlib.pyplot as pl
# the activation function and its derivative.
sigma = lambda z : 1 / (1 + np.exp(-z))
d_sigma = lambda z : np.cosh(z/2)**(-2) / 4

# initialise the network with it's structure
def reset_network (n1 = 6, n2 = 7, random=np.random) :
    global W1, W2, W3, b1, b2, b3
    W1 = random.randn(n1, 1) / 2
    W2 = random.randn(n2, n1) / 2
    W3 = random.randn(2, n2) / 2
    b1 = random.randn(n1, 1) / 2
    b2 = random.randn(n2, 1) / 2
    b3 = random.randn(2, 1) / 2

# feed forward each activation to the next layer and return all weighted sums and activations.
def network_function(a0) :
    z1 = W1 @ a0 + b1
    a1 = sigma(z1)
    z2 = W2 @ a1 + b2
    a2 = sigma(z2)
    z3 = W3 @ a2 + b3
    a3 = sigma(z3)
    return a0, z1, a1, z2, a2, z3, a3

# Cost function of a neural network with respect to a training set.
def cost(x, y) :
    return np.linalg.norm(network_function(x)[-1] - y)**2 / x.size<jupyter_output><empty_output><jupyter_text>BackpropagationStarting with layer 3, which is the easiest, and work backwards through the layers we derive the Jacobian of the cost function with respect to the weights and biases.Jacobians:,$$ \mathbf{J}_{\mathbf{W}^{(3)}} = \frac{\partial C}{\partial \mathbf{W}^{(3)}} $$$$ \mathbf{J}_{\mathbf{b}^{(3)}} = \frac{\partial C}{\partial \mathbf{b}^{(3)}} $$etc., where $C$ is the average cost function over the training set. i.e.,$$ C = \frac{1}{N}\sum_k C_k $$for the weight:$$ \frac{\partial C}{\partial \mathbf{W}^{(3)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{z}^{(3)}}   \frac{\partial \mathbf{z}^{(3)}}{\partial \mathbf{W}^{(3)}}   ,$$for the bias:$$ \frac{\partial C}{\partial \mathbf{b}^{(3)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{z}^{(3)}}   \frac{\partial \mathbf{z}^{(3)}}{\partial \mathbf{b}^{(3)}}   .$$partial derivatives:$$ \frac{\partial C}{\partial \mathbf{a}^{(3)}} = 2(\mathbf{a}^{(3)} - \mathbf{y}) $$$$ \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{z}^{(3)}} = \sigma'({z}^{(3)})$$$$ \frac{\partial \mathbf{z}^{(3)}}{\partial \mathbf{W}^{(3)}} = \mathbf{a}^{(2)}$$$$ \frac{\partial \mathbf{z}^{(3)}}{\partial \mathbf{b}^{(3)}} = 1$$<jupyter_code># Jacobian for the third layer weights:
def J_W3 (x, y) :
    # First get all the activations and weighted sums at each layer of the network.
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    # We'll use the variable J to store parts of our result as we go along, updating it in each line.
    # Firstly, we calculate dC/da3, using the expressions above.
    J = 2 * (a3 - y)
    # Next multiply the result we've calculated by the derivative of sigma, evaluated at z3.
    J = J * d_sigma(z3)
    # Then we take the dot product (along the axis that holds the training examples) with the final partial derivative,
    # i.e. dz3/dW3 = a2
    # and divide by the number of training examples, for the average over all training examples.
    J = J @ a2.T / x.size
    return J

# Jacobian for the bias.
def J_b3 (x, y) :
    # set up the activations.
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    # the first two partial derivatives of the Jacobian.
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    # we don't need to multiply by dz3/db3, because that is multiplying by 1.
    # Need to sum over all training examples however.
    J = np.sum(J, axis=1, keepdims=True) / x.size
    return J<jupyter_output><empty_output><jupyter_text>We'll nNext do the Jacobian for the Layer 2. The partial derivatives for this are,$$ \frac{\partial C}{\partial \mathbf{W}^{(2)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \left(   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}   \right)   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{z}^{(2)}}   \frac{\partial \mathbf{z}^{(2)}}{\partial \mathbf{W}^{(2)}}   ,$$$$ \frac{\partial C}{\partial \mathbf{b}^{(2)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \left(   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}   \right)   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{z}^{(2)}}   \frac{\partial \mathbf{z}^{(2)}}{\partial \mathbf{b}^{(2)}}   .$$This is very similar to the previous layer, with two exceptions:* There is a new partial derivative, in parentheses, $\frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}$* The terms after the parentheses are now one layer lower.Recall the new partial derivative takes the following form,$$ \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}} =   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{z}^{(3)}}   \frac{\partial \mathbf{z}^{(3)}}{\partial \mathbf{a}^{(2)}} =   \sigma'(\mathbf{z}^{(3)})   \mathbf{W}^{(3)}$$<jupyter_code>def J_W2 (x, y) :
    # the first two lines are identical to in J_W3.
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)    
    J = 2 * (a3 - y)
    # the next two lines implement da3/da2, first σ' and then W3.
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    # then the final lines are the same as in J_W3 but with the layer number bumped down.
    J = J * d_sigma(z2)
    J = J @ a1.T / x.size
    return J

def J_b2 (x, y) :
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    J = J * d_sigma(z2)
    J = np.sum(J, axis=1, keepdims=True) / x.size
    return J<jupyter_output><empty_output><jupyter_text>Layer 1 is very similar to Layer 2, but with an addition partial derivative term.$$ \frac{\partial C}{\partial \mathbf{W}^{(1)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \left(   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{a}^{(1)}}   \right)   \frac{\partial \mathbf{a}^{(1)}}{\partial \mathbf{z}^{(1)}}   \frac{\partial \mathbf{z}^{(1)}}{\partial \mathbf{W}^{(1)}}   ,$$$$ \frac{\partial C}{\partial \mathbf{b}^{(1)}} =   \frac{\partial C}{\partial \mathbf{a}^{(3)}}   \left(   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{a}^{(1)}}   \right)   \frac{\partial \mathbf{a}^{(1)}}{\partial \mathbf{z}^{(1)}}   \frac{\partial \mathbf{z}^{(1)}}{\partial \mathbf{b}^{(1)}}   .$$<jupyter_code>def J_W1 (x, y) :
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    J = J * d_sigma(z2)
    J = (J.T @ W2).T
    J = J * d_sigma(z1)
    J = J @ a0.T / x.size
    return J

def J_b1 (x, y) :
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    J = J * d_sigma(z2)
    J = (J.T @ W2).T
    J = J * d_sigma(z1)
    J = np.sum(J, axis=1, keepdims=True) / x.size
    return J<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19629858112276372, 'fraction_numerical': 0.0310919185687847, 'mean_word_length': 3.5310229178311907, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '660177', 'license': 'MIT', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2813, 'n_words': 913, 'path': 'math4ml/23_backpropagation.ipynb', 'repo_name': 'dpopadic/ml-res', 'size': '8105'}