<jupyter_start><jupyter_text>Batch Backpropagation in a Multi-Layer Neural NetworkWe explore backpropagation assuming the batch has two data points with features $\begin{bmatrix}-10 & 20\end{bmatrix}$ and $\begin{bmatrix}8 & -2\end{bmatrix}$ and the correct categories $1$ and $0$ which are one-hot encoded as $\begin{bmatrix}0 & 1\end{bmatrix}$ and $\begin{bmatrix}1 & 0\end{bmatrix}$.The parameters of the multi-layer neural network (the weights and biases) which minimize the loss are discovered by descending the loss gradient.In this exercise, we compute partial derivatives over batches of training data for a multi-layer neural network.<jupyter_code>import torch
import torch.nn.functional as F

weights1 = torch.nn.Parameter(torch.Tensor([[1,1],[2,2]]))
print("Parameters (weights) 1st layer: "+str(weights1.data))

weights2 = torch.nn.Parameter(torch.Tensor([[2, 1],[2,1]]))
print("Parameters (weights) 2nd layer: "+str(weights2.data))

bias1 = torch.nn.Parameter(torch.Tensor([[-1,-1]]))
print("Parameters (bias) 1st layer: "+str(bias1.data))

bias2 = torch.nn.Parameter(torch.Tensor([[5,5]]))
print("Parameters (bias) 2nd layer: "+str(bias2.data))

inputs = torch.autograd.Variable(torch.Tensor([[-10, 20],[8, -2]]))

target = torch.autograd.Variable(torch.LongTensor([1, 0]))
#print(target)

one_hot_target = torch.autograd.Variable(torch.Tensor([[0, 1],[1, 0]]))
#print(one_hot_target)<jupyter_output>Parameters (weights) 1st layer: 
 1  1
 2  2
[torch.FloatTensor of size 2x2]

Parameters (weights) 2nd layer: 
 2  1
 2  1
[torch.FloatTensor of size 2x2]

Parameters (bias) 1st layer: 
-1 -1
[torch.FloatTensor of size 1x2]

Parameters (bias) 2nd layer: 
 5  5
[torch.FloatTensor of size 1x2]<jupyter_text>We assume the weights are $\begin{bmatrix}1 & 1 \\ 2 & 2\end{bmatrix}$ for the lower layer and $\begin{bmatrix}2 & 1 \\ 2 & 1\end{bmatrix}$ for the higher layer at the start.The training is assumed to proceed one data point at a time (batch size of 1).For this iteration of training, the features are $\begin{bmatrix}-10 & 20\end{bmatrix}$ and $\begin{bmatrix}8 & -2\end{bmatrix}$ and the correct categories are $1$ and $0$ which are one-hot encoded as $\begin{bmatrix}0 & 1\end{bmatrix}$ and $\begin{bmatrix}1 & 0\end{bmatrix}$.<jupyter_code># Forward Pass

hidden = torch.mm(inputs, weights1) + bias1
print("a = "+str(hidden))

hidden_relu = F.relu(hidden)
print("h = relu(a) = "+str(hidden_relu))

result = torch.mm(hidden_relu, weights2) + bias2
print("c = "+str(result))

softmax_result = F.softmax(result, dim=1)
print("softmax(c) = "+str(softmax_result))

loss = F.cross_entropy(result, target)
print("Cross entropy loss: "+str(loss))

# Backward Pass

batch_size = inputs.size()[0]
grad_softmax = (softmax_result.data - one_hot_target.data)
grad_hidden_activation = grad_softmax.mm(weights2.data.t())
grad_hidden_pre_activation = grad_hidden_activation.clone()
grad_hidden_pre_activation[hidden.data < 0] = 0
grad_weight2 = hidden.data.t().mm(grad_softmax)
grad_weight2 = grad_weight2/batch_size
grad_bias2 = grad_softmax
grad_bias2 = torch.unsqueeze(torch.sum(grad_bias2, 0), 0)
grad_bias2 = grad_bias2/batch_size
grad_weight1 = inputs.data.t().mm(grad_hidden_pre_activation)
grad_weight1 = grad_weight1/batch_size
grad_bias1 = grad_hidden_pre_activation
grad_bias1 = torch.unsqueeze(torch.sum(grad_bias1, 0), 0)
grad_bias1 = grad_bias1/batch_size

print("MANUALLY COMPUTED PARTIAL DERIVATIVES")

print("\tThe manually computed gradient of the loss with respect to the weights of the second layer is "+str(grad_weight2))

print("\tThe manually computed gradient of the loss with respect to the weights of the first layer is "+str(grad_weight1))

print("\tThe manually computed gradient of the loss with respect to the biases of the second layer is "+str(grad_bias2))

print("\tThe manually computed gradient of the loss with respect to the biases of the first layer is "+str(grad_bias1))

# Autograd

loss.backward()

print("AUTOMATICALLY COMPUTED PARTIAL DERIVATIVES")

gradient = weights2.grad

print("\tThe automatically computed gradient of the loss with respect to the weights of the second layer is "+str(gradient.data))

gradient = weights1.grad

print("\tThe automatically computed gradient of the loss with respect to the weights of the first layer is "+str(gradient.data))

gradient = bias2.grad

print("\tThe automatically computed gradient of the loss with respect to the biases of the second layer is "+str(gradient.data))

gradient = bias1.grad

print("\tThe automatically computed gradient of the loss with respect to the biases of the first layer is "+str(gradient.data))

if weights1.grad is not None:
    weights1.grad.data.zero_()
if weights2.grad is not None:
    weights2.grad.data.zero_()
if bias1.grad is not None:
    bias1.grad.data.zero_()
if bias2.grad is not None:
    bias2.grad.data.zero_()<jupyter_output>a = Variable containing:
 29  29
  3   3
[torch.FloatTensor of size 2x2]

h = relu(a) = Variable containing:
 29  29
  3   3
[torch.FloatTensor of size 2x2]

c = Variable containing:
 121   63
  17   11
[torch.FloatTensor of size 2x2]

softmax(c) = Variable containing:
 1.0000e+00  6.4702e-26
 9.9753e-01  2.4726e-03
[torch.FloatTensor of size 2x2]

Cross entropy loss: Variable containing:
 29.0012
[torch.FloatTensor of size 1]

MANUALLY COMPUTED PARTIAL DERIVATIVES
	The manually computed gradient of the loss with respect to the weights of the second layer is 
 14.4963 -14.4963
 14.4963 -14.4963
[torch.FloatTensor of size 2x2]

	The manually computed gradient of the loss with respect to the weights of the first layer is 
 -5.0099  -5.0099
 10.0025  10.0025
[torch.FloatTensor of size 2x2]

	The manually computed gradient of the loss with respect to the biases of the second layer is 
 0.4988 -0.4988
[torch.FloatTensor of size 1x2]

	The manually computed gradient of the loss with respect [...]
{'fraction_non_alphanumeric': 0.11192504258943782, 'fraction_numerical': 0.04497444633730835, 'mean_word_length': 5.0277207392197125, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '628081', 'license': 'MIT', 'n_tokens_mistral': 2091, 'n_tokens_neox': 1921, 'n_words': 631, 'path': 'exercise_750.ipynb', 'repo_name': 'aiaioo/DeepLearningBasicsTutorial', 'size': '5870'}