<jupyter_start><jupyter_text>Week 8 Wednesday[Yuja recording](https://uci.yuja.com/V/Video?v=4446584&node=14938258&a=1900985263&autoplay=1)Before the recording, at the board we went over some different components related to Neural Networks and PyTorch, and especially we went over an example of performing gradient descent.The goal of today's class is to get more comfortable with the various components involved in building and training a neural network using PyTorch.<jupyter_code>import numpy as np
import torch
from torch import nn
from torchvision.transforms import ToTensor<jupyter_output><empty_output><jupyter_text>Gradient descent Gradient descent can be used to try to find a minimum of any differentiable function.  (Often it will only find a local minimum, not a global minimum, even if a global minimum exists.)  We usually use gradient descent for very complicated functions, but here we give an example of performing gradient descent to attempt to find a minimum of the function$$f(x,y) = (x-3)^2 + (y+2)^2 + 8.$$We call this function `loss_fn` so that the syntax is the same as what we're used to in PyTorch.<jupyter_code>loss_fn = lambda t: (t[0] - 3)**2 + (t[1] + 2)**2 + 8<jupyter_output><empty_output><jupyter_text>To perform gradient descent, you need to begin with an initial guess.  We guess (10,10) and then gradually adjust this, hoping to move towards a minimum.  Notice the decimal point after 10... this is a shortcut for telling PyTorch that these should be treated as floats.<jupyter_code>a = torch.tensor([10.,10], requires_grad=True)
a
loss_fn([10,10])
loss_fn(a)
type(loss_fn)<jupyter_output><empty_output><jupyter_text>Because we specified `requires_grad=True` as a keyword argument, we will be able to find gradients of computations involving `a`.  There isn't any gradient yet because we haven't computed one.<jupyter_code>a.grad<jupyter_output><empty_output><jupyter_text>Here we define a stochastic gradient descent optimizer like usual in PyTorch.  The first input is usually something like `model.parameters()`.  Here we try to use `a` as the first argument.  That is almost right, but we need to put it in a list (or some other type of *iterable*).<jupyter_code>optimizer = torch.optim.SGD(a, lr = 0.1)
optimizer = torch.optim.SGD([a], lr = 0.1)
loss = loss_fn(a)<jupyter_output><empty_output><jupyter_text>This next `optimizer.zero_grad()` is not important yet, but it is good to be in the habit, because otherwise multiple gradient computations will accumulate, and we want to start over each time.<jupyter_code>optimizer.zero_grad()
type(loss)<jupyter_output><empty_output><jupyter_text>Next we compute the gradient.  This typically uses an algorithm called *backpropagation*, which is where the name `backward` comes from.<jupyter_code>loss.backward()
a<jupyter_output><empty_output><jupyter_text>Now the `grad` attribute of `a` has a value.  You should be able to compute this value by hand in this case, since our `loss_fn` is so simple.<jupyter_code>a.grad<jupyter_output><empty_output><jupyter_text>Now we replace add a multiple (the learning rate `lr`) of the negative gradient to `a`.  Again, you should be able to compute this by hand in this case.  The formula is$$a \leadsto a - lr \cdot \nabla$$<jupyter_code>optimizer.step()
a
loss_fn = lambda t: (t[0] - 3)**2 + (t[1] + 2)**2 + 8<jupyter_output><empty_output><jupyter_text>Notice how the value of `a` is approaching the minimum (3,-2), and notice how `loss` is approaching the minimum of our `loss_fn`, which is 8.  (The only reason we're using the terms `loss` and `loss_fn` is because those are the terms we usually use in PyTorch.  In this case, `loss_fn` is just an ordinary two-variable function like from Math 2D which we are trying to minimize.)<jupyter_code>epochs = 20
a = torch.tensor([10.,10], requires_grad=True)
optimizer = torch.optim.SGD([a], lr = 0.1)
for i in range(epochs):
    loss = loss_fn(a)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print("Epoch " + str(i))
    print(a)
    print(loss)
    print("")<jupyter_output>Epoch 0
tensor([8.6000, 7.6000], requires_grad=True)
tensor(201., grad_fn=<AddBackward0>)

Epoch 1
tensor([7.4800, 5.6800], requires_grad=True)
tensor(131.5200, grad_fn=<AddBackward0>)

Epoch 2
tensor([6.5840, 4.1440], requires_grad=True)
tensor(87.0528, grad_fn=<AddBackward0>)

Epoch 3
tensor([5.8672, 2.9152], requires_grad=True)
tensor(58.5938, grad_fn=<AddBackward0>)

Epoch 4
tensor([5.2938, 1.9322], requires_grad=True)
tensor(40.3800, grad_fn=<AddBackward0>)

Epoch 5
tensor([4.8350, 1.1457], requires_grad=True)
tensor(28.7232, grad_fn=<AddBackward0>)

Epoch 6
tensor([4.4680, 0.5166], requires_grad=True)
tensor(21.2629, grad_fn=<AddBackward0>)

Epoch 7
tensor([4.1744, 0.0133], requires_grad=True)
tensor(16.4882, grad_fn=<AddBackward0>)

Epoch 8
tensor([ 3.9395, -0.3894], requires_grad=True)
tensor(13.4325, grad_fn=<AddBackward0>)

Epoch 9
tensor([ 3.7516, -0.7115], requires_grad=True)
tensor(11.4768, grad_fn=<AddBackward0>)

Epoch 10
tensor([ 3.6013, -0.9692], requires_grad=True)
te[...]<jupyter_text>If we want `a` to approach the minimum (3,-2) faster, we can make the learning rate bigger, but here is an example of what can go wrong if we make the learning rate too big.<jupyter_code>epochs = 20
a = torch.tensor([10.,10], requires_grad=True)
optimizer = torch.optim.SGD([a], lr = 10)
for i in range(epochs):
    loss = loss_fn(a)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print("Epoch " + str(i))
    print(a)
    print(loss)
    print("")<jupyter_output>Epoch 0
tensor([-130., -230.], requires_grad=True)
tensor(201., grad_fn=<AddBackward0>)

Epoch 1
tensor([2530., 4330.], requires_grad=True)
tensor(69681., grad_fn=<AddBackward0>)

Epoch 2
tensor([-48010., -82310.], requires_grad=True)
tensor(25151960., grad_fn=<AddBackward0>)

Epoch 3
tensor([ 912250., 1563850.], requires_grad=True)
tensor(9.0799e+09, grad_fn=<AddBackward0>)

Epoch 4
tensor([-17332690., -29713190.], requires_grad=True)
tensor(3.2778e+12, grad_fn=<AddBackward0>)

Epoch 5
tensor([3.2932e+08, 5.6455e+08], requires_grad=True)
tensor(1.1833e+15, grad_fn=<AddBackward0>)

Epoch 6
tensor([-6.2571e+09, -1.0726e+10], requires_grad=True)
tensor(4.2717e+17, grad_fn=<AddBackward0>)

Epoch 7
tensor([1.1888e+11, 2.0380e+11], requires_grad=True)
tensor(1.5421e+20, grad_fn=<AddBackward0>)

Epoch 8
tensor([-2.2588e+12, -3.8723e+12], requires_grad=True)
tensor(5.5669e+22, grad_fn=<AddBackward0>)

Epoch 9
tensor([4.2917e+13, 7.3573e+13], requires_grad=True)
tensor(2.0097e+25, grad_fn=<Add[...]<jupyter_text>Here is an example for what seems to be a good choice of `lr`.<jupyter_code>epochs = 20
a = torch.tensor([10.,10], requires_grad=True)
optimizer = torch.optim.SGD([a], lr = 0.25)
for i in range(epochs):
    loss = loss_fn(a)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print("Epoch " + str(i))
    print(a)
    print(loss)
    print("")<jupyter_output>Epoch 0
tensor([6.5000, 4.0000], requires_grad=True)
tensor(201., grad_fn=<AddBackward0>)

Epoch 1
tensor([4.7500, 1.0000], requires_grad=True)
tensor(56.2500, grad_fn=<AddBackward0>)

Epoch 2
tensor([ 3.8750, -0.5000], requires_grad=True)
tensor(20.0625, grad_fn=<AddBackward0>)

Epoch 3
tensor([ 3.4375, -1.2500], requires_grad=True)
tensor(11.0156, grad_fn=<AddBackward0>)

Epoch 4
tensor([ 3.2188, -1.6250], requires_grad=True)
tensor(8.7539, grad_fn=<AddBackward0>)

Epoch 5
tensor([ 3.1094, -1.8125], requires_grad=True)
tensor(8.1885, grad_fn=<AddBackward0>)

Epoch 6
tensor([ 3.0547, -1.9062], requires_grad=True)
tensor(8.0471, grad_fn=<AddBackward0>)

Epoch 7
tensor([ 3.0273, -1.9531], requires_grad=True)
tensor(8.0118, grad_fn=<AddBackward0>)

Epoch 8
tensor([ 3.0137, -1.9766], requires_grad=True)
tensor(8.0029, grad_fn=<AddBackward0>)

Epoch 9
tensor([ 3.0068, -1.9883], requires_grad=True)
tensor(8.0007, grad_fn=<AddBackward0>)

Epoch 10
tensor([ 3.0034, -1.9941], requires_grad=Tru[...]
{'fraction_non_alphanumeric': 0.13199749530369442, 'fraction_numerical': 0.08453350031308704, 'mean_word_length': 5.567434210526316, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 75, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '161901', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3307, 'n_tokens_neox': 2927, 'n_words': 826, 'path': '_build/html/_sources/Week8/Week8-Wednesday.ipynb', 'repo_name': 'ChristopherDavisUCI/UCI-Math-10-W22', 'size': '7985'}