<jupyter_start><jupyter_text>Beginning after the matrix multiplication lesson, which has decent notes, ending until the end of training loop (right before callbacks).   Lesson 8  Forward Pass  Imports<jupyter_code># workaround to download mnist data, set root to './', run and proceed, 
!wget www.di.ens.fr/~lelarge/MNIST.tar.gz
!tar -zxvf MNIST.tar.gz

import torchvision
import torchvision.transforms as transforms
root = './'
torchvision.datasets.MNIST(root=root,download=True)
!pip show torchvision
import operator

def test(a,b,cmp,cname=None):
    if cname is None: cname=cmp.__name__
    assert cmp(a,b),f"{cname}:\n{a}\n{b}"

def test_eq(a,b): test(a,b,operator.eq,'==')

from pathlib import Path
from IPython.core.debugger import set_trace
from fastai import datasets
import pickle, gzip, math, torch, matplotlib as mpl
import matplotlib.pyplot as plt
from torch import tensor
import torch.nn.functional as F

def near(a,b): return torch.allclose(a, b, rtol=1e-3, atol=1e-5)
def test_near(a,b): test(a,b,near)

# Make MNIST data work on Google Colab
from six.moves import urllib
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
from torch import nn
%load_ext autoreload
%autoreload 2
%matplotlib inline

def get_data():
    import os
    import torchvision.datasets as datasets
    # root = '../data'

    if not os.path.exists(root):
        os.mkdir(root)
    train_set = datasets.MNIST(root=root, train=True, download=True)
    test_set = datasets.MNIST(root=root, train=False, download=True)
    x_train, x_valid = train_set.train_data.split([50000, 10000])
    y_train, y_valid = train_set.train_labels.split([50000, 10000])
    return (x_train.view(50000, -1) / 256.0), y_train.float(), (x_valid.view(10000, -1))/ 256.0, y_valid.float()

# The geographic intuition for this is picturing x's around a horizontal line (mean), bring that mean down to 0,
# then scale x's by dividing them by the standard deviation
def normalize(x, mean, std): return (x-mean)/std
x_train, y_train, x_valid, y_valid = get_data()<jupyter_output>/usr/local/lib/python3.7/dist-packages/torchvision/datasets/mnist.py:58: UserWarning: train_data has been renamed data
  warnings.warn("train_data has been renamed data")
/usr/local/lib/python3.7/dist-packages/torchvision/datasets/mnist.py:48: UserWarning: train_labels has been renamed targets
  warnings.warn("train_labels has been renamed targets")<jupyter_text>Normalization - We want mean to be 0 and standard deviation to be 1 for easier convergence, so we normalize. - Notice how we use train_mean and train_std to normalize valid data as well - that's because we don't want validation dataset to be in a different scale<jupyter_code># before normalization
train_mean, train_std = x_train.mean(), x_train.std()
train_mean, train_std
x_train = normalize(x_train, train_mean, train_std)
x_valid = normalize(x_valid, train_mean, train_std)
# after normalization
train_mean, train_std = x_train.mean(), x_train.std()
train_mean, train_std
def assert_is_near_zero(a, threshold=1e-3): assert a.abs() < threshold, f"{a} is not near zero"
assert_is_near_zero(train_mean)<jupyter_output><empty_output><jupyter_text>Get shapes<jupyter_code>num_samples, image_size = x_train.shape 
num_classes = y_train.max() + 1 
nh = 50

n = num_samples
m = image_size
c = num_classes

n, m, c, nh<jupyter_output><empty_output><jupyter_text>Intialization - Initialization is *extremely* important. In 2019, they wrote a paper "Fixup Initialization: Residual Learning Without Normalization" in which they trained a 10,000 layer neural net WITHOUT normalization just by initializing everything carefully.  Xavier Initialization- To perform Standard Xavier Initialization, you just divide input by the sqrt(num_input_units), which would give you a mean of 0, and standard deviation of 1 / sqrt(m)<jupyter_code>def lin(x, w, b): return x@w + b
# Forward pass without Initialization
w1 = torch.randn(m, nh)
b1 = torch.zeros(nh)

t = lin(x_valid, w1, b1)
t.mean(), t.std() # terrible, you want ~ (0,1) (mean,std)
# Forward pass with Standard Xavier Init
w1 = torch.randn(m, nh) * math.sqrt(1/m)
b1 = torch.zeros(nh)

t = lin(x_valid, w1, b1)
t.mean(), t.std() # good
assert_is_near_zero(w1.mean())
assert_is_near_zero(w1.std() - 1/math.sqrt(m))<jupyter_output><empty_output><jupyter_text>Vanishing Activation/Gradient Problem- Remember after performing the matrix multiplication, you have to pass it through relu, but each time you do that, you cut all activation values that are below 0 to 0 and thereby reduces the standard deviation. If your network is very deep, your standard deviation will keep getting reduced (possibly down to 0) ![Screen Shot 2021-03-02 at 9.20.03 PM.png](attachment:a3f59b8e-febb-4866-8494-d2338a2b0374.png)<jupyter_code># clamp_min(n) means replace everything below n with n, in this case, relu means replacing everything negative with 0
# always try to use PyTorch function because they're generally implemented in C for you
def relu(x): return x.clamp_min(0.) 

t = relu(lin(x_valid, w1, b1)) 
t.mean(),t.std()<jupyter_output><empty_output><jupyter_text>Kaiming Initialization- The problem with Xavier Initialization is that it doesn't combat the vanishing gradient problem very well. - Kaiming initialization is almost identical to Xavier initialization but with a 2 on top; it will keep the std around $$\text{std} = \sqrt{\frac{2}{(1 + a^2) \times \text{fan_in}}}$$- This was introduced in the paper that described the Imagenet-winning approach from *He et al*: [Delving Deep into Rectifiers](https://arxiv.org/abs/1502.01852), which was also the first paper that claimed "super-human performance" on Imagenet (and, most importantly, it introduced resnets as well as Kaiming He initialization!) - So papers by competition winners are very good because they introduce MANY good ideas instead of just one tiny tweak.<jupyter_code># Forward pass with Kaiming Initialization
torch.manual_seed(42)
w1 = torch.randn(m, nh) * math.sqrt(2/m)
b1 = torch.zeros(nh)

t = relu(lin(x_valid, w1, b1))
t.mean(), t.std() 
# Forward pass with PyTorch's Kaiming Initialization, same thing
from torch.nn.init import kaiming_normal_

w1 = torch.empty(m, nh)
b1 = torch.zeros(nh)

torch.manual_seed(42)
kaiming_normal_(w1, mode='fan_out')
t = relu(lin(x_valid, w1, b1))
t.mean(), t.std()<jupyter_output><empty_output><jupyter_text>- Note: Kaiming initialization is very good but notice the mean is still not zero - we have good reasons to want them to be. So we can define our own new_relu to see if it helps with normalizing the mean. It's an intuitive thing to do and papers are written from these minor tweaks. Maybe it'll help a lot in practice<jupyter_code>def new_relu(x): return x.clamp_min(0.) - 0.5
# The new_relu seems to help!
torch.manual_seed(42)
w1 = torch.randn(m,nh) * math.sqrt(2./m)
t1 = new_relu(lin(x_valid, w1, b1))
t1.mean(), t1.std()<jupyter_output><empty_output><jupyter_text>Train a Model<jupyter_code>torch.manual_seed(42)

w1 = torch.empty(m, nh)
b1 = torch.zeros(nh)
w2 = torch.randn(nh,1)/math.sqrt(nh)
b2 = torch.zeros(1)

def model(x):
    l1 = lin(x, w1, b1)
    l2 = relu(l1)
    l3 = lin(l2, w2, b2)
    return l3
%timeit -n 10 _=model(x_valid)
assert model(x_valid).shape == torch.Size((x_valid.shape[0],1))<jupyter_output><empty_output><jupyter_text>Loss Function- We wrongly use the MSE for now just for simplicity's sake<jupyter_code>def mse(output, targ): return (output.squeeze(-1) - targ).pow(2).mean()
y_train, y_valid = y_train.float(), y_valid.float()
pred = model(x_train)
y_train.shape
pred.shape # not the exact shape, need to squeeze in mse
mse(pred, y_train)<jupyter_output><empty_output><jupyter_text>Backward Pass - During backward pass, you calculate the gradient of every w1, b1, w2, b2 with respect to the loss- For each of the function below, we take the derivative of each layer in terms of loss, storing the result in that layer's .grad -- in other words, x.grad stores the result of dloss/dx. Note x is the denominator, the layer.<jupyter_code>def mse_grad(inp, targ):
    # gradient of loss with respect to the previous layer, so it's pred.grad == dloss/dpred
    inp.grad = 2. * (inp.squeeze() - targ).unsqueeze(-1) / inp.shape[0]
def relu_grad(inp, out):
    # inp.grad == dloss/dinp == dout/dinp * dloss/dout  
    inp.grad = (inp > 0).float() * out.grad
def lin_grad(inp, out, w, b):
    # dloss / dl
    inp.grad = out.grad @ w.t() 
    # dloss / dw
    w.grad = (inp.unsqueeze(-1) * out.grad.unsqueeze(1)).sum(0)
    # dloss / db
    b.grad = out.grad.sum(0)<jupyter_output><empty_output><jupyter_text>Full Pass: Forward + Backward<jupyter_code>from torch.nn import init
torch.manual_seed(42)

# Our forward + backward loop

w1 = torch.zeros(m,nh)
init.kaiming_normal_(w1, mode='fan_out')
b1 = torch.zeros(nh)
w2 = torch.randn(nh,1)/math.sqrt(nh)
b2 = torch.zeros(1)

def forward_and_backward(inp, target):
    # forward
    l1 = lin(inp, w1, b1)
    l2 = relu(l1)
    pred = lin(l2, w2, b2)
    loss = mse(pred, target)    
    
    # backward
    # pred.grad = dloss/dpred
    mse_grad(pred, target) 
    # l2.grad = dloss/dl2 = dloss/dpred * dpred/dl2
    # w2.grad = dloss/dw2 = dloss/dpred * dpred/dw2
    # b2.grad = dloss/db2 = dloss/dpred * dpred/db2
    lin_grad(l2, pred, w2, b2) 
    # l1.grad = dloss/dl1 = dloss/dl2 * dl2/dl1
    relu_grad(l1, l2)
    # x.grad = dloss/dx = dloss/dl1 * dl1/dx
    # w1.grad = dloss/dw1 = dloss/dl1 * dl1/dw1
    # b1.grad = dloss/db1 = dloss/dl1 * dl1/db1
    lin_grad(inp, l1, w1, b1)

forward_and_backward(x_train, y_train)
# pytorch's forward + backward loop
# by setting .requires_grad to True, pytorch's autograd will automatically set the .grad for us 
w1_2 = w1.clone().requires_grad_(True)
w2_2 = w2.clone().requires_grad_(True)
b1_2 = b1.clone().requires_grad_(True)
b2_2 = b2.clone().requires_grad_(True)
x_train_2 = x_train.clone().requires_grad_(True)

def forward(inp, targ):
    # forward pass:
    l1 = lin(inp, w1_2, b1_2)
    l2 = relu(l1)
    pred = lin(l2, w2_2, b2_2)
    # we don't actually need the loss in backward!
    return mse(pred, targ) 

loss = forward(x_train_2, y_train)
loss.backward() 
test_near(w1_2.grad, w1.grad)
test_near(w2_2.grad, w2.grad)
test_near(b1_2.grad, b1.grad)
test_near(b2_2.grad, b2.grad)
test_near(x_train_2.grad, x_train.grad)<jupyter_output><empty_output><jupyter_text>Refactor Forward & Backward Functions into Same Classes<jupyter_code>class Relu():
  def __call__(self, input):
    self.input = input
    self.output = input.clamp_min(0.) 
    return self.output

  def backward(self):
    self.input.grad = (self.input > 0).float() * self.output.grad
class Lin():
  def __init__(self, w, b): 
    self.w = w
    self.b = b

  def __call__(self, input):
    self.input = input
    self.output = input@self.w + self.b
    return self.output
  
  def backward(self):
    self.input.grad = self.output.grad @ self.w.t()
    self.w.grad = (self.input.unsqueeze(-1) * self.output.grad.unsqueeze(1)).sum(0)
    self.b.grad = self.output.grad.sum(0)
class Mse():
  def __call__(self, pred, targ):
    self.pred = pred
    self.targ = targ
    return (pred.squeeze(-1) - targ).pow(2).mean()

  def backward(self):
    self.pred.grad = 2. * (self.pred.squeeze() - self.targ).unsqueeze(-1) / self.pred.shape[0]
class Model:
  def __init__(self, w1, w2, b1, b2):
    self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
    self.loss = Mse()
  
  def __call__(self, x, target):
    for layer in self.layers:
      x = layer(x)
    return self.loss(x, target)
  
  def backward(self):
    self.loss.backward()
    for layer in reversed(self.layers):
      layer.backward()
w1.grad, b1.grad, w2.grad, b2.grad = [None for _ in range(4)]; print(w1.grad, b1.grad, w2.grad, b2.grad)
model = Model(w1, w2, b1, b2)
# forward pass
%time loss = model(x_train, y_train)
# backward pass
%time model.backward()
test_near(w1_2.grad, w1.grad)
test_near(w2_2.grad, w2.grad)
test_near(b1_2.grad, b1.grad)
test_near(b2_2.grad, b2.grad)
test_near(x_train_2.grad, x_train.grad)<jupyter_output><empty_output><jupyter_text>Refactor out Repetitive Code  Notice how we're basically recreating nn.Linear and nn.Module etc.<jupyter_code>class Module():
  def __call__(self, *args):
    self.args = args
    self.output = self.forward(*args)
    return self.output

  def backward(self): self.bwd(self.output, *self.args)
'''
Module is the one that takes in all the arguments during dumb call __call__ and pass them down to the 
specific forward / bwd. There are no self.input, only self.args and self.output.
'''

# Relu(input), *args is input
class Relu(Module):
  def forward(self, input):
    return input.clamp_min(0.) 

  def bwd(self, output, input):
    input.grad = (input > 0).float() * output.grad
# Lin(input), *args is input
class Lin(Module):
  def __init__(self, w, b): 
    self.w, self.b = w, b

  def forward(self, input):
    return input@self.w + self.b
  
  def bwd(self, output, input):
    input.grad = output.grad @ self.w.t()
    # optimization via matrix multiplication instead of multiplication and summing (einsum could also be used to improve performance here)
    self.w.grad = input.t() @ output.grad
    self.b.grad = output.grad.sum(0)
    # print("input, w, b shapes: ", input.shape, self.w.shape, self.b.shape)
    # print("input.grad, w.grad, b.grad shapes: ", input.grad.shape, self.w.grad.shape, self.b.grad.shape)
# Mse(input), *args are pred and targ
class Mse(Module):
  def forward(self, pred, targ):
    return (pred.squeeze(-1) - targ).pow(2).mean()

  def bwd(self, out, pred, targ):
    pred.grad = 2. * (pred.squeeze() - targ).unsqueeze(-1) / pred.shape[0]

class Model:
  def __init__(self, w1, w2, b1, b2):
    self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
    self.loss = Mse()
  
  def __call__(self, x, target):
    for layer in self.layers:
      x = layer(x)
    return self.loss(x, target)
  
  def backward(self):
    self.loss.backward()
    for layer in reversed(self.layers):
      layer.backward()
w1.grad, b1.grad, w2.grad, b2.grad = [None for _ in range(4)]; print(w1.grad, b1.grad, w2.grad, b2.grad)
model = Model(w1, w2, b1, b2)
# forward pass
%time loss = model(x_train, y_train)
# backward pass
%time model.backward()
test_near(w1_2.grad, w1.grad)
test_near(w2_2.grad, w2.grad)
test_near(b1_2.grad, b1.grad)
test_near(b2_2.grad, b2.grad)
test_near(x_train_2.grad, x_train.grad)<jupyter_output><empty_output><jupyter_text>Lesson 9  How to do Research - If you see something weird in PyTorch code, don't assume it's correct. - When it comes to deep learning, no one knows what he's doing. And also a lot of the code is just that way because of legacy and no longer a good idea. It's important to question the status quo.- Re-implement from scratch, play with it, do experiments with it to see if it actually gives better results. (example of what Jeremy did in 02a and 02b notebooks) If not, reach out to the team and ask why. You can publish your results by converting your Jupyter Notebook into Gist with Gist It.   Cross Entropy Loss<jupyter_code># use nn.Linear and don't write how to deal with loss into the model yet
class Model(nn.Module):
  def __init__(self, input_dim, hidden_dim, output_dim):
    super().__init__()
    self.layers = [nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, output_dim)]
  
  def __call__(self, x):
    for layer in self.layers:
      x = layer(x)
    return x
int(c)
model = Model(m, nh, int(c))<jupyter_output><empty_output><jupyter_text>To get the cross entropy loss, we first pass the numerical values through a softmax to get p(x). Then do $$ -\sum x\, \log p(x) $$<jupyter_code>def log_softmax(x): return (x.exp() / x.exp().sum(-1, keepdim=True)).log()
pred = model(x_train) # numerical value
pred = log_softmax(pred) # a softmax<jupyter_output><empty_output><jupyter_text>Let's look at the targets for the first 3 images<jupyter_code>y_train[:3]<jupyter_output><empty_output><jupyter_text>And our model's prediction for them (keep in mind only the target class matter since we're dealing with one-hot vectors here)<jupyter_code>print(pred[0][5])
print(pred[1][0])
print(pred[2][4])
# numpy / pytorch trick to get them quicker, these are the class in the softmax (p(x)) that matter
pred[[0,1,2],[5,0,4]]
y_train.long()
# negative log likelihood loss
# pred[range(target.shape[0]), target] selects the predicted likelihoods for the correct classes, which if you multiply by 1, gives you the loss for that image.
# You then average all images. 
# target is a vector
def nll(pred, target):
  return - pred[range(target.shape[0]), target.long()].mean()<jupyter_output><empty_output><jupyter_text>Optimization for Numerical StabilityBecause computers cannot do perfect math i.e. divisions can only be accurate to a certain number of decimals, computers can't tell differences between two really big but different numbers, there are mathematical tricks we can do to make the computation more stable.Here we optimize log_softmax in two ways**1. Using the formula** $$\log \left ( \frac{a}{b} \right ) = \log(a) - \log(b)$$ **2. Use the LogSumExp trick** Reference to this great [article](https://nhigham.com/2021/01/05/what-is-the-log-sum-exp-function/).Basically $$LSE \left (x \right ) = \log \left ( \sum_{j=1}^{n} e^{x_{j}} \right )$$ is called the Log Sum Exponential - it is useful because it approximates the max function max(x), but it is smooth / differentiable at all points unlike the max function. Note its property (it is very close to the max(x), bounded by a difference of log(n)): $$ \max \left (x \right ) <= LSE \left (x \right ) <= \max \left (x \right ) + \log \left (n \right )$$ which is derived by taking log of $$ e^{\max \left (x \right )} <= \sum_{j=1}^{n} e^{x_{j}} <= n e^{\max \left (x \right )}$$In fact, if x = [0 t], LSE approximates the ReLu, which is max(t, 0). LSE([0 t]), which is just log(1 + e^t), is known as the softplus function. It is a smooth approximation of ReLU.Now, you shouldn't type $$\sum_{j=1}^{n} e^{x_{j}}$$ directly in PyTorch because **if x is big, you can get numerical overflow very fast (with just double-digit x) - note this is an exponential. Watch out for overflows any time you see exponentials!**So we use this trick:$$\log \left ( \sum_{j=1}^{n} e^{x_{j}} \right ) = \log \left ( e^{a} \sum_{j=1}^{n} e^{x_{j}-a} \right ) = a + \log \left ( \sum_{j=1}^{n} e^{x_{j}-a} \right )$$where a is the maximum of the $x_{j}$.Note now you will only get negative exponents. No overflow. Any underflows are harmless.<jupyter_code>def test_log_softmax_equal(func1, func2):
  loss1 = nll(func1(model(x_train)), y_train)
  loss2 = nll(func2(model(x_train)), y_train)
  test_near(loss1, loss2)
# first optimization, avoid a lot of divisions (and thus inaccuracies)
def log_softmax_1(x): 
  # the first term is really x.exp().log() but that's just x
  return x - (x.exp().sum(-1, keepdim=True)).log()
def logsumexp(x):
  a = x.max(-1).values
  return a + (x-a.unsqueeze(-1)).exp().sum(-1).log()
# test our logsumexp against pytorch's
pred = model(x_train)
test_near(logsumexp(pred), pred.logsumexp(-1))

# second optimization. As mentioned above, to avoid overflow, we do second optimization with LSE (log sum exponential) 
def log_softmax_2(x):
  lse = x.logsumexp(-1) # use PyTorch's native logsumexp
  return x - lse.unsqueeze(-1)<jupyter_output><empty_output><jupyter_text>- F.cross_entropy is just first pass pred through **log_softmax**, then pass that though **nll**<jupyter_code># test the 2 optimized log_softmax
test_log_softmax_equal(log_softmax, log_softmax_1)
test_log_softmax_equal(log_softmax, log_softmax_2)

# test our nll against pytorch's implementation
pred = model(x_train)
our_logsoftmax_nll_loss = nll(log_softmax(pred), y_train)
torch_logsoftmax_nll_loss = F.nll_loss(F.log_softmax(pred, -1), y_train.long())
torch_crossentropy_loss = F.cross_entropy(pred, y_train.long())

test_near(our_logsoftmax_nll_loss, torch_logsoftmax_nll_loss)
test_near(torch_logsoftmax_nll_loss, torch_crossentropy_loss)<jupyter_output><empty_output><jupyter_text>Training Loop  Metrics - Loss function is the metrics to help the model train, whereas accuracy is a nice metric for human to see.<jupyter_code>loss_fn = F.cross_entropy
def get_accuracy(out, target): 
  pred = torch.argmax(out, dim=1)
  return (pred == target).float().mean()
print(y_train)
print(y_train.long())<jupyter_output>tensor([5., 0., 4.,  ..., 8., 4., 8.])
tensor([5, 0, 4,  ..., 8, 4, 8])<jupyter_text>First Training Loop - Check how good the model is before training<jupyter_code>batch_size = 64
x_first_batch, y_first_batch = x_train[:batch_size], y_train[:batch_size].long()
out = model(x_first_batch)
print("accuracy: ", get_accuracy(out, y_first_batch))
print("loss: ", loss_fn(out, y_first_batch))
print(x_train.shape)
print(y_train.shape)
print(n)
print((n-1)//batch_size + 1)
print(len(y_train) // batch_size + 1)
y_train, y_valid = y_train.long(), y_valid.long()
epoch_num = 1
batch_num = len(y_train) // batch_size + 1
lr = 0.5

def fit(model):
  for epoch in range(epoch_num):
    for i in range(batch_num):
      batch_start = i * batch_size
      batch_end = batch_start + batch_size
      x_batch = x_train[batch_start:batch_end]
      y_batch = y_train[batch_start:batch_end]
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update prop
      with torch.no_grad():
        for layer in model.layers:
          if hasattr(layer, 'weight'):
            layer.weight -= lr * layer.weight.grad
            layer.bias -= lr * layer.bias.grad
            layer.weight.grad.zero_()
            layer.bias.grad.zero_()

fit(model)
print("accuracy: ", get_accuracy(model(x_first_batch), y_first_batch))
print("loss: ", loss_fn(model(x_first_batch), y_first_batch))<jupyter_output>accuracy:  tensor(0.7500)
loss:  tensor(0.9114, grad_fn=<NllLossBackward>)<jupyter_text>First Refactor: `model.parameters()` - Above, the chunk of code below `with torch.no_grad():` can be reduced. Instead of for looping through each layer in the model to find the parameters, wouldn't it be nice if we could just call `model.parameters()`?- To achieve this refactoring, we need to (1) have the model inherit from `nn.Module` (2) initialize the layers properly so that the parameters are registered to be found by `model.parameters()`  Understanding model.parameters()- To understand how parameters get registered in `model.parameters()`, let's write out how a model behaves when it inherits from nn.Module<jupyter_code>class ModelInheritingNNModule():
  def __init__(self, n_in, n_hidden, n_out):
    self._modules = {}
    self.l1 = nn.Linear(n_in, n_hidden)
    self.l2 = nn.Linear(n_hidden, n_out)
  
  def __setattr__(self, key, value):
    if not key.startswith("_"): self._modules[key] = value # override the native __setattr__ to put the key-value pair into _modules
    super().__setattr__(key, value) # still run the native __setattr__ as it's supposed to
  
  def __repr__(self): return f'{self._modules}'

  def parameters(self):
    for layer in self._modules.values():
      for parameter in layer.parameters():
        yield parameter
our_model = ModelInheritingNNModule(m,nh,10); print(our_model)
print([parameter.shape for parameter in our_model.parameters()])<jupyter_output>{'l1': Linear(in_features=784, out_features=50, bias=True), 'l2': Linear(in_features=50, out_features=10, bias=True)}
[torch.Size([50, 784]), torch.Size([50]), torch.Size([10, 50]), torch.Size([10])]<jupyter_text>- We can achieve the same as above by just inheriting from nn.Module<jupyter_code>class Model(nn.Module):
  def __init__(self, n_in, n_hidden, n_out):
    super().__init__()
    self.l1 = nn.Linear(n_in, n_hidden)
    self.l2 = nn.Linear(n_hidden, n_out)
our_model = Model(m,nh,10); print(our_model)
print([parameter.shape for parameter in our_model.parameters()])<jupyter_output>Model(
  (l1): Linear(in_features=784, out_features=50, bias=True)
  (l2): Linear(in_features=50, out_features=10, bias=True)
)
[torch.Size([50, 784]), torch.Size([50]), torch.Size([10, 50]), torch.Size([10])]<jupyter_text>Refactor the model with `nn.ModuleList` and `nn.Sequential` - Instead of setting every layer separately, we use a list, `self.layers`, to generate a `ModuleList` (a list where each item is registered in the modules and its parameters found by `model.parameters()`)<jupyter_code>input_dim, hidden_dim, output_dim = m, nh, 10
class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.layers = nn.ModuleList([nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, output_dim)])
  
  def __call__(self, x):
    for layer in self.layers:
      x = layer(x)
    return x
model = Model(); print(model)
print([parameter.shape for parameter in model.parameters()])<jupyter_output>Model(
  (layers): ModuleList(
    (0): Linear(in_features=784, out_features=50, bias=True)
    (1): ReLU()
    (2): Linear(in_features=50, out_features=10, bias=True)
  )
)
[torch.Size([50, 784]), torch.Size([50]), torch.Size([10, 50]), torch.Size([10])]<jupyter_text>- We can declare the model above more concisely by using `nn.Sequential`<jupyter_code>model = nn.Sequential(nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, output_dim)); print(model)
print([parameter.shape for parameter in model.parameters()])
def fit(model):
  for epoch in range(epoch_num):
    for i in range(batch_num):
      batch_start = i * batch_size
      batch_end = batch_start + batch_size
      x_batch = x_train[batch_start:batch_end]
      y_batch = y_train[batch_start:batch_end]
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update prop
      with torch.no_grad():
        for p in model.parameters(): p -= p.grad * lr
        model.zero_grad()

fit(model)
print("accuracy: ", get_accuracy(model(x_first_batch), y_first_batch))
print("loss: ", loss_fn(model(x_first_batch), y_first_batch))<jupyter_output>accuracy:  tensor(0.9375)
loss:  tensor(0.3866, grad_fn=<NllLossBackward>)<jupyter_text>Second Refactor: Optimizer Let's replace our previous manually coded optimization step:```pythonwith torch.no_grad():    for p in model.parameters(): p -= p.grad * lr    model.zero_grad()```and instead use just:```pythonopt.step()opt.zero_grad()```<jupyter_code># We're practially implementing torch.optim.SGD here, minus the weight decay stuff
class Optimizer():
  def __init__(self, parameters, lr=0.5):
    self.parameters, self.lr = list(parameters), lr

  def step(self):
    with torch.no_grad():
      for p in self.parameters: p -= p.grad * self.lr
  
  def zero_grad(self):
    for p in self.parameters: p.grad.zero_()
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = Optimizer(model.parameters())
def track_training(training_function):
  def tracked_training_function(model, opt):
    print("## accuracy before fit: ", get_accuracy(model(x_first_batch), y_first_batch))
    print("## loss before fit: ", loss_fn(model(x_first_batch), y_first_batch))
    training_function(model, opt)
    print("## accuracy after fit: ", get_accuracy(model(x_first_batch), y_first_batch))
    print("## loss after fit: ", loss_fn(model(x_first_batch), y_first_batch))
  return tracked_training_function
@track_training
def fit(model, opt):
  for epoch in range(epoch_num):
    for i in range(batch_num):
      batch_start = i * batch_size
      batch_end = batch_start + batch_size
      x_batch = x_train[batch_start:batch_end]
      y_batch = y_train[batch_start:batch_end]
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update prop
      opt.step()
      opt.zero_grad()

fit(model, opt)<jupyter_output>## accuracy before fit:  tensor(0.0781)
## loss before fit:  tensor(2.2999, grad_fn=<NllLossBackward>)
## accuracy after fit:  tensor(0.8906)
## loss after fit:  tensor(0.3722, grad_fn=<NllLossBackward>)<jupyter_text>Dataset & DataLoader  Unify `x_train` and `y_train` into `Dataset` class<jupyter_code>class Dataset():
  def __init__(self, x, y): self.x, self.y = x, y
  def __len__(self): return len(self.x)
  def __getitem__(self, i): return self.x[i], self.y[i]
train_ds, valid_ds = Dataset(x_train, y_train), Dataset(x_valid, y_valid)
xb, yb = train_ds[0:5]; print(xb)
assert xb.shape == (5, 28 * 28)
assert yb.shape == (5,)
from torch import optim
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)<jupyter_output><empty_output><jupyter_text>- Training loop is now:<jupyter_code>@track_training
def fit(model, opt):
  for epoch in range(epoch_num):
    for i in range(batch_num):
      batch_start = i * batch_size
      batch_end = batch_start + batch_size
      x_batch, y_batch = train_ds[batch_start:batch_end]
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update prop
      opt.step()
      opt.zero_grad()

fit(model, opt)<jupyter_output>## accuracy before fit:  tensor(0.0938)
## loss before fit:  tensor(2.3165, grad_fn=<NllLossBackward>)
## accuracy after fit:  tensor(0.9219)
## loss after fit:  tensor(0.4996, grad_fn=<NllLossBackward>)<jupyter_text>Dataloader Previously, our loop iterated over batches (xb, yb) like this:```pythonfor i in range((n-1)//bs + 1):    xb,yb = train_ds[i*bs : i*bs+bs]    ...```Let's make our loop much cleaner, using a data loader:```pythonfor xb,yb in train_dl:    ...```<jupyter_code>class DataLoader():
  def __init__(self, dataset, batch_size): self.dataset, self.batch_size = dataset, batch_size
  
  # yields `something` for each loop through `for something in Dataloader`, which iterate through the entire Dataloader object
  def __iter__(self):
    for i in range(0, len(self.dataset), self.batch_size): 
      yield self.dataset[i:i+self.batch_size]
train_dl = DataLoader(train_ds, batch_size)
valid_dl = DataLoader(valid_ds, batch_size)
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)
@track_training
def fit(model, opt):
  for epoch in range(epoch_num):
    for x_batch, y_batch in train_dl:
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update gradient
      opt.step()
      opt.zero_grad()

fit(model, opt)<jupyter_output>## accuracy before fit:  tensor(0.1094)
## loss before fit:  tensor(2.2933, grad_fn=<NllLossBackward>)
## accuracy after fit:  tensor(0.9219)
## loss after fit:  tensor(0.2581, grad_fn=<NllLossBackward>)<jupyter_text>Improve the Dataloader  Random Sampling - We want to shuffle the order of our training set for each iteration to not get stuck on local minimum. Think about it this for: for any fixed datast X, there's a fixed geometric curve for loss L. That curve would have some local minima. If we have the same X, if we get stuck in a local minima, there's nothing we can do anymore. However, if we use minibatches and random shuffling, we change the X (and thereby the shape of L) with every minibatch; therefore, even when we get stuck in the local minima for one X, we're likely still able to descend during the next batch (which has a different shape). [Reference](https://stats.stackexchange.com/questions/245502/why-should-we-shuffle-data-while-training-a-neural-network).- Below, random sampler will randomly select the indices, dataloader will pick those examples out into the batch.<jupyter_code>class Sampler():
  def __init__(self, dataset, batch_size, shuffle=False):
    self.dataset_size, self.batch_size, self.shuffle = len(dataset), batch_size, shuffle

  def __iter__(self):
    self.indices = torch.randperm(self.dataset_size) if self.shuffle else torch.arange(self.dataset_size)
    for i in range(0, self.dataset_size, self.batch_size): 
      yield self.indices[i:i+self.batch_size]
small_ds = Dataset(*train_ds[:10])
s = Sampler(small_ds,2,False)
[o for o in s]
s = Sampler(small_ds,3,True)
[o for o in s]
''' collate function prepares a batch
batch: [(x1, y1), (x2, y2), (x3, y3)]

*batch destructure all the elements from batch, so in this trivial example, it turns 1 list into 3 tuples:
(x1, y1)
(x2, y2)
(x3, y3)

zip(*batch) will zip these tuples vertically:
(x1, x2, x3)
(y1, y2, y3)

Now you use torch.stack to turn these tuples above into tensors
'''
def collate(batch):
  batch_examples_x, batch_examples_y = zip(*batch)
  return torch.stack(batch_examples_x), torch.stack(batch_examples_y)
class DataLoader():
  def __init__(self, dataset, sampler, collate_fn = collate):
    self.dataset, self.sampler, self.collate_fn = dataset, sampler, collate_fn
  
  def __iter__(self):
    for batch_indices in self.sampler:
      batch = [self.dataset[i] for i in batch_indices]
      yield self.collate_fn(batch)
train_sampler = Sampler(train_ds, batch_size, shuffle=True)
train_dl = DataLoader(train_ds, train_sampler)
valid_sampler = Sampler(valid_ds, batch_size, shuffle=False)
valid_dl = DataLoader(valid_ds, valid_sampler)

model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)
fit(model, opt)<jupyter_output>## accuracy before fit:  tensor(0.0781)
## loss before fit:  tensor(2.3632, grad_fn=<NllLossBackward>)
## accuracy after fit:  tensor(0.9062)
## loss after fit:  tensor(0.2265, grad_fn=<NllLossBackward>)<jupyter_text>PyTorch Dataloader - PyTorch's RandomSampler is equivalent to our sampler with `shuffle=True`- PyTorch's SequentialSampler is equivalent to our sampler with `shuffle=False`- We can also not specify the Sampler and just use the default.<jupyter_code>from torch.utils.data import DataLoader, SequentialSampler, RandomSampler
train_dl = DataLoader(train_ds, batch_size, sampler=RandomSampler(train_ds), collate_fn=collate)
valid_dl = DataLoader(valid_ds, batch_size, sampler=SequentialSampler(valid_ds), collate_fn=collate)

model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)
fit(model, opt)
# Default
train_dl = DataLoader(train_ds, batch_size, shuffle=True, drop_last=True)
valid_dl = DataLoader(valid_ds, batch_size, shuffle=False)

model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)
fit(model, opt)<jupyter_output>## accuracy before fit:  tensor(0.1094)
## loss before fit:  tensor(2.3361, grad_fn=<NllLossBackward>)
## accuracy after fit:  tensor(0.9219)
## loss after fit:  tensor(0.2515, grad_fn=<NllLossBackward>)<jupyter_text>Add Validation<jupyter_code>def fit(model, opt):
  for epoch in range(epoch_num):
    model.train()
    for x_batch, y_batch in train_dl:
      # forward pass
      out = model(x_batch)
      loss = loss_fn(out, y_batch)
      # backward pass
      loss.backward()
      # update gradient
      opt.step()
      opt.zero_grad()
    
    model.eval() # disable dropout and batchnorm to get the loss properly
    with torch.no_grad():
      total_loss, total_accuracy = 0., 0.
      for x_batch, y_batch in valid_dl:
        out = model(x_batch)
        total_loss += float(loss_fn(out, y_batch))
        total_accuracy += get_accuracy(out, y_batch)
      batch_num = len(valid_dl)
      print("epoch: ", epoch, ", average loss per batch: ", total_loss / batch_num, "average accuracy per batch: ", total_accuracy / batch_num)
    
    final_avg_loss = total_loss / batch_num
    final_accuracy = total_accuracy / batch_num
    return final_avg_loss, final_accuracy
def get_dls(train_ds, valid_ds, batch_size, **kwargs):
  # we can use double batch size for valid because no need to store gradients
  return DataLoader(train_ds, batch_size=batch_size, shuffle=True, **kwargs), DataLoader(valid_ds, batch_size=batch_size*2, **kwargs)
train_dl, valid_dl = get_dls(train_ds, valid_ds, batch_size)
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
opt = optim.SGD(model.parameters(), lr=lr)
loss, acc = fit(model, opt)<jupyter_output>epoch:  0 , average loss per batch:  0.3296205705456153 average accuracy per batch:  tensor(0.9141)
{'fraction_non_alphanumeric': 0.1261070465922218, 'fraction_numerical': 0.020820727212717972, 'mean_word_length': 4.541685718640451, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 131, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '370365', 'license': 'Apache-2.0', 'n_tokens_mistral': 13309, 'n_tokens_neox': 12776, 'n_words': 3635, 'path': 'fastai_foundation_notes_1.ipynb', 'repo_name': 'ghlai9665/course-v3', 'size': '36358'}