<jupyter_start><jupyter_text>normalize the train & valida input data.see we use training values for valid data normalization<jupyter_code>x_train_mean, x_train_std = x_train.mean(), x_train.std()
x_train_mean, x_train_std 
x_train = normalize(x_train, x_train_mean, x_train_std)
x_valid = normalize(x_valid, x_train_mean, x_train_std)<jupyter_output><empty_output><jupyter_text>see diff in mean & std values<jupyter_code>x_train_mean, x_train_std = x_train.mean(), x_train.std()
x_train_mean, x_train_std 
def test_near_zero(a, tol=1e-3):
    assert a.abs()<tol, f"Near zero: {a}"
test_near_zero(x_train.mean())
test_near_zero(1 - x_train.std())
n, m = x_train.shape  
c = y_train.max()  + 1 # 1 extra class for the input which we cannot classify
n, m, c<jupyter_output><empty_output><jupyter_text>ARCHITECTURE linear, relu, kaiming init & model  we will first see the effect of random initialization vs [Kaiming initialization 2.2](https://arxiv.org/abs/1502.01852)<jupyter_code>nh = 50 # hidden layers
# randn: Returns a tensor filled with random numbers from a normal distribution with mean `0` and variance `1`
w1 = torch.randn(m, nh)/math.sqrt(m)
w2 = torch.randn(nh, 1)/math.sqrt(m)
b1 = torch.zeros(nh)
b2 = torch.zeros(1)

test_near_zero(w1.mean())
test_near_zero(w1.std()- 1/math.sqrt(m))
x_valid.mean(), x_valid.std() # close to 0 & 1
def lin(x, w, b):
    # linear layer
    return x@w + b

t = lin(x_valid, w1, b1)
t.mean(), t.std()
def relu(x):
    return x.clamp_min(0.)

t1 = relu(t) # t from lin(..)
t1.mean(), t1.std()
def kaiming_init(m, nh):
    return torch.randn(m, nh) * math.sqrt(2./m)
w1 = kaiming_init(m , nh)
t = relu(lin(x_valid, w1, b1))
w1.mean(), w1.std(), t.mean(), t.std()<jupyter_output><empty_output><jupyter_text>we can compare this kaiming_init with Pytorch kaiminig init<jupyter_code>from torch.nn import init
w1 = torch.zeros(m, nh)
init.kaiming_normal_(w1, mode="fan_out") 
#Choosing `fan_in` preserves the magnitude of the variance of the weights in the
#forward pass. Choosing `fan_out` preserves the magnitudes in the backwards pass.
t = relu(lin(x_valid, w1, b1))
w1.mean(), w1.std(), t.mean(), t.std()<jupyter_output><empty_output><jupyter_text>our relu is having mean > 0.5. Let's clip at 0.5 mercilessly.<jupyter_code>def relu(x):
    return x.clamp_min(0.) - 0.5
w1 = kaiming_init(m, nh)
t1 = relu(lin(x_valid, w1, b1))
w1.mean(), w1.std(), t.mean(), t.std()<jupyter_output><empty_output><jupyter_text>Our First model fun<jupyter_code>def model(xb):
    l1 = lin(xb, w1, b1)
    l2 = relu(l1)
    l3 = lin(l2, w2, b2)
    return l3
# w2 = kaiming_init(nh, 1)
%timeit -n 10 _= model(x_valid)
assert model(x_valid).shape == torch.Size([x_valid.shape[0],1])<jupyter_output><empty_output><jupyter_text>Our First Loss Fun Mean Sq Error (MSE). to keep this simple<jupyter_code># squeeze will remove ,1] ; unsqueeze does the opposite
model(x_valid).shape, model(x_valid).squeeze(-1).shape
def mse(output, targ):
    return (output.squeeze(-1) - targ).pow(2).mean()
y_train, y_valid = y_train.float(), y_valid.float()<jupyter_output><empty_output><jupyter_text>predict the output<jupyter_code>preds = model(x_train)
x_train.shape, preds.shape, y_train.shape
# 50000 samples of 784 col (28 * 28 pic)
mse(preds, y_train)<jupyter_output><empty_output><jupyter_text>We will calculate the gradienct (i.e slope by differentiation) for backward passwe will store gradient at `g` attribute<jupyter_code>def mse_grad(inp, targ):
    # grad of loss wrt o/p of prev layer
    inp.g = 2. * (inp.squeeze() - targ).unsqueeze(-1) / inp.shape[0] # this is gradient ;)
def relu_grad(inp, out):
    # grad of relu wrt input activations
    inp.g = (inp>0).float() * out.g
def lin_grad(inp, out, w, b):
    # grad of matrix mul wrt input:
    inp.g = out.g @ w.t()
    w.g = (inp.unsqueeze(-1) * out.g.unsqueeze(1)).sum(0)
    b.g = out.g.sum(0)
def forward_and_backward(inp, targ):
    # forward pass:
    l1 = inp @ w1 + b1
    l2 = relu(l1)
    out = l2 @ w2 + b2
    # we don't actually need the loss in backward!
    loss = mse(out, targ)
    
    # backward pass:
    mse_grad(out, targ)
    lin_grad(l2, out, w2, b2)
    relu_grad(l1, l2)
    lin_grad(inp, l1, w1, b1)
forward_and_backward(x_train, y_train)<jupyter_output><empty_output><jupyter_text>var w1, w2 b1 b2 are available because we declared it earlier<jupyter_code>w1g = w1.g.clone()
w2g = w2.g.clone()
b1g = b1.g.clone()
b2g = b2.g.clone()
ig = x_train.g.clone()
xt2 = x_train.clone().requires_grad_(True)
w12 = w1.clone().requires_grad_(True)
w22 = w2.clone().requires_grad_(True)
b12 = b1.clone().requires_grad_(True)
b22 = b2.clone().requires_grad_(True)<jupyter_output><empty_output><jupyter_text>Let's test if your calculation are correct and matches with Pytorch<jupyter_code>xt2 = x_train.clone().requires_grad_(True) # true -> train it with grad calculations
w12 = w1.clone().requires_grad_(True)
w22 = w2.clone().requires_grad_(True)
b12 = b1.clone().requires_grad_(True)
b22 = b2.clone().requires_grad_(True)
def forward(inp, targ):
    l1 = inp @ w12 + b12
    l2 = relu(l1)
    out = l2 @ w22 + b22
    # no loss calc
    return mse(out, targ)
loss = forward(xt2, y_train)
loss.backward()<jupyter_output><empty_output><jupyter_text>verify our calculations<jupyter_code>w22.grad.shape, w2g.shape
test_near(w22.grad, w2g)
test_near(b22.grad, b2g)
test_near(w12.grad, w1g)
test_near(b12.grad, b1g)
test_near(xt2.grad, ig )<jupyter_output><empty_output><jupyter_text>RefactoringLets convert fun into classes.<jupyter_code>class Relu():
    def __call__(self, inp):
        self.inp = inp
        self.out = inp.clamp_min(0.) - 0.5
        return self.out
    
    def backward(self):
        self.inp.g = (self.inp>0).float() * self.out.g
class Lin():
    def __init__(self, w, b):
        self.w, self.b = w, b
        
    def __call__(self, inp):
        self.inp = inp
        self.out = inp @ self.w + self.b
        return self.out
    
    def backward(self):
        self.inp.g = self.out.g @ self.w.t()
        # DONT TRY THIS AT HOME.
        # big matrix multiplication. There are better way to do this.
        self.w.g = (self.inp.unsqueeze(-1) * self.out.g.unsqueeze(1)).sum(0)
        self.b.g = self.out.g.sum(0)
class Mse():
    def __call__(self, inp, targ):
        self.inp = inp
        self.targ = targ
        self.out = (inp.squeeze() - targ).pow(2).mean()
        return self.out
    
    def backward(self):
        self.inp.g = 2 * (self.inp.squeeze() - self.targ).unsqueeze(-1) / self.targ.shape[0]
class Model():
    def __init__(self, w1, b1, w2, b2, loss=Mse):
        self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
        self.loss = loss()
        
    def __call__(self, x, targ):
        for l in self.layers:
            x = l(x)
        return self.loss(x, targ)
    
    def backward(self):
        self.loss.backward()
        for l in reversed(self.layers):
            l.backward()
            
# Reset
w1.g, b1.g, w2.g, b2.g = [None] * 4
model = Model(w1, b1, w2, b2)
%time loss = model(x_train, y_train)
%time model.backward()<jupyter_output>CPU times: user 3.83 s, sys: 3.96 s, total: 7.79 s
Wall time: 3.97 s<jupyter_text>verify our calculations<jupyter_code>test_near(w2g, w2.g)
test_near(b2g, b2.g)
test_near(w1g, w1.g)
test_near(b1g, b1.g)
test_near(ig, x_train.g)<jupyter_output><empty_output><jupyter_text>Some more Refactoring. Adding Module.forward()<jupyter_code>class Module():
    def __call__(self, *args):
        self.args = args
        self.out = self.forward(*args)
        return self.out
    
    def forward(self):
        raise Exception("Not Implemented")
        
    def backward(self):
        self.bwd(self.out, *self.args)
class Relu(Module):
    def forward(self, inp):
        return inp.clamp_min(0) - .5
    
    def bwd(self, out, inp):
        # calculate gradient
        inp.g = (inp>0).float() * out.g
class Lin(Module):
    def __init__(self, w, b):
        self.w, self.b = w, b
        
    def forward(self, inp):
        return inp@self.w + self.b
    
    def bwd(self, out, inp):
        inp.g = out.g @ self.w.t()
        self.w.g = torch.einsum("bi,bj->ij", inp, out.g) 
        self.b.g = out.g.sum(0)
class Mse(Module):
    def forward (self, inp, targ):
        return (inp.squeeze() - targ).pow(2).mean()
    
    def bwd(self, out, inp, targ): 
        inp.g = 2*(inp.squeeze()-targ).unsqueeze(-1) / targ.shape[0]
class Model():
    def __init__(self, loss=Mse): # w1, b1, w2, b2,
        self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
        self.loss = loss()
        
    def __call__(self, x, targ):
        for l in self.layers:
            x = l(x)
        return self.loss(x, targ)
    
    def backward(self):
        self.loss.backward()
        for l in reversed(self.layers):
            l.backward()
            
w1.g, b1.g, w2.g, b2.g = [None]*4
model = Model()
%time loss = model(x_train, y_train)
%time model.backward()
test_near(w2g, w2.g)
test_near(b2g, b2.g)
test_near(w1g, w1.g)
test_near(b1g, b1.g)
test_near(ig, x_train.g)<jupyter_output><empty_output><jupyter_text>We will see timing without einsum support<jupyter_code>class Lin(Module):
    def __init__(self, w, b): 
        self.w,self.b = w,b
        
    def forward(self, inp): 
        return inp @ self.w + self.b
    
    def bwd(self, out, inp):
        inp.g = out.g @ self.w.t()
        self.w.g = inp.t() @ out.g
        self.b.g = out.g.sum(0)

w1.g,b1.g,w2.g,b2.g = [None]*4
model = Model()
%time loss = model(x_train, y_train)
%time model.backward()<jupyter_output>CPU times: user 248 ms, sys: 84 ms, total: 332 ms
Wall time: 165 ms<jupyter_text>In this case, it is almost similiar. But it won't be always<jupyter_code>test_near(w2g, w2.g)
test_near(b2g, b2.g)
test_near(w1g, w1.g)
test_near(b1g, b1.g)
test_near(ig, x_train.g)<jupyter_output><empty_output><jupyter_text>Let's see how Pytorch nn module will perform<jupyter_code>from torch import nn

class Model(nn.Module):
    def __init__(self, n_in, nh, n_out):
        super().__init__()
        self.layers = [nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out)]
        self.loss = mse
        
    def __call__(self, x, targ):
        for l in self.layers:
            x = l(x)
        return self.loss(x.squeeze(), targ)
model = Model(m, nh, 1)
%time loss = model(x_train, y_train)
%time loss.backward()<jupyter_output>CPU times: user 264 ms, sys: 4 ms, total: 268 ms
Wall time: 89.2 ms
{'fraction_non_alphanumeric': 0.14726749760306806, 'fraction_numerical': 0.028667305848513902, 'mean_word_length': 3.357142857142857, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 76, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '298473', 'license': 'Apache-2.0', 'n_tokens_mistral': 4417, 'n_tokens_neox': 4251, 'n_words': 970, 'path': 'dl2/RESNET from scratch.ipynb', 'repo_name': 'nik-hil/ai_experiment', 'size': '10430'}