<jupyter_start><jupyter_text>Importing Necessary Libraries<jupyter_code># supressing warnings
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import os

# mount google drive
from google.colab import drive
drive.mount('/content/gdrive')
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
from fastai.vision.all import *
from fastbook import *<jupyter_output><empty_output><jupyter_text>Loading Dataset & Visualizing it in different way<jupyter_code>path = untar_data(URLs.MNIST_SAMPLE)
# Checking path
path.ls()
# storing path of 3 & 7 of train dataset
threes = (path/'train'/'3').ls().sorted()
sevens = (path/'train'/'7').ls().sorted()
threes
# checking single image of 3
im3_path = threes[1]
im3 = Image.open(im3_path) # uses PIL library in bg
im3
# visualizing pixels of im3 image in numpy array
array(im3)
# visualizing pixels of im3 image in tensor object
tensor(im3)
#@title
im3_t = tensor(im3)
df = pd.DataFrame(im3_t[3:25,4:21]) # total 28*28
df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')
three_tensors = [tensor(Image.open(o)) for o in threes]
seven_tensors = [tensor(Image.open(o)) for o in sevens]
len(three_tensors),len(seven_tensors)
three_tensors[1]
show_image(three_tensors[1])
stacked_threes = torch.stack(three_tensors).float()/255
stacked_sevens = torch.stack(seven_tensors).float()/255
stacked_threes.shape,stacked_sevens.shape
show_image(stacked_threes.mean(0))
mean3 = stacked_threes.mean(0)
mean7 = stacked_sevens.mean(0)<jupyter_output><empty_output><jupyter_text>Checking distance of a 3 image from mean3 & mean7.- We can use this 2 ways, absolute distance & RMSE- RMSE will penalize bigger mistakes more heavily than the absolute loss.<jupyter_code>a_3 = stacked_threes[1]
a_7 = stacked_sevens[1]
dist_3_abs = (a_3 - mean3).abs().mean()
dist_3_sqr = ((a_3 - mean3)**2).mean().sqrt()
print('Absolute distance from mean3:',dist_3_abs,'\nRMSE distance from mean3:',dist_3_sqr)
dist_7_abs = (stacked_threes[1] - mean7).abs().mean()
dist_7_sqr = ((stacked_threes[1] - mean7)**2).mean().sqrt()
dist_7_abs,dist_7_sqr
print('Absolute distance from mean7:',dist_7_abs,'\nRMSE distance from mean7:',dist_7_sqr)<jupyter_output>Absolute distance from mean7: tensor(0.1586) 
RMSE distance from mean7: tensor(0.3021)<jupyter_text>- In both type of distance, we can see that image 3 is closer to mean3 than mean7. We can say that image 3 is more likely a image of 3 than an image of 7.<jupyter_code># Absolute distance(L1) & RMSE(L2) distance is inbuilt in Pythorch & generally imported as 'F'(Fastai imports that as F aswell)
F.l1_loss(stacked_threes[1].float(),mean7), F.mse_loss(stacked_threes[1],mean7).sqrt()
# Building tensor for validation dataset (test dataset)
valid_3_tens = torch.stack([tensor(Image.open(o)) for o in (path/'valid'/'3').ls()])
valid_3_tens = valid_3_tens.float()/255
valid_7_tens = torch.stack([tensor(Image.open(o)) for o in (path/'valid'/'7').ls()])
valid_7_tens = valid_7_tens.float()/255
valid_3_tens.shape,valid_7_tens.shape
# doing same as dist_3_abs/l1 loss, but building a custom function to demonstrate + it can use broadcasting to calculate all images in stacked_threes or valid_3
def mnist_distance(a,b): return (a-b).abs().mean((-1,-2))
mnist_distance(a_3, mean3)
valid_3_tens.shape, mean3.shape
# this is cool, we sent a tensor as a & b dimenssion is different as you can see above
valid_3_dist = mnist_distance(valid_3_tens, mean3)
valid_3_dist, valid_3_dist.shape
(valid_3_tens-mean3).shape # see how 28*28 becomes 1010*28*28
# if distance is less from mean3=>true, else false
def is_3(x): return mnist_distance(x,mean3) < mnist_distance(x,mean7)
is_3(a_3), is_3(a_3).float()
# Checking accuracy of model in validity dataset
accuracy_3s = is_3(valid_3_tens).float().mean()
accuracy_7s = 1-is_3(valid_7_tens).float().mean()
accuracy_3s,accuracy_7s,(accuracy_3s+accuracy_7s)/2<jupyter_output><empty_output><jupyter_text>Not a Neural Network so far- We have so far got mean of pixels of an image of 3. Then we checked the pixel similarity to classify images to 3 & 7.- Model is not learning itself & there is no way to improve itself.- We have not yet used weights, activation funtion or gradient descent to update weight.- We are only using a distance metric to measure validity image from ideal mean image.  Using Neural Network  Preparing Data for pytorch<jupyter_code>train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1, 28*28) # just concating 3 & 7 matrix
train_y = tensor([1]*len(threes) + [0]*len(sevens)).unsqueeze(1) # label 1 & 0 for classifying 3

train_x.shape,train_y.shape
dset = list(zip(train_x,train_y)) # dataset in pytorch required to feed in this format
x,y = dset[0]
x.shape,y
# Doing same for validity set
valid_x = torch.cat([valid_3_tens, valid_7_tens]).view(-1, 28*28)
valid_y = tensor([1]*len(valid_3_tens) + [0]*len(valid_7_tens)).unsqueeze(1)

valid_dset = list(zip(valid_x,valid_y)) # Pytorch Standard dataset that dataloader expects
dl = DataLoader(dset, batch_size=256) # dset already in (x,y) format, dataloader creates 256 mini batches
valid_dl = DataLoader(valid_dset, batch_size=256) # same for validity dataset

xb,yb = first(dl)
xb.shape,yb.shape<jupyter_output><empty_output><jupyter_text>Manually doing some stuffs to show how that would be working in pytorch  Putting together all function we are gonna use<jupyter_code># creating function to initialize parameter using a function
def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()

# a single neuron (Pytorch got `nn.Linear` function that'll replace init_params too)
def linear1(xb): return xb@weights + bias # @ used for matrix multiplication

# Loss function that we'll be used in optimization step(Gradient descent)
def mnist_loss(predictions, targets):
 predictions = predictions.sigmoid()
 return torch.where(targets==1, 1-predictions, predictions).mean()

# gradient calculation
def calc_grad(xb, yb, model):
 preds = model(xb)
 loss = mnist_loss(preds, yb)
 loss.backward()

# call calc_grad, update bias & repeat for epoch no of time
def train_epoch(model, lr, params):
  for xb,yb in dl:
    calc_grad(xb, yb, model)
    for p in params:
      p.data -= p.grad*lr
      p.grad.zero_()

# accuracy of model for validation dataset(a single batch)
def batch_accuracy(xb, yb):
 preds = xb.sigmoid()
 correct = (preds>0.5) == yb
 return correct.float().mean()

# pass all batches to batch_accuracy function to calculate accuracy of all batches
def validate_epoch(model):
 accs = [batch_accuracy(model(xb), yb) for xb,yb in valid_dl]
 return round(torch.stack(accs).mean().item(), 4)
weights = init_params((28*28,1))
bias = init_params(1)
weights.shape,bias.shape
# Predicting for single training example
(train_x[0]*weights.T).sum() + bias # predictiong with initial parameter i.e random
preds = linear1(train_x)
preds
corrects = (preds>0.0).float() == train_y
corrects.float().mean().item() # gettting 53% accuracy in random initialization(in next run it'll change though)
# trying it with a sample before running it in whole batch
batch = train_x[:4]
calc_grad(batch, train_y[:4], linear1)
weights.grad.mean(),bias.grad
# because if we calculate gradient again, it'll add it to previously calculated gradient, so we set gradient to zero after using it
weights.grad.zero_()
bias.grad.zero_();
validate_epoch(linear1)
lr = 1.
params = weights,bias
for i in range(20):
 train_epoch(linear1, lr, params)
 print(validate_epoch(linear1), end=' ')<jupyter_output>0.6258 0.7894 0.8978 0.9271 0.9398 0.9486 0.9549 0.9613 0.9643 0.9657 0.9667 0.9687 0.9687 0.9711 0.9726 0.9735 0.9745 0.975 0.975 0.975<jupyter_text>Doing it Pytorch way<jupyter_code># this will initialize biases & create linear model i.e a single neuron's function
linear_model = nn.Linear(28*28,1)

# to see the parameters
w,b = linear_model.parameters()
w.shape,b.shape
# training single instance
def train_epoch(model, opt):
 for xb,yb in dl:
  calc_grad(xb, yb, model)
  opt.step()
  opt.zero_grad()

# training epoch no of times
def train_model(model, opt, epochs):
 for i in range(epochs):
  train_epoch(model, opt)
  print(validate_epoch(model), end=' ')

# Pytorch's optimization function
opt_pytorch = SGD(linear_model.parameters(), lr) # lr is learning rate we defined previously

train_model(linear_model, opt_pytorch, 20) # pass model, optimization function, no of epoch<jupyter_output>0.4932 0.832 0.8452 0.9126 0.9341 0.9473 0.9561 0.9634 0.9663 0.9678 0.9692 0.9727 0.9746 0.9751 0.9756 0.9766 0.978 0.978 0.978 0.979<jupyter_text>SGD function explained<jupyter_code># something similar to pytorch's SGD class
class BasicOptim:
 def __init__(self,params,lr):
   self.params,self.lr = list(params),lr
 def step(self, *args, **kwargs):
  for p in self.params:
    p.data -= p.grad.data * self.lr
 def zero_grad(self, *args, **kwargs):
  for p in self.params:
    p.grad = None

opt_manual = BasicOptim(linear_model.parameters(), lr)
train_model(linear_model, opt_manual, 20)<jupyter_output>0.979 0.979 0.9795 0.9795 0.9795 0.98 0.98 0.98 0.9805 0.9819 0.9819 0.9814 0.9809 0.9809 0.9814 0.9814 0.9814 0.9814 0.9814 0.9814<jupyter_text>Fast.ai's builtin train_model- There is a train_model equivalent in fast.ai- for that we have to pass dataloader in (train_dl,  validity_dl) format<jupyter_code>dls = DataLoaders(dl, valid_dl)
learn = Learner(dls, nn.Linear(28*28,1), opt_func=SGD, loss_func=mnist_loss, metrics=batch_accuracy) # Fastai module `Learner`
# help(Learner)
learn.fit(10, lr=1.)<jupyter_output><empty_output><jupyter_text>**As you can see, there’s nothing magic about the PyTorch and fastai classes. They are just convenient prepackaged pieces that make your life a bit easier**  So far model is linear- Adding non-linearity- Why do we need to add non-linearity?(ans later)<jupyter_code># building a custom neuron
simple_net = nn.Sequential(
 nn.Linear(28*28,30),
 nn.ReLU(),
 nn.Linear(30,1)
)

learn = Learner(dls, simple_net, opt_func=SGD, loss_func=mnist_loss, metrics=batch_accuracy)
learn.fit(40, lr=0.1) # we have taken learning rate low than previously, so 40 epoch for good response
plt.plot(L(learn.recorder.values).itemgot(2));<jupyter_output><empty_output><jupyter_text>- Mathmetically this non-linear function can solve any problem to any level of accuracy (the neural network), given that correct set of parameters are provided.  Example of 18-layer model- We are gonna use a 18 layer model but **single epoch**.- But that's going to give us near 100% accuracy<jupyter_code># this is fast with GPU
dls = ImageDataLoaders.from_folder(path)
learn = cnn_learner(dls, resnet18, pretrained=False,
 loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(1, 0.1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11492010405053883, 'fraction_numerical': 0.051839464882943144, 'mean_word_length': 5.707165109034268, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '156259', 'license': 'MIT', 'n_tokens_mistral': 4141, 'n_tokens_neox': 3829, 'n_words': 1109, 'path': 'MNIST_digit_classifier.ipynb', 'repo_name': 'addicted-ai/fastai-tests', 'size': '10764'}