<jupyter_start><jupyter_text>"MNIST: Distinguishing Threes from Sevens"> Using Fastai to classify threes and sevens from the MNIST dataset.- toc: true- badges: true- comments: true- categories: [fastai, jupyter]  IntroductionI am following along with Chapter 4 of {% cite fastbook2020 %}.  Download dataset<jupyter_code>from fastai.vision.all import *
#collapse-output
path = untar_data(URLs.MNIST_SAMPLE)<jupyter_output><empty_output><jupyter_text>So what got downloaded?<jupyter_code>path.ls()<jupyter_output><empty_output><jupyter_text>That huge path is a pain to look at so shorten it by setting the BASE_PATH.<jupyter_code>Path.BASE_PATH = path
path.ls()<jupyter_output><empty_output><jupyter_text>What do we have under train?<jupyter_code>(path/'train').ls()<jupyter_output><empty_output><jupyter_text>What do we have under the 7?<jupyter_code>(path/'train'/'7').ls().sorted()<jupyter_output><empty_output><jupyter_text>We have 6,265 images of sevens. Look at one using the PIL library.<jupyter_code>Image.open((path/'train'/'7').ls().sorted()[0])
seven_tensors = [tensor(Image.open(pic_path)).float()/255. for pic_path in (path/'train'/'7').ls().sorted()]
three_tensors = [tensor(Image.open(pic_path)).float()/255. for pic_path in (path/'train'/'3').ls().sorted()]
len(seven_tensors), len(three_tensors)<jupyter_output><empty_output><jupyter_text>Use Fastai convenience function _show\_image_ to display the tensor<jupyter_code>show_image(seven_tensors[0])
stacked_sevens = torch.stack(seven_tensors)
stacked_threes = torch.stack(three_tensors)
stacked_sevens.shape, stacked_threes.shape<jupyter_output><empty_output><jupyter_text>Assemble train and validation setAssemble the training data. Each input will be a vector of 784 values.<jupyter_code>train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1,28*28)
train_y = tensor([1]*len(three_tensors) + [0]*len(seven_tensors)).unsqueeze(1)
train_x.shape, train_y.shape<jupyter_output><empty_output><jupyter_text>Assemble the validation data<jupyter_code>valid_7_tensors = torch.stack([tensor(Image.open(pic_path)).float()/255. for pic_path in (path/'valid'/'7').ls().sorted()])
valid_3_tensors = torch.stack([tensor(Image.open(pic_path)).float()/255. for pic_path in (path/'valid'/'3').ls().sorted()])
valid_7_tensors.shape, valid_3_tensors.shape
valid_x = torch.cat([valid_3_tensors, valid_7_tensors]).view(-1,28*28)
valid_y = tensor([1]*len(valid_3_tensors) + [0]*len(valid_7_tensors)).unsqueeze(1)
valid_x.shape, valid_y.shape<jupyter_output><empty_output><jupyter_text>Initialize Parameters<jupyter_code>def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()<jupyter_output><empty_output><jupyter_text>We have one weight for each pixel in the image. We will also have a bias term.<jupyter_code>weights = init_params((28*28,1))
bias = init_params(1)
weights.shape, bias.shape<jupyter_output><empty_output><jupyter_text>Get PredictionsCalculate the prediction for a single image<jupyter_code>(train_x[0]*weights.T).sum() + bias
train_x[0].shape, weights.shape, weights.T.shape, (train_x[0]*weights).shape
def linear1(xb): return xb @ weights + bias
preds = linear1(train_x)
preds<jupyter_output><empty_output><jupyter_text>Compute Loss<jupyter_code>def sigmoid(x): return 1./(1. + torch.exp(-x))
def mnist_loss(predictions, targets):
    preds = predictions.sigmoid()
    return torch.where(targets==1, 1-preds, preds).mean()
mnist_loss(preds, train_y)<jupyter_output><empty_output><jupyter_text>Mini-Batches<jupyter_code>ds = L(enumerate(string.ascii_lowercase))
ds<jupyter_output><empty_output><jupyter_text>Each mini-batch is a tuple of some number of training examples and their corresponding labels<jupyter_code>dl = DataLoader(ds,batch_size=5,shuffle=True)
first(dl), list(dl)
dset = list(zip(train_x, train_y))
dl = DataLoader(dset, batch_size=256)

valid_dset = list(zip(valid_x, valid_y))
valid_dl = DataLoader(valid_dset, batch_size=256)<jupyter_output><empty_output><jupyter_text>Simulate a batch of training examples.<jupyter_code>batch = train_x[:4]
batch.shape
preds = linear1(batch)
preds
loss = mnist_loss(preds, train_y[:4])
loss
loss.backward()
weights.grad.shape, weights.grad.mean(), bias.grad<jupyter_output><empty_output><jupyter_text>Gradient computation<jupyter_code>def calc_grad(xb,yb,model):
    preds = model(xb)
    loss = mnist_loss(preds, yb)
    loss.backward()<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>def train_epoch(model, lr, params):
    for xb, yb in dl:
        calc_grad(xb,yb,model)
        for p in params:
            p.data -= p.grad*lr
            p.grad.zero_()<jupyter_output><empty_output><jupyter_text>Batch Accuracy<jupyter_code>def batch_accuracy(preds, yb):
    preds = preds.sigmoid() #note
    correct = (preds > 0.5).float() == yb
    return correct.float().mean()
batch_accuracy(linear1(batch), train_y[:4])
def validate_epoch(model):
    accs = [batch_accuracy(model(xb), yb) for xb, yb in valid_dl]
    return round(torch.stack(accs).mean().item(), 4)
validate_epoch(linear1)<jupyter_output><empty_output><jupyter_text>Train one epoch<jupyter_code>lr = 1

weights = init_params((28*28,1))
bias = init_params(1)
params = weights, bias

train_epoch(linear1, lr, params)
validate_epoch(linear1)<jupyter_output><empty_output><jupyter_text>Train multiple epochs<jupyter_code>lr = 1

weights = init_params((28*28,1))
bias = init_params(1)
params = weights, bias

for i in range(20):
    train_epoch(linear1, lr, params)
    print(validate_epoch(linear1), end=' ')<jupyter_output>0.6765 0.8387 0.8988 0.9267 0.9374 0.9462 0.954 0.9565 0.9599 0.9618 0.9638 0.9653 0.9667 0.9682 0.9682 0.9706 0.9711 0.9721 0.9721 0.9721<jupyter_text>Refactor  Replace gradient update with a home grown optimizer<jupyter_code>class BasicOptim(nn.Module):
    def __init__(self,params,lr): self.params, self.lr = list(params), lr
        
    def step(self,*args,**kwargs): 
        for p in self.params: p.data -= p.grad*self.lr
            
    def zero_grad(self,*args,**kwargs): 
        for p in self.params: p.grad = None
lr = 1

weights = init_params((28*28,1))
bias = init_params(1)
params = weights, bias

opt = BasicOptim(params, lr)

def train_epoch(model, lr, params):
    for xb, yb in dl:
        calc_grad(xb,yb,model)
        opt.step()
        opt.zero_grad()

for i in range(20):
    train_epoch(linear1, lr, params)
    print(validate_epoch(linear1), end=' ')<jupyter_output>0.6269 0.8379 0.9174 0.9409 0.9496 0.9545 0.9589 0.9623 0.9658 0.9672 0.9687 0.9697 0.9692 0.9697 0.9721 0.9721 0.9721 0.9726 0.9726 0.9731<jupyter_text>Replace init_params and linear1 with Pytorch nn.Linear nn.Linear holds both the weights and bias and takes care of initializing the parameters.<jupyter_code>linear_model = nn.Linear(28*28,1)
weights, bias = linear_model.parameters()
weights.shape, bias.shape
lr = 1

linear_model = nn.Linear(28*28,1)
opt = BasicOptim(linear_model.parameters(), lr)

def train_epoch(model, lr, params):
    for xb, yb in dl:
        calc_grad(xb,yb,model)
        opt.step()
        opt.zero_grad()

for i in range(20):
    train_epoch(linear_model, lr, params)
    print(validate_epoch(linear_model), end=' ')<jupyter_output>0.4932 0.4932 0.6816 0.8687 0.9185 0.936 0.9502 0.958 0.9638 0.9658 0.9678 0.9697 0.9712 0.9741 0.9746 0.9761 0.9765 0.9775 0.9785 0.9785<jupyter_text>Replace home grown optimizer with fastai SGD<jupyter_code>lr = 1

linear_model = nn.Linear(28*28,1)
opt = SGD(linear_model.parameters(), lr)

def train_epoch(model, lr, params):
    for xb, yb in dl:
        calc_grad(xb,yb,model)
        opt.step()
        opt.zero_grad()

for i in range(20):
    train_epoch(linear_model, lr, params)
    print(validate_epoch(linear_model), end=' ')<jupyter_output>0.4932 0.8447 0.8398 0.9126 0.9336 0.9478 0.9551 0.9629 0.9658 0.9678 0.9692 0.9717 0.9741 0.9751 0.9761 0.9761 0.977 0.978 0.9785 0.9785<jupyter_text>Replace training loop with Fastai Learner.fit Observe how we are able to pass in the mnist_loss and batch_accuracy functions. Recall that within these functions we pass the predictions through the sigmoid function.<jupyter_code>#collapse-output
dls = DataLoaders(dl, valid_dl)
learn = Learner(dls, nn.Linear(28*28,1), loss_func=mnist_loss
                , opt_func=SGD
                , metrics = batch_accuracy)
learn.fit(20,lr=1)<jupyter_output><empty_output><jupyter_text>Add a rectified linear unit<jupyter_code>#collapse-output
simple_net = nn.Sequential(nn.Linear(28*28,30), 
                           nn.ReLU(), 
                           nn.Linear(30,1))

learn = Learner(dls, simple_net, loss_func=mnist_loss
                  , opt_func=SGD
                  , metrics=batch_accuracy)


learn.fit(n_epoch=40, lr=0.1)<jupyter_output><empty_output><jupyter_text>learn.recorder records the output from the training process. The three items recorded here are train_loss, valid_loss and batch_accuracy<jupyter_code>learn.recorder.values[:2]<jupyter_output><empty_output><jupyter_text>Plot how the accuracy evolved during the training.<jupyter_code>plt.plot(L(learn.recorder.values).itemgot(2))<jupyter_output><empty_output><jupyter_text>The final accuracy is as below:<jupyter_code>learn.recorder.values[-1][2]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1300681154719429, 'fraction_numerical': 0.06087144556168234, 'mean_word_length': 5.564939673527324, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '600608', 'license': 'Apache-2.0', 'n_tokens_mistral': 3799, 'n_tokens_neox': 3481, 'n_words': 675, 'path': '_notebooks/2021-08-06-MNIST-Threes-And-Sevens.ipynb', 'repo_name': 'nasheqlbrm/blog', 'size': '9249'}