<jupyter_start><jupyter_text>Logistic Regression on MNIST Database using Pytorch<jupyter_code>import torch 
import torchvision
from torchvision.datasets import MNIST<jupyter_output><empty_output><jupyter_text>Working with ImageWe'll use the famous [*MNIST Handwritten Digits Database*](http://yann.lecun.com/exdb/mnist/) as our training dataset. It consists of 28px by 28px grayscale images of handwritten digits (0 to 9) and labels for each image indicating which digit it represents. Here are some sample images from the dataset:![mnist-sample](https://i.imgur.com/CAYnuo1.jpg)  Data Preprocessing<jupyter_code># download dataset - pytorch will automatically converted them to a Pytorch Dataset object

dataset = MNIST(root='data/', download=True)
len(dataset)
# creating test_dataset using train=False

test_dataset = MNIST(root='data/', train=False)
len(test_dataset)
# sample element

dataset[0]   # it gives :- image(part of pillow), 5 (label digit)
import matplotlib.pyplot as plt  #indicates to jupyter we want to plot graphs within the notebook, without this it will show in popup
%matplotlib inline                
image, label = dataset[0]
plt.imshow(image, cmap='gray')
print('Label:', label)
image, label = dataset[10]
plt.imshow(image, cmap='gray')
print('Label:', label)<jupyter_output>Label: 3<jupyter_text>Pytorch dosen't know how to work with images, we need to convert images to tensors by specifying transform while creating our dataset<jupyter_code>import torchvision.transforms as transform
# dataset of tensors

dataset = MNIST(root='data/',
                train=True,
               transform=transform.ToTensor())
img_tensor, label = dataset[0]
print(img_tensor.shape,label)
img_tensor
# some sample values insode the tensor

print(img_tensor[:,10:15,10:15])
print(torch.max(img_tensor),torch.min(img_tensor))
# plot image by passing 28x28 matrix

plt.imshow(img_tensor[0,10:15,10:15], cmap='gray')<jupyter_output><empty_output><jupyter_text>Training and Validation Datasets1. **Training set** - used to train the model, i.e., compute the loss and adjust the model's weights using gradient descent.2. **Validation set** - used to evaluate the model during training, adjust hyperparameters (learning rate, etc.), and pick the best version of the model.3. **Test set** - used to compare different models or approaches and report the model's final accuracy.In the MNIST dataset, there are 60,000 training images and 10,000 test images. The test set is standardized so that different researchers can report their models' results against the same collection of images. Since there's no predefined validation set, we must manually split the 60,000 images into training and validation datasets. Let's set aside 10,000 randomly chosen images for validation. We can do this using the `random_spilt` method from PyTorch.<jupyter_code># manually split training set to 50000-10000 validation set

# using random_split

from torch.utils.data import random_split

train_ds, val_ds = random_split(dataset, [50000, 10000])
len(train_ds), len(val_ds)
# Creating Data_loaders to helpus load data in batches - size=128

from torch.utils.data import DataLoader

batch_size = 128

train_loader = DataLoader(train_ds, batch_size, shuffle=True) # shuffle=True for the training data loader to ensure batches generated in each epoch are different  # randomization helps generalize and speed=up the training process
val_loader = DataLoader(val_ds, batch_size )<jupyter_output><empty_output><jupyter_text>Model* A **logistic regression** model is almost identical to a linear regression model. It contains weights and bias matrices, and the output is obtained using simple matrix operations (`pred = x @ w.t() + b`). * As we did with linear regression, we can use `nn.Linear` to create the model instead of manually creating and initializing the matrices.* Since `nn.Linear` expects each training example to be a vector, each `1x28x28` image tensor is _flattened_ into a vector of size 784 `(28*28)` before being passed into the model. * The output for each image is a vector of size 10, with each element signifying the probability of a particular target label (i.e., 0 to 9). The predicted label for an image is simply the one with the highest probability.<jupyter_code>import torch.nn as nn

input_size = 28*28
num_classes = 10

#Logistic regression model

model = nn.Linear(input_size, num_classes)
print(model.weight.shape)
model.weight
print(model.bias.shape)
model.bias<jupyter_output>torch.Size([10])<jupyter_text>so total parameters = 7850<jupyter_code># taking 1st batch of 100 images from our bataset and pass them into our model

for images, labels in train_loader:
    print(labels)
    print(images.shape)
    outputs = model(images)
    print(outputs)
    break<jupyter_output>tensor([7, 2, 1, 5, 4, 8, 6, 0, 0, 5, 3, 7, 9, 0, 8, 4, 5, 2, 5, 9, 3, 0, 6, 6,
        4, 9, 2, 1, 5, 9, 3, 3, 4, 1, 0, 4, 2, 0, 0, 5, 6, 8, 9, 7, 6, 0, 4, 1,
        2, 1, 9, 2, 6, 8, 5, 2, 8, 7, 7, 9, 7, 6, 7, 5, 9, 5, 5, 9, 1, 7, 5, 0,
        7, 7, 2, 5, 0, 8, 1, 1, 7, 8, 7, 4, 7, 3, 1, 6, 4, 9, 9, 3, 9, 3, 5, 6,
        6, 5, 6, 4, 2, 4, 5, 1, 8, 7, 0, 3, 9, 9, 5, 2, 7, 6, 9, 8, 5, 1, 3, 2,
        3, 6, 4, 1, 3, 9, 7, 3])
torch.Size([128, 1, 28, 28])<jupyter_text>error occurred bcoz, input data dosen't have right shape. images are of the shape 1x28x28, but we need them to be vectors of size 784 ie we need to flatten them --> using .reshape<jupyter_code>images.shape
images.reshape(128,784).shape<jupyter_output><empty_output><jupyter_text>To include this additional function within out model, we need to define a custom model by extending the nn.Module classInside the `__init__` constructor method, we instantiate the weights and biases using `nn.Linear`. And inside the `forward` method, which is invoked when we pass a batch of inputs to the model, we flatten the input tensor and pass it into `self.linear`.`xb.reshape(-1, 28*28)` indicates to PyTorch that we want a *view* of the `xb` tensor with two dimensions. The length along the 2nd dimension is 28\*28 (i.e., 784). One argument to `.reshape` can be set to `-1` (in this case, the first dimension) to let PyTorch figure it out automatically based on the shape of the original tensor.Note that the model no longer has `.weight` and `.bias` attributes (as they are now inside the `.linear` attribute), but it does have a `.parameters` method that returns a list containing the weights and bias.<jupyter_code># extending nn.Module class

class MnistModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
        
    def forward(self, xb):
        xb = xb.reshape(-1, 784)
        out = self.linear(xb)
        return out
    
    
model = MnistModel() #object
model.linear
print(model.linear.weight.shape, model.linear.bias.shape)
list(model.parameters()) # bundle all weights and biases
# we can use custom model in same as before 

for images, labels in train_loader:
    outputs = model(images)
    break
    
print('output.shape :', outputs.shape)
print('Sample outputs layer 0 :', outputs[0].data)
print('Sample outputs layer 0 and 1 :', outputs[:2].data)<jupyter_output>output.shape : torch.Size([128, 10])
Sample outputs layer 0 : tensor([-0.1530, -0.4023,  0.0156, -0.0506,  0.0551, -0.2542,  0.0123, -0.3361,
        -0.1795,  0.1995])
Sample outputs layer 0 and 1 : tensor([[-0.1530, -0.4023,  0.0156, -0.0506,  0.0551, -0.2542,  0.0123, -0.3361,
         -0.1795,  0.1995],
        [-0.2933, -0.3964,  0.2764, -0.0050, -0.0035, -0.0671,  0.2888, -0.2759,
         -0.1555,  0.0238]])<jupyter_text>For each of the 100 input images, we get 10 outputs, one for each class. As discussed earlier, we'd like these outputs to represent probabilities. Each output row's elements must lie between 0 to 1 and add up to 1, which is not the case. To convert the output rows into probabilities, we use the softmax function, which has the following formula:![softmax](https://i.imgur.com/EAh9jLN.png)First, we replace each element `yi` in an output row by `e^yi`, making all the elements positive. ![](https://www.montereyinstitute.org/courses/DevelopmentalMath/COURSE_TEXT2_RESOURCE/U18_L1_T1_text_final_6_files/image001.png)Then, we divide them by their sum to ensure that they add up to 1. The resulting vector can thus be interpreted as probabilities.we'll use the implementation that's provided within PyTorch because it works well with multidimensional tensors (a list of output rows in our case).<jupyter_code>import torch.nn.functional as F
outputs[0:2]
probs = F.softmax(outputs, dim=1) # apply softmax for each output row # see why dim=0 dosen't workout

print("sample probabilities:\n", probs[:2].data) #sample probs

print("Sum: ", torch.sum(probs[0]).item()) # addup probs of an output row
# predicted label for each index is index of element with max probab

max_probs, preds = torch.max(probs, dim=1)

print(preds)
print(max_probs)
# we have not trained yet

labels<jupyter_output><empty_output><jupyter_text>Evaluation Metric and Loss Function<jupyter_code>outputs[:2]
preds == labels
torch.sum(preds == labels) # this is telling how many out of 128 were get predicted correctly<jupyter_output><empty_output><jupyter_text>Accuracy of predictions<jupyter_code>def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
accuracy(outputs,labels)<jupyter_output><empty_output><jupyter_text>The `==` operator performs an element-wise comparison of two tensors with the same shape and returns a tensor of the same shape, containing `True` for unequal elements and `False` for equal elements. Passing the result to `torch.sum` returns the number of labels that were predicted correctly. Finally, we divide by the total number of images to get the accuracy. Note that we don't need to apply softmax to the outputs since its results have the same relative order. This is because `e^x` is an increasing function, i.e., if `y1 > y2`, then `e^y1 > e^y2`. The same holds after averaging out the values to get the softmax.Let's calculate the accuracy of the current model on the first batch of data.   Accuracy is an excellent way for us (humans) to evaluate the model. However, we can't use it as a loss function for optimizing our model using gradient descent for the following reasons:1. It's not a differentiable function. `torch.max` and `==` are both non-continuous and non-differentiable operations, so we can't use the accuracy for computing gradients w.r.t the weights and biases.2. It doesn't take into account the actual probabilities predicted by the model, so it can't provide sufficient feedback for incremental improvements. For these reasons, accuracy is often used as an **evaluation metric** for classification, but not as a loss function. A commonly used loss function for classification problems is the **cross-entropy**, which has the following formula:![cross-entropy](https://i.imgur.com/VDRDl1D.png)While it looks complicated, it's actually quite simple:* For each output row, pick the predicted probability for the correct label. E.g., if the predicted probabilities for an image are `[0.1, 0.3, 0.2, ...]` and the correct label is `1`, we pick the corresponding element `0.3` and ignore the rest.* Then, take the [logarithm](https://en.wikipedia.org/wiki/Logarithm) of the picked probability. If the probability is high, i.e., close to 1, then its logarithm is a very small negative value, close to 0. And if the probability is low (close to 0), then the logarithm is a very large negative value. We also multiply the result by -1, which results is a large postive value of the loss for poor predictions.![](https://www.intmath.com/blog/wp-content/images/2019/05/log10.png)* Finally, take the average of the cross entropy across all the output rows to get the overall loss for a batch of data.Unlike accuracy, cross-entropy is a continuous and differentiable function. It also provides useful feedback for incremental improvements in the model (a slightly higher probability for the correct label leads to a lower loss). These two factors make cross-entropy a better choice for the loss function.As you might expect, PyTorch provides an efficient and tensor-friendly implementation of cross-entropy as part of the `torch.nn.functional` package. Moreover, it also performs softmax internally, so we can directly pass in the model's outputs without converting them into probabilities.<jupyter_code>probs
loss_fn = F.cross_entropy
# loss for current batch of data

loss = loss_fn(outputs, labels)
print(loss)<jupyter_output>tensor(2.3150, grad_fn=<NllLossBackward>)<jupyter_text>Training the model<jupyter_code>def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    optimizer = opt_func(model.parameters(), lr)
    history = [] # for reading epoch-wise results
    
    for epoch in range(epochs):
        
        #training phase
        
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step() # change the gradients using the learning rate
            optimizer.zero_grad()
            
            
        #validation phase
        
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
        
        
    return history
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader] #list comprehension 
    return model.validation_epoch_end(outputs)<jupyter_output><empty_output><jupyter_text>Gaps to fill-in are : training_step, validation_step, validation_epoch_end, epoch_end used by : fit, evaluate<jupyter_code>class MnistModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
        
    def forward(self, xb):
        xb = xb.reshape(-1, 784)
        out = self.linear(xb)
        return out
    
    def training_step(self, batch):
        images, labels = batch
        out = self(images)                         # Generate predictions
        loss = F.cross_entropy(out, labels)        # Calculate loss
        return loss
    
    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                          # Generate predictions
        loss = F.cross_entropy(out, labels)         #loss
        acc = accuracy(out, labels)                 # clac accuracy
        return {'val_loss': loss, 'val_acc': acc}
    
    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss  = torch.stack(batch_losses).mean()  #combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_accs = torch.stack(batch_accs).mean()     #combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_accs.item()}
    
    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))
        
        
model = MnistModel()
result0 = evaluate(model, val_loader)
result0<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>history1 = fit(5, 0.001, model, train_loader, val_loader)
history2 = fit(5, 0.001, model, train_loader, val_loader)
history3 = fit(5, 0.001, model, train_loader, val_loader)
history4 = fit(5, 0.001, model, train_loader, val_loader)<jupyter_output>Epoch [0], val_loss: 0.7295, val_acc: 0.8467
Epoch [1], val_loss: 0.7123, val_acc: 0.8480
Epoch [2], val_loss: 0.6966, val_acc: 0.8495
Epoch [3], val_loss: 0.6822, val_acc: 0.8513
Epoch [4], val_loss: 0.6691, val_acc: 0.8527<jupyter_text>Visualization<jupyter_code>history = [result0] + history1 + history2 + history3 + history4
accuracies = [result['val_acc'] for result in history]
plt.plot(accuracies, '-x')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.title('Accuarcy vs No of epochs');<jupyter_output><empty_output><jupyter_text>Testing on Individual Images<jupyter_code>from torchvision import transforms
test_dataset = MNIST(root='data/',
                     train = False,
                     transform = transforms.ToTensor())
# sample image from the dataset

img, label = test_dataset[0]
plt.imshow(img[0], cmap='gray')
print('shape', img.shape)
print('label',label)
# predict_image function to predict single label 

def predict_image(img, model):
    xb = img.unsqueeze(0)
    yb = model(xb)
    
    _, preds = torch.max(yb, dim=1)
    return preds[0].item()<jupyter_output><empty_output><jupyter_text>`img.unsqueeze` simply adds another dimension at the begining of the 1x28x28 tensor, making it a 1x1x28x28 tensor, which the model views as a batch containing a single image.<jupyter_code>img, label = test_dataset[0]
plt.imshow(img[0], cmap='gray')
print('label:',label, ', predicted:',predict_image(img, model))
img, label = test_dataset[1]
plt.imshow(img[0], cmap='gray')
print('label:',label, ', predicted:',predict_image(img, model))
img, label = test_dataset[5]
plt.imshow(img[0], cmap='gray')
print('label:',label, ', predicted:',predict_image(img, model))
img, label = test_dataset[193]
plt.imshow(img[0], cmap='gray')
print('label:',label, ', predicted:',predict_image(img, model))
img, label = test_dataset[1839]
plt.imshow(img[0], cmap='gray')
print('label:',label, ', predicted:',predict_image(img, model))         # Here Model in Breaking Up
# overall loss and accuracy of model on test set

test_loader = DataLoader(test_dataset, batch_size=256)
result = evaluate(model, test_loader)
result<jupyter_output><empty_output><jupyter_text>Saving and Loading the Model<jupyter_code>torch.save(model.state_dict(), 'mnist-logistic.pth') # .state_dict returns Ordered Dict containig all the weights and bias matrices mapped to right attributes of the model
model.state_dict()
# To load the model weights

model_2 = MnistModel()
model_2.load_state_dict(torch.load('mnist-logistic.pth'))
model_2.state_dict()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10237003480470692, 'fraction_numerical': 0.035357162587702336, 'mean_word_length': 4.285255474452555, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 75, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '244037', 'license': 'Apache-2.0', 'n_tokens_mistral': 6039, 'n_tokens_neox': 5527, 'n_words': 2054, 'path': '_notebooks/2021-06-01-Logistic-MNIST-Pytorch.ipynb', 'repo_name': 'mr-siddy/ML-blog', 'size': '18101'}