<jupyter_start><jupyter_text>Linear Regression for image classification with PyTorchCredits: \https://jovian.ai/aakashns/03-logistic-regression<jupyter_code>import torch
import torchvision
from torchvision.datasets import MNIST
# Download training dataset
dataset = MNIST(root='data/', download=True)
len(dataset)
test_dataset = MNIST(root='data/', train=False)
len(test_dataset)
import matplotlib.pyplot as plt
%matplotlib inline
image, label = dataset[0]
plt.imshow(image, cmap='gray')
print('Label:', label)<jupyter_output>Label: 5<jupyter_text>We need to convert the images into tensors. We can do this by specifying a transform while creating our dataset. PyTorch datasets allow us to specify one or more transformation functions which are applied to the images as they are loaded. `torchvision.transforms` contains many such predefined functions, and we'll use the `ToTensor` transform to convert images into PyTorch tensors.<jupyter_code>import torchvision.transforms as transforms
# MNIST dataset (images and labels)
dataset = MNIST(root='data/', train=True, transform=transforms.ToTensor())
img_tensor, label = dataset[0]
print(img_tensor.shape, label)<jupyter_output>torch.Size([1, 28, 28]) 5<jupyter_text>The image is now converted to a 1x28x28 tensor. The first dimension is used to keep track of the color channels. Since images in the MNIST dataset are grayscale, there's just one channel. Other datasets have images with color, in which case there are 3 channels: red, green and blue (RGB).<jupyter_code>print(img_tensor[:,10:15,10:15])
print(torch.max(img_tensor), torch.min(img_tensor))<jupyter_output>tensor([[[0.0039, 0.6039, 0.9922, 0.3529, 0.0000],
         [0.0000, 0.5451, 0.9922, 0.7451, 0.0078],
         [0.0000, 0.0431, 0.7451, 0.9922, 0.2745],
         [0.0000, 0.0000, 0.1373, 0.9451, 0.8824],
         [0.0000, 0.0000, 0.0000, 0.3176, 0.9412]]])
tensor(1.) tensor(0.)<jupyter_text>Training and Validation Datasets<jupyter_code>from torch.utils.data import random_split

train_ds, val_ds = random_split(dataset, [50000, 10000])
len(train_ds), len(val_ds)<jupyter_output><empty_output><jupyter_text>We can now created data loaders to help us load the data in batches. We'll use a batch size of 128.<jupyter_code>from torch.utils.data import DataLoader

batch_size = 128

train_loader = DataLoader(train_ds, batch_size, shuffle=True)
val_loader = DataLoader(val_ds, batch_size)<jupyter_output><empty_output><jupyter_text>ModelSince `nn.Linear` expects the each training example to be a vector, each `1x28x28` image tensor needs to be flattened out into a vector of size `784 (28*28)`, before being passed into the model.<jupyter_code>import torch.nn as nn

input_size = 28*28
num_classes = 10

# Logistic regression model
model = nn.Linear(input_size, num_classes)
print(model.weight.shape)
model.weight
print(model.bias.shape)
model.bias<jupyter_output>torch.Size([10])<jupyter_text>Our images are of the shape `1x28x28`, but we need them to be vectors of size 784 i.e. we need to flatten them out. We'll use the `.reshape` method of a tensor, which will allow us to efficiently 'view' each image as a flat vector, without really chaging the underlying data. To include this additional functionality within our model, we need to define a custom model, by extending the `nn.Module` class from PyTorch<jupyter_code>class MnistModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
        
    def forward(self, xb):
        xb = xb.reshape(-1, 784)
        out = self.linear(xb)
        return out
    
model = MnistModel()<jupyter_output><empty_output><jupyter_text>Note that the model no longer has `.weight` and `.bias` attributes (as they are now inside the `.linear` attribute), but it does have a `.parameters` method which returns a list containing the weights and bias, and can be used by a PyTorch optimizer.<jupyter_code>print(model.linear.weight.shape, model.linear.bias.shape)
list(model.parameters())
for images, labels in train_loader:
    outputs = model(images)
    break

print('outputs.shape : ', outputs.shape)
print('Sample outputs :\n', outputs[:2].data)<jupyter_output>outputs.shape :  torch.Size([128, 10])
Sample outputs :
 tensor([[ 0.2555, -0.0367,  0.1905,  0.2261, -0.1945,  0.0038, -0.0754, -0.0099,
          0.1901, -0.0313],
        [ 0.1399,  0.2762,  0.2464,  0.0913, -0.1057, -0.0498, -0.1166,  0.0078,
          0.2616, -0.1155]])<jupyter_text>The softmax function is included in the `torch.nn.functional` package, and requires us to specify a dimension along which the softmax must be applied.<jupyter_code>import torch.nn.functional as F

# Apply softmax for each output row
probs = F.softmax(outputs, dim=1)

# Look at sample probabilities
print("Sample probabilities:\n", probs[:2].data)

# Add up the probabilities of an output row
print("Sum: ", torch.sum(probs[0]).item())
max_probs, preds = torch.max(probs, dim=1)
print(preds)
print(max_probs)
labels<jupyter_output><empty_output><jupyter_text>Evaluation Metric and Loss Function<jupyter_code>def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
accuracy(outputs, labels)
loss_fn = F.cross_entropy

# Loss for current batch of data
loss = loss_fn(outputs, labels)
print(loss)<jupyter_output>tensor(2.3698, grad_fn=<NllLossBackward>)<jupyter_text>Training the model<jupyter_code>class MnistModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
        
    def forward(self, xb):
        xb = xb.reshape(-1, 784)
        out = self.linear(xb)
        return out
    
    def training_step(self, batch):
        images, labels = batch 
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss
    
    def validation_step(self, batch):
        images, labels = batch 
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss, 'val_acc': acc}
        
    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}
    
    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))
    
model = MnistModel()
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase 
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history
result0 = evaluate(model, val_loader)
result0
## Train for 5 epochs:
history1 = fit(5, 0.001, model, train_loader, val_loader)
## 5 more epochs:
history2 = fit(5, 0.001, model, train_loader, val_loader)
history3 = fit(5, 0.001, model, train_loader, val_loader)
history4 = fit(5, 0.001, model, train_loader, val_loader)
history = [result0] + history1 + history2 + history3 + history4
accuracies = [result['val_acc'] for result in history]
plt.plot(accuracies, '-x')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.title('Accuracy vs. No. of epochs')<jupyter_output><empty_output><jupyter_text>Testing with individual images<jupyter_code>test_dataset = MNIST(root='data/', train=False, transform=transforms.ToTensor())
img, label = test_dataset[0]
plt.imshow(img[0], cmap='gray')
print('Shape:', img.shape)
print('Label:', label)
img.unsqueeze(0).shape<jupyter_output><empty_output><jupyter_text>`img.unsqueeze` simply adds another dimension at the begining of the `1x28x28` tensor, making it a `1x1x28x28` tensor, which the model views as a batch containing a single image.<jupyter_code>def predict_image(img, model):
    xb = img.unsqueeze(0)
    yb = model(xb)
    _, preds  = torch.max(yb, dim=1)
    return preds[0].item()
img, label = test_dataset[0]
plt.imshow(img[0], cmap='gray')
print('Label:', label, ', Predicted:', predict_image(img, model))
test_loader = DataLoader(test_dataset, batch_size=256)
result = evaluate(model, test_loader)
result<jupyter_output><empty_output><jupyter_text>Saving and loading the model<jupyter_code>torch.save(model.state_dict(), 'mnist-logistic.pth')
model.state_dict()<jupyter_output><empty_output><jupyter_text>To load the model weights, we can instante a new object of the class `MnistModel`, and use the `.load_state_dict method`.<jupyter_code>model2 = MnistModel()
model2.load_state_dict(torch.load('mnist-logistic.pth'))
model2.state_dict()
test_loader = DataLoader(test_dataset, batch_size=256)
result = evaluate(model2, test_loader)
result<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11759773608636412, 'fraction_numerical': 0.040247353526883974, 'mean_word_length': 4.452571428571429, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '501100', 'license': 'MIT', 'n_tokens_mistral': 3401, 'n_tokens_neox': 3154, 'n_words': 861, 'path': 'Tutorial/03-logistic-regression.ipynb', 'repo_name': 'danhtaihoang/pytorch-deeplearning', 'size': '9541'}