<jupyter_start><jupyter_text>Working With The CIFAR10 Dataset This is it. You've seen how to define a simple convolutional neural network, compute loss w.r.t. the graph Variables, and make gradient updates manually and with `torch.nn.optim` package.Now you might be thinking: What about the data?Generally, when you have to deal with image, text, audio or video data, you can use standard python packages that load data into a numpy array. Then you can convert this array into a `torch.*Tensor`.- For images, packages such as `Pillow`, `OpenCV` are useful.- For audio, packages such as `scipy` and `librosa`.- For text, either raw Python or Cython based loading, or `NLTK` and `SpaCy` are useful.Specifically for [Computer vision](), the creators of pytorch have generously created a package called `torchvision`, that has data loaders for common datasets such as Imagenet, CIFAR10, MNIST, etc. and data transformers for images, viz., `torchvision.datasets` and `torch.utils.data.DataLoader`. This provides a huge convinence from writing boiler plate code.We will use the **CIFAR10 dataset**. It has the classes: *‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’*. The images in CIFAR-10 are of size `3x32x32`, i.e. 3-channel color images of `32x32` pixels in size.![CIFAR10 Dataset](../images/cifar10.png) Training an image classifierWe will do the following steps in order:1. Load and normalizing the CIFAR10 training and test datasets using `torchvision`.2. Define a Convolution Neural Network.3. Define a loss function.4. Train the network on the training data.5. Test the network on the test data. 1. Loading and normalizing CIFAR10Using `torchvision`, it’s extremely easy to load CIFAR10.<jupyter_code>%matplotlib inline

# file manipulation
import os.path

# arrays and visualization
import numpy as np
import matplotlib.pyplot as plt

# pytorch imports
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.autograd import Variable

# Special package provided by pytorch
import torchvision
import torchvision.transforms as transforms<jupyter_output><empty_output><jupyter_text>Let's define some *Hyperparameters* we're gonna need later on.<jupyter_code>## Hyperparameters.


# image channel 3=RGB, 1=Grayscale
img_channels = 3

# Class labels.
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
num_classes = len(classes)

# Data directory.
data_dir = '../datasets'  # Dataset directory.
download = True           # Download dataset iff not already downloaded.
normalize = True          # Maybe normalize training data.

# Training parameters
batch_size = 16  # Mini-batch size.
lr = 1e-2        # Optimizer's learning rate.
epochs = 5       # Number of full passes over entire dataset.<jupyter_output><empty_output><jupyter_text>The output of the `torchvision` dataset are PILImage images of range [0, 1]. We transform them to Tensors of normalized range [-1, 1].Define the data directory, i.e. where the data should be downloaded to. With the use of `os.path` module.**NOTE:** `data_dir` could be modified to fit your use.<jupyter_code># Should normalize images or not.
# Normalization helps convergence.
if normalize:
    # Transform rule: Convert to Tensor, Normalize images in range -1 to 1.
    transform = transforms.Compose([transforms.ToTensor(), 
                                    transforms.Normalize((0.5, 0.5, 0.5), 
                                                         (0.5, 0.5, 0.5))])
else:
    # Transform rule: Convert to Tensor without normalizing image
    transform = transforms.Compose([transforms.ToTensor()])

# Download the training set and apply the transform rule to each.
trainset = torchvision.datasets.CIFAR10(root=data_dir, train=True, download=download, transform=transform)
# Load the training set into mini-batches and shuffle them
trainset = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=2)

# Download the testing set and apply the transform rule to each.
testset = torchvision.datasets.CIFAR10(root=data_dir, train=False, download=download, transform=transform)
# Load the testing set into mini-batches and shuffle them as well.
testset = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=True, num_workers=2)
# Helper function to plot images and labels
def imshow(images, labels, pred=None, smooth=True):
    images = images / 2 + 0.5 if normalize else images
    # Create figure with sub-plots.
    fig, axes = plt.subplots(4, 4)

    # Adjust vertical spacing if we need to print ensemble and best-net.
    wspace, hspace = 0.2, 0.8 if pred is not None else 0.4
    fig.subplots_adjust(hspace=hspace, wspace=wspace)

    for i, ax in enumerate(axes.flat):
        # Interpolation type.
        smooth = 'spline16' if smooth else 'nearest'

        # Plot image.
        ax.imshow(np.transpose(images[i], (1, 2, 0)), interpolation=smooth)
            
        # Name of the true class.
        labels_name = classes[labels[i]]

        # Show true and predicted classes.
        if pred is None:
            xlabel = f'True: {labels_name}'
        else:
            # Name of the predicted class.
            pred_name = classes[pred[i]]
            
            xlabel = f'True: {labels_name}\nPred: {pred_name}'

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)
        
        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])
    
    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


# Visualization function to visualize dataset.
def visualize(data, smooth=False):
    # Iterate over the data.
    data_iter = iter(data)
    
    # Unpack images and labels.
    images, labels = data_iter.next()
    
    # Free up memory
    del data_iter
    
    # Call to helper function for plotting images.
    imshow(images, labels=labels, smooth=smooth)



# Let's visualize some training set.
visualize(trainset)<jupyter_output><empty_output><jupyter_text>2. Define a Convolution Neural NetworkIt's time to define our neural network. You've already seen how to define a simple convolutional neural network in the last section. But this time, instead of a single color channel, we have 3-color channels, because the CIFAR10 dataset contains colored images.<jupyter_code>class Network(nn.Module):
    
    def __init__(self, **kwargs):
        super(Network, self).__init__()
        
        # Hyper-parameters
        self._img_channels = kwargs.get('img_channels')
        self._num_classes = kwargs.get('num_classes')
        
        # 2 convolutional & 3 fully connected layers
        self.conv1 = nn.Conv2d(self._img_channels, 16, 2)
        self.conv2 = nn.Conv2d(16, 32, 2)
        flatten_size = self.conv2.out_channels * 7 * 7
        self.fc1 = nn.Linear(flatten_size, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, self._num_classes)
    
    def forward(self, x):
        # Convolutional layers
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        # Flatten layer
        x = x.view(-1, self._flatten(x))
        # Fully connected layers
        x = F.relu(self.fc1(x))     # relu + linear
        x = F.dropout(x, p=0.2)     # 20% dropout
        x = F.relu(self.fc2(x))     # relu + linear
        # Output layer
        x = self.fc3(x)             # linear
        return x
    
    def _flatten(self, x):
        size = x.size()[1:]  # input shape excluding batch dim.
        return torch.Tensor(size).numel()
# Instantiate the network and pass in our parameters.
net = Network(img_channels=img_channels, num_classes=len(classes))<jupyter_output><empty_output><jupyter_text>3. Define a Loss function and optimizerLet’s use a Classification Cross-Entropy loss and Adam optimizer.<jupyter_code># Loss function criterion
loss_func = nn.CrossEntropyLoss()
# Adam optimizer
optimizer = optim.Adam(net.parameters(), lr=lr)<jupyter_output><empty_output><jupyter_text>4. Train the NetworkThis is when things start to get interesting. We simply have to loop over our data iterator, and feed the inputs to the network and optimize it.<jupyter_code># Loop over the data multiple times.
for epoch in range(epochs):

    # Loop through the training dataset (batch by batch).
    for i, data in enumerate(trainset):
        
        # Get the inputs and labels.
        inputs, labels = data
        
        # Wrap them in Variable (explained in section 2).
        inputs, labels = Variable(inputs), Variable(labels)
        
        # Zero the optimizer gradient buffer
        # to prevent gradient accumulation.
        optimizer.zero_grad()
        
        # Forward and backward propagation.
        outputs = net(inputs)
        loss = loss_func(outputs, labels)
        loss.backward()
        
        # Update learnable parameters w.r.t the loss.
        optimizer.step()
        
        # Print statistics.
        print(f'\rEpoch: {epoch+1:,}\tIter: {i+1:,}\tLoss: {loss.data[0]:.4f}', end='')

    # Line break.
    print()


print('\nFinished training!')<jupyter_output>Epoch: 1	Iter: 3,125	Loss: 1.7392
Epoch: 2	Iter: 3,125	Loss: 1.8330
Epoch: 3	Iter: 3,125	Loss: 2.0041
Epoch: 4	Iter: 3,125	Loss: 1.7069
Epoch: 5	Iter: 3,125	Loss: 1.5338

Finished training!<jupyter_text>5. Test the network on the test dataWe have trained the network for 5 epochs (passes over the training data). Let's check if the network has learnt anything.How we check this is by comparing the ground-truth labels over the one the network predicted. We'll keep track of the ones predicted correctly by creating a list, and appending to the list if the prediction was the same as the ground-truth.Alright, that been said, let's familarize ourselves with the data one more time by plotting a few from the `testset`.<jupyter_code># Look at some test data.
visualize(testset)<jupyter_output><empty_output><jupyter_text>Okay, now let's make some predictions with our network.<jupyter_code># Let's make some predictions on the testset.
test_iter = iter(testset)
images, labels = test_iter.next()

# Convert images to `autograd.Variable` 
# before passing through the network.
output = net(Variable(images))<jupyter_output><empty_output><jupyter_text>The outputs are "energies" for the 10 classes. *Higher the energy for a class, the more the network thinks that the image is of the particular class*. So, let’s get the index of the highest energy:<jupyter_code># torch.max returns a tuple: (value, index)
# Take the argmax of the predicted output.
_, predictions = torch.max(output.data, dim=1)

# Visualize the predictions.
imshow(images, labels=labels, pred=predictions, smooth=True)<jupyter_output><empty_output><jupyter_text>Maybe not so bad, huh? Let's see how the result performs on the entire dataset.<jupyter_code># Keep track of correct prediction and total.
correct, total = 0, 0

# Looping through the testset.
for data in testset:
    
    # Unpack the images and labels
    # from each mini-batch.
    images, labels = data
    
    # Pass image through the network
    # to make predictions.
    outputs = net(Variable(images))
    
    # Pretrieve the index with maximum score.
    _, pred = torch.max(outputs.data, dim=1)
    
    # Get the batch size and add on to the total.
    total += labels.size(0)
    
    # Count the number of correct predictions.
    # pred == outputs means where the predictions
    # equals to the ground-truth.
    correct += (pred == labels).sum()


# Print the accuracy on the testset.
print('Accuracy on testset = {:.2%}'.format(correct/total))<jupyter_output>Accuracy on testset = 46.46%<jupyter_text>Well, that's slightly better than random guessing, which is 10% accuracy (randomly picking a class out of 10 classes). Seems like the network learnt something.Hmmm, what are the classes that performed well, and the classes that did not perform well:<jupyter_code># Each index in the `correct_class` stores
# the correct classification for that class;
# while `total_class` stores the total number
# of times we go through the class.
correct_class = torch.zeros(10)
total_class = torch.zeros(10)

# Loop through all dataset
# one batch at a time.
for data in testset:
    # Get the current batch images and labels
    images, labels = data
    
    # Pass the images through the network
    outputs = net(Variable(images))
    
    # Take the index of the maximum scores
    # returned by the network.
    _, pred = torch.max(outputs.data, dim=1)
    
    # Where the pred equals the labels will
    # return 1; and 0 otherwise.
    correct = (pred == labels).squeeze()
    
    # Loop through the batch labels
    for i, label in enumerate(labels):
        # Add on the correct predictions
        # and total for the current label.
        correct_class[label] += correct[i]
        total_class[label] += 1


# Calculate accuracy and sort in descending order
accuracy = correct_class / total_class
accuracy, _ = torch.sort(accuracy, descending=True)

for i, acc in enumerate(accuracy):
    print(f'Accuracy of {classes[i]} \t = {acc:.2%}')<jupyter_output>Accuracy of plane 	 = 73.20%
Accuracy of car 	 = 62.40%
Accuracy of bird 	 = 61.30%
Accuracy of cat 	 = 58.40%
Accuracy of deer 	 = 49.80%
Accuracy of dog 	 = 47.10%
Accuracy of frog 	 = 35.60%
Accuracy of horse 	 = 31.50%
Accuracy of ship 	 = 23.50%
Accuracy of truck 	 = 21.80%
{'fraction_non_alphanumeric': 0.09223946784922395, 'fraction_numerical': 0.018847006651884702, 'mean_word_length': 3.3382494389227317, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '667172', 'license': 'MIT', 'n_tokens_mistral': 4291, 'n_tokens_neox': 3989, 'n_words': 1508, 'path': '04 - Training a classifier/Working with CIFAR10 dataset.ipynb', 'repo_name': 'victor-iyi/pytorch-examples', 'size': '13530'}