<jupyter_start><jupyter_text>Multilayer Perceptrons with Auto-differentiation!![](http://cs231n.github.io/assets/nn1/neural_net.jpeg)* Today we will (again) train a multilayer perceptron!    Differently from the last tutorial we will let the PyTorch library do most of the complicated work for us.* The goal of this notebook is therefore to make you get familiar with what can considered as a general PyTorch cooking recipe, which we will re-use in the coming weeks.* We will be training a multilayer perceptron on the CIFAR-10 dataset, a popular dataset of images that is used in Computer-Vision (CV) research<jupyter_code>import numpy as np
import torch, torchvision
import torch.nn as nn
from torchvision import datasets
import torchvision.transforms as transforms
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>1. Load the dataset* PyTorch has the nice advantgae of providing us with popular CV datasets that can be easily downloaded.    The following script will download the CIFAR-10 dataset for us.    * Since it is computationally very expensive to load the entire dataset into memory, before feeding batches of images to the neural network, you will have to look into PyTorch's dataloaders.<jupyter_code>transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = datasets.CIFAR10(root = "./data", train = True, download = True, transform = transform)
testset = datasets.CIFAR10(root = "./data", train = False, download = True, transform = transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')<jupyter_output>Files already downloaded and verified
Files already downloaded and verified<jupyter_text>Visualize the Images of CIFAR-10* Once everything is properly loaded you should be able to visualize some of the samples of the CIFAR-10<jupyter_code>def show_images(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()
    
dataiter = iter(trainloader)
images, labels = dataiter.next()
show_images(torchvision.utils.make_grid(images))
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))<jupyter_output><empty_output><jupyter_text>2. Creating a MLP model and train itWe are now ready to define all the necessary hyperparameters that are required to construct a proper MLP.Similarly to last week tutorial explore the data carefully in order to know how to set the following parameters!  Hyperparameters<jupyter_code>input_dim = 32 * 32 * 3
hidden_dim = 50
output_dim = 10
learning_rate = 1e-3
num_epochs = 1<jupyter_output><empty_output><jupyter_text>Computation GraphYou are free to define any kind of architecture that you think is reasonable for the problem of classifying CIFAR-10 images. The constructor of the neural network will represent the different components of the computation-graph, note that it is very similar to the way we have programmed our own neural network last week.You will also have to define the forward pass yourself.<jupyter_code>class net(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(net, self).__init__()
        self.input_dim = input_dim
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc3 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = x.view(x.size(0), self.input_dim)
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.relu(x)
        x = self.fc3(x)
        return x<jupyter_output><empty_output><jupyter_text>Create the NetworkYou are now ready to create your neural network object. You will also have to define an appropriate loss function to minimize and an optimization algorithm.<jupyter_code>model = net(input_dim, hidden_dim, output_dim)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.RMSprop(model.parameters(), lr = learning_rate)<jupyter_output><empty_output><jupyter_text>Train the NetworkOnce you have your neural network you just need to train it. You will have to compute the output of the forward pass, compare it to your desired output and update the parameters of the network by backpropagation. Fortunately for you this week this only requires less than 5 lines of code in PyTorch!<jupyter_code>def train(num_epochs):
    for i in range(num_epochs):
        tmp_loss = []
        for (x, y) in trainloader:
            outputs = model(x)
            loss = criterion(outputs, y)
            tmp_loss.append(loss.item())
            optimizer.zero_grad()
            loss.backward()
            #print(model.fc1.weight.grad.norm())
            optimizer.step()

        print("Loss at {}th epoch: {}".format(i, round(np.mean(tmp_loss), 4)))
train(5)<jupyter_output>Loss at 0th epoch: 1.7358
Loss at 1th epoch: 1.6096
Loss at 2th epoch: 1.5543
Loss at 3th epoch: 1.5238
Loss at 4th epoch: 1.5084<jupyter_text>3. Evaluate the final performance of the NetworkLastly once you think your network is properly trained it will be time to test its performance on the testing-set of the CIFAR-10 dataset. Compute the final accuracy of your model.<jupyter_code>def test():
    with torch.no_grad():
        correct = 0
        total = 0
        for inputs, targets in testloader:
            outputs = model(inputs)
            loss = criterion(outputs, targets)
            _, predicted = outputs.max(1)
            total += targets.size(0)
            correct += predicted.eq(targets).sum().item()
        
        print('Accuracy of the model on the testing images: %d %%' % (100 * correct / total))
test()<jupyter_output>Accuracy of the model on the testing images: 46 %
{'fraction_non_alphanumeric': 0.08778563208942956, 'fraction_numerical': 0.017918790070688806, 'mean_word_length': 3.6407322654462244, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '455691', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1867, 'n_tokens_neox': 1762, 'n_words': 653, 'path': 'tutorials/lecture_04/solution_tutorial_04.ipynb', 'repo_name': 'Roger-Li/info8010-deep-learning', 'size': '6083'}