<jupyter_start><jupyter_text>Multi-layer Perceptron on CIFAR10Based on the previous exercise, we will train a multi-layer network, also known as multi-layer perceptron. In this architecture, we will have 'hidden layers', i.e. layers that receive input not from the input directly but from previous layers in the network and that are not directly observed in the output.  Preperation<jupyter_code># load tensorboard extension
%load_ext tensorboard
# import torch and other libraries
import os
import numpy as np
import sklearn.metrics as metrics
import matplotlib.pyplot as plt
from tqdm import trange

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.optim import Adam
from torch.utils.tensorboard import SummaryWriter
# check if we have gpu support
# colab offers free gpus, however they are not activated by default.
# to activate the gpu, go to 'Runtime->Change runtime type'. 
# Then select 'GPU' in 'Hardware accelerator' and click 'Save'
have_gpu = torch.cuda.is_available()
# we need to define the device for torch, yadda yadda
if have_gpu:
    print("GPU is available")
    device = torch.device('cuda')
else:
    print("GPU is not available, training will run on the CPU")
    device = torch.device('cpu')
# we refactor some codes from previous notebooks into a python file 'utils.py'
# to reuse it in later notebooks.
# run this in google colab to get the utils.py file
!wget https://raw.githubusercontent.com/constantinpape/dl-teaching-resources/main/exercises/classification/utils.py
# we will reuse the training function, validation function and
# data preparation from the previous notebook
import utils
cifar_dir = './cifar10'
!cifar2png cifar10 cifar10
categories = os.listdir('./cifar10/train')
categories.sort()<jupyter_output><empty_output><jupyter_text>Multi-layer perceptronIn this execise, we go from a single layer network to a network with multiple layers, also known as multi-layer perceptron, or MLP.We still use fully connected layers (`nn.Linear`), i.e. each neuron in a given layer receives input from all neurons in the previous layer.Imporantly, we apply a non-linearity to all layer outputs.Otherwise, the layers could be collapsed into a single layer by matrix multiplication.<jupyter_code>class MLP(nn.Module):
    def __init__(self, n_pixels, n_classes):
        super().__init__()
        self.n_pixels = n_pixels
        self.n_classes = n_classes
        
        # here, we define the structure of the MLP.
        # it's imporant that we use a non-linearity after each 
        # fully connected layer! Here we use the rectified linear
        # unit, short ReLu
        self.layers = nn.Sequential(
            nn.Linear(n_pixels, 400),
            nn.ReLU(),
            nn.Linear(400, 200),
            nn.ReLU(),
            nn.Linear(200, 100),
            nn.ReLU(),
            nn.Linear(100, 50),
            nn.ReLU(),
            nn.Linear(50, n_classes),
            nn.LogSoftmax(dim=1)
        )
    
    def forward(self, x):
        x = x.view(-1, self.n_pixels)
        x = self.layers(x)
        return x
# instantiate the model
model = MLP(3072, 10)
model.to(device)
# get training and validation data
train_dataset, val_dataset = utils.make_cifar_datasets(cifar_dir)
# instantiate loaders, loss, optimizer and tensorboard

train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=25)

optimizer = Adam(model.parameters(), lr=1.e-4)

loss_function = nn.NLLLoss()
loss_function.to(device)

tb_logger = SummaryWriter('runs/log_mlp')
%tensorboard --logdir runs
# train for a couple of epochs
n_epochs = 6
for epoch in trange(n_epochs):
    utils.train(model, train_loader, loss_function, optimizer,
                device, epoch, tb_logger=tb_logger)
    step = (epoch + 1) * len(train_loader)
    utils.validate(model, val_loader, loss_function,
                   device, step,
                   tb_logger=tb_logger)
# evaluate the model on test data
test_dataset = utils.make_cifar_test_dataset(cifar_dir)
test_loader = DataLoader(test_dataset, batch_size=25)
predictions, labels = utils.validate(model, test_loader, loss_function,
                                     device, step=0, tb_logger=None)
import matplotlib.pyplot as plt
print("Test accuracy:")
accuracy = metrics.accuracy_score(labels, predictions)
print(accuracy)

fig, ax = plt.subplots(1, figsize=(8, 8))
utils.make_confusion_matrix(labels, predictions, categories, ax)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08493696084936961, 'fraction_numerical': 0.011944260119442602, 'mean_word_length': 3.6763185108583247, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '68177', 'license': 'MIT', 'n_tokens_mistral': 1408, 'n_tokens_neox': 1347, 'n_words': 459, 'path': 'exercises/classification/3_multi_layer_perceptron.ipynb', 'repo_name': 'constantinpape/dl-teaching-resources', 'size': '4521'}