<jupyter_start><jupyter_text>3 - AlexNetIn this notebook we will be implementing a modified version of [AlexNet](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf), a neural network model that uses convolutional neural network (CNN) layers and was designed for the [ImageNet challenge](http://www.image-net.org/challenges/LSVRC/). AlexNet is famous for winning the ImageNet challenge in 2012 by beating the second place competitor by over 10% accuracy and kickstarting the interest in deep learning for computer vision.The image below shows the architecture of AlexNet.![](assets/alexnet.png)Confusingly, there are two "paths" of processing through the network. This is due to the original AlexNet model being implemented on two GPUs in parallel. Almost all implementations of AlexNet are now on a single GPU and our implementation is too.We are now moving on from the MNIST dataset and from now on we will be using the [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. CIFAR10 consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. The classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. ![](https://github.com/bentrevett/pytorch-image-classification/blob/master/assets/cifar10.png?raw=1)We will also show how to intialize the weights of our neural network and how to find a suitable learning rate using a modified version of the [learning rate finder](https://arxiv.org/abs/1506.01186).Like the previous notebooks we'll implement our model, measure its performance on the dataset, and then have a short look into seeing what the model has learned.Most of this notebook will be similar to the previous ones thus we will skim over code that has been shown before. We can look at the previous notebook for a refresher if needed. Data ProcessingAs always, we'll import the modules we need. A new import is the `_LRScheduler` which we will use to implement our learning rate finder.<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import _LRScheduler
import torch.utils.data as data

import torchvision.transforms as transforms
import torchvision.datasets as datasets
# import tensorflow as tf
import tensorboard as tb

from sklearn import decomposition
from sklearn import manifold
from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
import matplotlib.pyplot as plt
import numpy as np
from torch.utils.tensorboard import SummaryWriter
import copy
import random
import time
!ls<jupyter_output>1_mlp.ipynb	       4_vgg.ipynb     assets	       tut3-model.pt
2_lenet.ipynb	       5_resnet.ipynb  init_params.pt
3_alexnet-Copy1.ipynb  LICENSE	       misc
3_alexnet.ipynb        README.md       tensor_log<jupyter_text>We set the random seed so all of our experiments can be reproduced.<jupyter_code>SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True<jupyter_output><empty_output><jupyter_text>We calculate the mean and standard deviation of our data so we can normalize it.Our dataset is made up of color images but three color channels (red, green and blue), compared to MNIST's black and white images with a single color channel. To normalize our data we need to calculate the means and standard deviations for each of the color channels independently. To do this we pass a tuple containing the axes we want to take the means and standard deviations over to the `mean` and `std` functions and we receive a list of means and standard deviations for each of the three color channels.<jupyter_code>ROOT = '.data'

train_data = datasets.CIFAR10(root = ROOT, 
                              train = True, 
                              download = True)

means = train_data.data.mean(axis = (0,1,2)) / 255
stds = train_data.data.std(axis = (0,1,2)) / 255

print(f'Calculated means: {means}')
print(f'Calculated stds: {stds}')<jupyter_output>Files already downloaded and verified
Calculated means: [0.49139968 0.48215841 0.44653091]
Calculated stds: [0.24703223 0.24348513 0.26158784]<jupyter_text>Next up is defining the transforms for data augmentation. The images in the CIFAR10 dataset are significantly more complex than the MNIST dataset. They are larger, have three times the amount of pixels and are more cluttered. This makes them harder to learn and consequently means we should use less augmentation.A new transform we use is `RandomHorizontalFlip`. This, with a probability of `0.5` as specified, flips the image horizontally. So an image of a horse facing to the right will be flipped so it will face to the left. We couldn't do this in the MNIST dataset as we are not expecting our test set to contain any flipped digits, however natural images, such as those in the CIFAR10 dataset, can potentially be flipped as they still make visual sense.As our `means` and `stds` are now already in lists we do not need to enclose them in lists as we did for the single channel images in the MNIST dataset.<jupyter_code>train_transforms = transforms.Compose([
                           transforms.RandomRotation(5),
                           transforms.RandomHorizontalFlip(0.5),
                           transforms.RandomCrop(32, padding = 2),
                           transforms.ToTensor(),
                           transforms.Normalize(mean = means, 
                                                std = stds)
                       ])

test_transforms = transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize(mean = means, 
                                                std = stds)
                       ])<jupyter_output><empty_output><jupyter_text>Next, as standard, we'll load the dataset with our transforms...<jupyter_code>train_data = datasets.CIFAR10(ROOT, 
                              train = True, 
                              download = True, 
                              transform = train_transforms)


test_data = datasets.CIFAR10(ROOT, 
                             train = False, 
                             download = True, 
                             transform = test_transforms)<jupyter_output>Files already downloaded and verified
Files already downloaded and verified<jupyter_text>...create a validation set from our training set...<jupyter_code>VALID_RATIO = 0.9

n_train_examples = int(len(train_data) * VALID_RATIO)
n_valid_examples = len(train_data) - n_train_examples

train_data, valid_data = data.random_split(train_data, 
                                           [n_train_examples, n_valid_examples])<jupyter_output><empty_output><jupyter_text>...and ensure our validation set uses the test transforms.<jupyter_code>valid_data = copy.deepcopy(valid_data)
valid_data.dataset.transform = test_transforms<jupyter_output><empty_output><jupyter_text>We print out the number of examples in each set of data to ensure everything has gone OK so far.<jupyter_code>print(f'Number of training examples: {len(train_data)}')
print(f'Number of validation examples: {len(valid_data)}')
print(f'Number of testing examples: {len(test_data)}')<jupyter_output>Number of training examples: 45000
Number of validation examples: 5000
Number of testing examples: 10000<jupyter_text>Now, we'll create a function to plot some of the images in our dataset to see what they actually look like.Note that by default PyTorch handles images that are arranged `[channel, height, width]`, but `matplotlib` expects images to be `[height, width, channel]`, hence we need to `permute` our images before plotting them.Ignore the `normalize` argument for now, we'll explain it shortly. Then, we'll actually plot the images.We get both the images and the labels from the training set and convert the labels, which are originally stored as integers, into their human readable class by using the data's `classes` dictionary.When we plot them we see lots of warnings. This is because `matplotlib` is expecting the values of every pixel to be between $[0, 1]$, however our normalization will cause them to be outside this range. By default `matplotlib` will then clip these values into the $[0,1]$ range. This clipping causes all of the images to look a bit weird - all of the colors are oversaturated. A solution to this is to *renormalize* the images so each pixel is between $[0,1]$. This is done by clipping the pixel values between the maximum and minimum within an image and then scaling each pixel between $[0,1]$ using these maximum and minimums. As we can see the images below look a lot more like we were expecting, along with the rotations and cropping. We'll be normalizing our images by default from now on, so we'll write a function that does it for us which we can use whenever we need to renormalize an image.<jupyter_code>def normalize_image(image):
    image_min = image.min()
    image_max = image.max()
    image.clamp_(min = image_min, max = image_max)
    image.add_(-image_min).div_(image_max - image_min + 1e-5)
    return image<jupyter_output><empty_output><jupyter_text>As before, we'll check what images look like with Sobel filters applied to them. The filters are still 2-dimensional but they are expanded to a depth of three dimensions inside the `plot_filter` function.Below is a filter which detects horizontal lines. Here's a filter that detects vertical lines. We'll also do the same for subsampling/pooling. As before, the higher filter sizes in the pooling layers means more information is lost, i.e. the image becomes lower resolution. The final bit of the data processing is creating the iterators.We use a much larger batch size here than in previous models. Generally, when using a GPU, a larger batch size means our model trains faster. Our model has significantly more parameters and the images it is training on are much larger, than the previous notebook, so will generally take longer. We offset this as much as we can by using a batch size of 256 instead of 64.<jupyter_code>BATCH_SIZE = 256

train_iterator = data.DataLoader(train_data, 
                                 shuffle = True, 
                                 batch_size = BATCH_SIZE)

valid_iterator = data.DataLoader(valid_data, 
                                 batch_size = BATCH_SIZE)

test_iterator = data.DataLoader(test_data, 
                                batch_size = BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Defining the ModelNext up is defining the model.The actual model itself is no more difficult to understand than the previous model, LeNet. It is made up of convolutional layers, pooling layers and ReLU activation functions. See the previous notebook for a refresher on these concepts. There are only two new concepts introduced here, `nn.Sequential` and `nn.Dropout`.We can think of `Sequential` as like our transforms introduced earlier for data augmentation. We provide `Sequential` with multiple layers and when the `Sequential` module is called it will apply each layer, in order, to the input. There is no difference between using a `Sequential` and having each module defined in the `__init__` and then called in `forward` - however it makes the code significantly shorter.We have one `Sequential` model, `features`, for all of the convolutional and pooling layers, then we flatten then data and pass it to the `classifier`, another `Sequential` model which is made up of linear layers and the second new concept, *dropout*.Dropout is a form of [*regularization*](https://en.wikipedia.org/wiki/Regularization_(mathematics)). As our models get larger, to perform more accurately on richer datasets, they start having a significantly higher number of parameters. The problem with lots of parameters is that our models begin to *overfit*. That is, they do not learn general image features whilst learning to classify images but instead simply memorize images within the training set. This is bad as it will cause our model to achieve poor performance on the validation/testing set. To solve this overfitting problem, we use regularization. Dropout is just one method of regularization, other common ones are *L1 regularization*, *L2 regularization* and *weight decay*.Dropout works by randomly setting a certain fraction, 0.5 here, of the neurons in a layer to zero. This effectively adds noise to the training of the neural network and causes neurons to learn with "less" data as they are only getting half of the information from a previous layer with dropout applied. It can also be thought of as causing your model to learn multiple smaller models with less parameters. Dropout is only applied when the model is training. It needs to be "turned off" when validating, testing or using the model for inference.As mentioned in the previous notebook, during the convolutional and pooling layers the activation function should be placed **after** the pooling layer to reduce computational cost.In the linear layers, dropout should be applied **after** the activation function. Although when using ReLU activation functions the same result is achieved if dropout is before or after, see [here](https://sebastianraschka.com/faq/docs/dropout-activation.html).One last thing to mention is that the very first convolutional layer has an `in_channel` of three. That is because we are handling color images that have three channels (red, green and blue) instead of the single channel grayscale images from the MNIST dataset. This doesn't change the way any of the convolutional filter works, it just means the first filter has a depth of three instead of a depth of one.<jupyter_code>class AlexNet(nn.Module):
    def __init__(self, output_dim):
        super().__init__()
        
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, 3, 2, 1), #in_channels, out_channels, kernel_size, stride, padding
            nn.MaxPool2d(2), #kernel_size
            nn.ReLU(),
            
            nn.Conv2d(64, 192, 3, padding = 1),
            nn.MaxPool2d(2),
            nn.ReLU(),
            
            nn.Conv2d(192, 384, 3, padding = 1),
            nn.ReLU(),
            
            nn.Conv2d(384, 256, 3, padding = 1),
            nn.ReLU(),
            
            nn.Conv2d(256, 256, 3, padding = 1),
            nn.MaxPool2d(2),
            nn.ReLU(),
        )
        
        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(256 * 2 * 2, 4096),
            nn.Sigmoid(),
            nn.Dropout(0.5),
            nn.Linear(4096, 4096),
            nn.Sigmoid(),
            nn.Linear(4096, output_dim),
        )

    def forward(self, x):
        x = self.features(x)
        h = x.view(x.shape[0], -1)
        x = self.classifier(h)
        return x, h<jupyter_output><empty_output><jupyter_text>We'll create an instance of our model with the desired amount of classes.<jupyter_code>OUTPUT_DIM = 10

model = AlexNet(OUTPUT_DIM)<jupyter_output><empty_output><jupyter_text>Then we'll see how many trainable parameters our model has. Our LeNet architecture had ~44k, but here we have 23.2M parameters - and AlexNet is a relatively small model for computer vision.<jupyter_code>def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')<jupyter_output>The model has 22,682,186 trainable parameters<jupyter_text>Training the ModelNext up, we'll initialize the parameters of our model.PyTorch's default initialization is usually fine however by manually trying different initialization schemes we can usually squeeze out a slight performance improvement.We initialize parameters in PyTorch by creating a function that takes in a PyTorch module, checking what type of module it is, and then using the `nn.init` methods to actually initialize the parameters.For our convolutional layers, we'll initialize the weights from a Normal distribution with a standard deviation given by:$$\frac{\text{gain}}{\sqrt{\text{fan mode}}}$$The value of $\text{gain}$ depends on the non-linearity we will be using after the convolutional layer and we simply tell the initialization function that we are using ReLU which sets the gain to $\sqrt{2}$. The fan mode can be either `fan_in` or `fan_out`. `fan_in` is the number of connections coming into the layer and `fan_out` is the number of connections going out of the layer. For the first convolutional layer the input is from 3x3x3 filter, so the `fan_in` is 27 and the output is 64x3x3,  so the `fan_out` is 576. We leave it to the default `fan_in` mode. This initialization scheme is called *Kaiming Normal*, also known as *He Normal*. See the [paper](https://arxiv.org/abs/1502.01852) to learn more about how it was devised.For the linear layers we initialize with a Normal distribution but this time the standard deviation is given by:$$\text{gain} \times \sqrt{\frac{2}{\text{fan_in} + \text{fan_out}}}$$Confusingly, instead of just telling the initialization function which non-linearity we want to use and have it calculate the gain for us, we have to tell it what gain we want to use. Luckily, `nn.init` has a `calculate_gain` function which does that for us, and we just tell it we are using ReLUs. This type of initialize scheme is called *Xavier Normal*, also known as *Glorot Normal*. See the [paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) for the theory behind it. For both types of layer we initialize the bias terms to zeros.Why do we even need to initialize our parameters this way? See [this](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79) article for a great explanation, but the gist of it is that just like how we normalized our input data to have a mean of 0 and a standard deviation of 1, we also want the outputs of each activation function (and therefore the inputs to the subsequent layer) to also have a mean of 0 and a standard deviation of 1. These initialization schemes, by taking account the number of connections in to and out of a layer as well as the non-linearity used, help achieve this normalization effect when initializing weights.<jupyter_code>def initialize_parameters(m):
    if isinstance(m, nn.Conv2d):
        nn.init.kaiming_normal_(m.weight.data, nonlinearity = 'relu')
        nn.init.constant_(m.bias.data, 0)
    elif isinstance(m, nn.Linear):
        nn.init.xavier_normal_(m.weight.data, gain = nn.init.calculate_gain('relu'))
        nn.init.constant_(m.bias.data, 0)<jupyter_output><empty_output><jupyter_text>We apply the initialization by using the model's `apply` method. This will call the given function on every module and sub-module within the model.<jupyter_code>model.apply(initialize_parameters)<jupyter_output><empty_output><jupyter_text>Next up is the learning rate finder. The code here is taken from a stripped down and slightly modified version of the excellent [pytorch-lr-finder](https://github.com/davidtvs/pytorch-lr-finder). **Note**: the learning rate finder is more of an art than a science. It is not going to find an exact learning rate to 10 decimal places which will always give us 100% accuracy - but it is usually going to be better than just picking a learning rate out of thin air.  The most commonly used optimizer used is Adam. Adam's default learning rate is usually a fine choice but, much like how we manually initialized our parameters to potentially get some performance improvement, we can try and calculate an optimal learning rate manually.How does the learning rate finder work? We give the finder our model, optimizer and criterion we want to use. However we give it an optimizer with a much lower learning rate than we are expecting to use. We then train the model on the batches of data from the training set - calculating the loss and updating the parameters. After each batch we increase the learning rate exponentially from the initial, extremely low learning rate to a learning rate we know will be too high. This repeats until our loss diverges (over 5x the best loss achieved) or we reach our defined maximum learning rate. At each batch we are recording the learning rate and the loss achieved on that batch. By plotting them against each other we can find a suitable learning rate - but more on how to do that in a bit.The losses calculated are usually quite noisy so we actually save the exponentially weighted average of the loss calculated. We also want to use our initialized parameters, not the ones obtained by upgrading the parameters when performing the learning rate finder. Hence we save the model parameters to disk when initializing the finder and then they are reset to our desired initialized ones just before the `range_test` function returns by loading the initial values from disk. To prepare to use the range finder we define an initial, very low starting learning rate and then create an instance of the optimizer we want to use with that learning rate.We then define the loss function we want to use, the device we'll use and place our model and criterion on to our device.<jupyter_code>START_LR = 1e-7

# optimizer = optim.Adam(model.parameters(), lr = START_LR)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

criterion = nn.CrossEntropyLoss()

model = model.to(device)
criterion = criterion.to(device)<jupyter_output><empty_output><jupyter_text>Next, we'll finally use the range finder.We first create an instance of the finder class with the model, optimizer, loss function and device. Then we use `range_test` with the training iterator, the maximum learning rate and the number of iterations we want to use.<jupyter_code>END_LR = 10
NUM_ITER = 100

# lr_finder = LRFinder(model, optimizer, criterion, device)
# lrs, losses = lr_finder.range_test(train_iterator, END_LR, NUM_ITER)<jupyter_output><empty_output><jupyter_text>Next, we can plot the learning rate against the loss. As our learning rate was scaled up exponentially we want to plot it on a logarithmic scale. We also do not want to plot the last few values as they are usually where the loss is very high and makes it difficult to examine the graph in detail. You can also skip the first few values as nothing interesting happens there. As we can see, the loss begins flat and then begins to decrease rapidly before reaching a minimum and starting to increase. How can we read this plot and get the optimal learning rate? According to [this](https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html) article, we should look for the loss begins to flatten, this is around $10^{-2}$ below, and then reduce that by a factor of 10, which gives us a found learning rate of $10^{-3}$ or $0.001$. We can now create a new optimizer with our found learning rate.Ironically, the learning rate value we found, $0.001$ is actually Adam's default learning rate!<jupyter_code>FOUND_LR = 1e-3

optimizer = optim.Adam(model.parameters(), lr = FOUND_LR)<jupyter_output><empty_output><jupyter_text>The rest of the notebook is pretty similar to the previous notebooks from these tutorials.We define a function to calculate accuracy...<jupyter_code>def calculate_accuracy(y_pred, y):
    top_pred = y_pred.argmax(1, keepdim = True)
    correct = top_pred.eq(y.view_as(top_pred)).sum()
    acc = correct.float() / y.shape[0]
    return acc<jupyter_output><empty_output><jupyter_text>...and a function to implement our training loop.As we are using dropout we need to make sure to "turn it on" when training by using `model.train()`.<jupyter_code>def train(model, iterator, optimizer, criterion, device, epoch):
    
    epoch_loss = 0
    epoch_acc = 0
    
    model.train()
    count=0
    for (x, y) in iterator:
        
        x = x.to(device)
        y = y.to(device)
        
        optimizer.zero_grad()
                
        y_pred, _ = model(x)
        
        loss = criterion(y_pred, y)
        
        acc = calculate_accuracy(y_pred, y)
        
        loss.backward()
        
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        count+=1
#         writer.add_scalar('Batch/Train/loss', loss, epoch*len(iterator) + count)
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>We also define an evaluation loop, making sure to "turn off" dropout with `model.eval()`.<jupyter_code>def evaluate(model, iterator, criterion, device):
    
    epoch_loss = 0
    epoch_acc = 0
    
    model.eval()
    
    with torch.no_grad():
        
        for (x, y) in iterator:

            x = x.to(device)
            y = y.to(device)

            y_pred, _ = model(x)

            loss = criterion(y_pred, y)

            acc = calculate_accuracy(y_pred, y)

            epoch_loss += loss.item()
            epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)<jupyter_output><empty_output><jupyter_text>Next, we define a function to tell us how long an epoch takes.<jupyter_code>def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs<jupyter_output><empty_output><jupyter_text>Then, finally, we train our model.We get a best validation loss of ~76% accuracy.<jupyter_code>EPOCHS = 25

best_valid_loss = float('inf')

for epoch in range(EPOCHS):
    
    start_time = time.monotonic()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion, device, epoch)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion, device)
        
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut3-model.pt')

    end_time = time.monotonic()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
#     writer.add_scalar('Epoch/Train/loss', train_loss, epoch)
#     writer.add_scalar('Epoch/Train/accuracy', train_acc, epoch)
#     writer.add_scalar('Epoch/Valid/loss', valid_loss, epoch)
#     writer.add_scalar('Epoch/Valid/accuracy', valid_acc, epoch)
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')<jupyter_output>Epoch: 01 | Epoch Time: 0m 19s
	Train Loss: 3.087 | Train Acc: 9.99%
	 Val. Loss: 2.336 |  Val. Acc: 9.29%
Epoch: 02 | Epoch Time: 0m 19s
	Train Loss: 2.340 | Train Acc: 9.87%
	 Val. Loss: 2.327 |  Val. Acc: 9.64%
Epoch: 03 | Epoch Time: 0m 19s
	Train Loss: 2.330 | Train Acc: 9.87%
	 Val. Loss: 2.332 |  Val. Acc: 10.91%
Epoch: 04 | Epoch Time: 0m 18s
	Train Loss: 2.098 | Train Acc: 19.63%
	 Val. Loss: 1.814 |  Val. Acc: 30.61%
Epoch: 05 | Epoch Time: 0m 19s
	Train Loss: 1.676 | Train Acc: 35.37%
	 Val. Loss: 1.527 |  Val. Acc: 41.88%
Epoch: 06 | Epoch Time: 0m 19s
	Train Loss: 1.448 | Train Acc: 45.71%
	 Val. Loss: 1.342 |  Val. Acc: 51.04%
Epoch: 07 | Epoch Time: 0m 18s
	Train Loss: 1.277 | Train Acc: 53.31%
	 Val. Loss: 1.227 |  Val. Acc: 55.98%
Epoch: 08 | Epoch Time: 0m 18s
	Train Loss: 1.175 | Train Acc: 58.03%
	 Val. Loss: 1.144 |  Val. Acc: 59.82%
Epoch: 09 | Epoch Time: 0m 20s
	Train Loss: 1.089 | Train Acc: 61.07%
	 Val. Loss: 1.012 |  Val. Acc: 63.49%
Epoch: 10 | Epoch Time: [...]<jupyter_text>We then load the parameters of our model that achieved the best validation loss and evaluate this model on the test set to achieve a ~75% accuracy.<jupyter_code>model.load_state_dict(torch.load('tut3-model.pt'))

test_loss, test_acc = evaluate(model, test_iterator, criterion, device)

print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')<jupyter_output>Test Loss: 0.700 | Test Acc: 76.51%
{'fraction_non_alphanumeric': 0.07320466626235239, 'fraction_numerical': 0.020441653883200742, 'mean_word_length': 3.6774570332054064, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 90, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '536198', 'license': 'MIT', 'n_tokens_mistral': 8323, 'n_tokens_neox': 7659, 'n_words': 3421, 'path': 'alexnet-Copy1.ipynb', 'repo_name': 'NONOThingC/NNDL2021-homework', 'size': '28031'}