<jupyter_start><jupyter_text>MNIST MLP With Training, Validation and Test SetsIn this notebook we build a simple MLP for the MNIST dataset. The interesting part is the split of the training `DataLoader` into a training and a validation part, by means of an instance of the `SubsetRandomSampler` class. In fact, it's not possible to directly index or subset a `DataLoader`. This is why the `sampler=` argument in a `DataLoader` is so important.<jupyter_code>import numpy as np
import torch
from torchvision import datasets
import torchvision.transforms as transforms
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn as nn
import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>This notebook was written to run on my desktop computer. If you want to run it elsewhere you have to change the location where the MNIST dataset will be downloaded.<jupyter_code># location of the MNIST images
ROOT = '~/.pytorch'
# number of subprocesses to use for data loading
NUM_WORKERS = 0
# how many samples per batch to load
BATCH_SIZE = 32
# percentage of training set to use as validation
VALID_SIZE = 0.2
# Size of the hidden layers
HIDDEN_SIZE = 512
# Size of the output layer
OUTPUT_SIZE = 10

# Set device
CUDA_DEVICE = 'cuda'
if torch.cuda.is_available():
    device = torch.device(CUDA_DEVICE)
else:
    torch.device('cpu')<jupyter_output><empty_output><jupyter_text>**QUESTION**: when we create the training and the validation `DataLoaders`, it appears to me that we are forced to use the same transformations for both. Therefore, how can I apply data augmentation to the training data loader but not to the validation one? Do we have to split the data in different folders? Probably.We start as usual from the training and test `DataLoaders` that come with MNIST.<jupyter_code># convert data to torch.FloatTensor
transform = transforms.ToTensor()

# choose the training and test datasets
train_data = datasets.MNIST(root=ROOT, train=True,
                            download=True, transform=transform)
test_data = datasets.MNIST(root=ROOT, train=False,
                           download=True, transform=transform)<jupyter_output><empty_output><jupyter_text>The code below creates the samplers for the training and the validation sets.<jupyter_code># obtain training indices that will be used for validation
num_train = len(train_data)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(VALID_SIZE * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)<jupyter_output><empty_output><jupyter_text>With these samplers, we can build the three data loaders.<jupyter_code># prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=BATCH_SIZE,
                                           sampler=train_sampler,
                                           num_workers=NUM_WORKERS)
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=BATCH_SIZE,
                                           sampler=valid_sampler,
                                           num_workers=NUM_WORKERS)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=BATCH_SIZE,
                                          num_workers=NUM_WORKERS)

images, labels = next(iter(train_loader))<jupyter_output><empty_output><jupyter_text>We can now define the network architecture.<jupyter_code>class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, HIDDEN_SIZE)
        self.fc2 = nn.Linear(HIDDEN_SIZE, HIDDEN_SIZE)
        self.fc3 = nn.Linear(HIDDEN_SIZE, OUTPUT_SIZE)

    def forward(self, x):
        x = x.view(-1, 784)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x<jupyter_output><empty_output><jupyter_text>For convenience, we can define a function that runs the typical operations of a training or validation loop and, based on this, create a `fit` function.<jupyter_code>def train_valid_loop(dataloader, model, criterion, optimizer, every=10, what='train'):
    running_loss = 0
    running_acc = 0
    batch_num = 0

    for images, labels in dataloader:
        batch_num += 1
        images, labels = images.to(device), labels.to(device)
        logits = model(images)
        loss = criterion(logits, labels)
        if what == 'train':
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        running_loss += loss
        _, preds = torch.max(logits, dim=1)
        running_acc += (preds == labels).type(torch.IntTensor).sum()

        if batch_num % every == 0:
            print(('Batch N.: {:3}, ' + what.title() + 'Loss: {:.3f}, ' +
                   what.title() + 'Acc.: {:.3f}').format(
                       batch_num,
                       running_loss.item()/(batch_num*dataloader.batch_size),
                       running_acc.item()/(batch_num*dataloader.batch_size)
            ))<jupyter_output><empty_output><jupyter_text>The fit function will run the `train_valid_loop` function with or without the computation of the gradients, based on the type of data loader.<jupyter_code>def fit(epochs, model, criterion, optimizer, every):
    for epoch in range(epochs):
        print(f'--- Epoch: {epoch} ---')
        # Train loop
        model.train()
        print('\n--- Train loop ----\n')
        train_valid_loop(train_loader, model, criterion,
                         optimizer, every, 'train')

        # Validation loop
        model.eval()
        print('\n--- Validation loop ---\n')
        with torch.no_grad():
            train_valid_loop(valid_loader, model, criterion,
                             optimizer, every, what='valid')<jupyter_output><empty_output><jupyter_text>We can now instantiate the model, define the loss function and the optimizer, and run the training/validation loop.<jupyter_code>model = Net()
model = model.to(device)
criterion = nn.CrossEntropyLoss(reduction='sum')
optimizer = torch.optim.RMSprop(model.parameters())
fit(3, model, criterion, optimizer, every=100)<jupyter_output>--- Epoch: 0 ---

--- Train loop ----

Batch N.: 100, TrainLoss: 12.142, TrainAcc.: 0.627
Batch N.: 200, TrainLoss: 6.354, TrainAcc.: 0.735
Batch N.: 300, TrainLoss: 4.396, TrainAcc.: 0.776
Batch N.: 400, TrainLoss: 3.398, TrainAcc.: 0.805
Batch N.: 500, TrainLoss: 2.794, TrainAcc.: 0.822
Batch N.: 600, TrainLoss: 2.398, TrainAcc.: 0.834
Batch N.: 700, TrainLoss: 2.107, TrainAcc.: 0.844
Batch N.: 800, TrainLoss: 1.888, TrainAcc.: 0.851
Batch N.: 900, TrainLoss: 1.711, TrainAcc.: 0.859
Batch N.: 1000, TrainLoss: 1.575, TrainAcc.: 0.864
Batch N.: 1100, TrainLoss: 1.461, TrainAcc.: 0.869
Batch N.: 1200, TrainLoss: 1.370, TrainAcc.: 0.872
Batch N.: 1300, TrainLoss: 1.289, TrainAcc.: 0.875
Batch N.: 1400, TrainLoss: 1.218, TrainAcc.: 0.879
Batch N.: 1500, TrainLoss: 1.160, TrainAcc.: 0.881

--- Validation loop ---

Batch N.: 100, ValidLoss: 0.252, ValidAcc.: 0.936
Batch N.: 200, ValidLoss: 0.251, ValidAcc.: 0.935
Batch N.: 300, ValidLoss: 0.266, ValidAcc.: 0.933
--- Epoch: 1 ---

--- Train [...]
{'fraction_non_alphanumeric': 0.10667034178610804, 'fraction_numerical': 0.0337651598676957, 'mean_word_length': 3.2020845396641575, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '655796', 'license': 'MIT', 'n_tokens_mistral': 2436, 'n_tokens_neox': 2181, 'n_words': 688, 'path': 'convolutional-neural-networks/mnist-mlp/my_mnist_mlp.ipynb', 'repo_name': 'gdario/deep-learning-v2-pytorch', 'size': '7256'}