<jupyter_start><jupyter_text>Pytorch Tutorial 3. Training and Evaluating a Model- General training setup- Managing and loading datasets- Experience the conventions of training and evaluating Setup torch and variables<jupyter_code>import sys

import torch, torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data
import torchvision.transforms as transforms
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>General training setupWe define the ```device``` variable, which specifies where we will run our training process.<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)<jupyter_output><empty_output><jupyter_text>CPU is the default device. If you are to run our model on cpu, the following content isn't relevent.However if we want to run our model on cuda, two thing must be satisfied:1. The input data (tensors) should be on cuda.  2. The model parameters should be on cuda.This can be achieved in two different ways:1. Creating tensors with argument device:    ```python    x = torch.randn((4,5), device=device)    ```2. Moving an object to ```device```:    ```python    model = SimpleNet().to(device)    ```    ```python    input_data = input_data.to(device)    ```  Managing Datasets```torchvision.datasets``` provides famous datasets, which are listed in the [documentation](https://pytorch.org/docs/stable/torchvision/datasets.html).Let us toy with the simplest dataset, MNIST. You can apply transforms to the data here with the ```transform``` argument.<jupyter_code>transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
MNIST_train = torchvision.datasets.MNIST(root='MNIST', train=True, transform=transform, download=True)
MNIST_test = torchvision.datasets.MNIST(root='MNIST', train=False, transform=transform, download=True)
MNIST_train
MNIST_test<jupyter_output><empty_output><jupyter_text>DataLoaders load batches from the dataset. Here we just used ```shuffle=True```, but it is also possible to have ```shuffle=False``` and use an instance from ```torch.utils.data.RandomSampler``` and pass it to the parameter ```sampler```.<jupyter_code>train_data_loader = torch.utils.data.DataLoader(MNIST_train, batch_size=64, shuffle=True)
test_data_loader = torch.utils.data.DataLoader(MNIST_test, batch_size=64, shuffle=False)<jupyter_output><empty_output><jupyter_text>Creating custom datasets is also possible by definig a class that inherits ```torch.utils.data.Dataset```. You need to write functions ```__init__```, ```__getitem__```, and ```__len__```.<jupyter_code>class CustomDataset(torch.utils.data.Dataset):
    
    def __init__(self, filename, division):
        assert division in ['train', 'test']
        with np.load(filename) as f:
            self.x, self.y = f[f'x_{dtype}'], f[f'y_{dtype}']
        
    def __getitem__(self, ind):
        return self.x[ind], self.y[ind]

    def __len__(self):
        assert len(self.x) == len(self.y)
        return len(self.x)<jupyter_output><empty_output><jupyter_text>Train and EvaluateHere we toy with the simplist network, LeNet, and use the MNIST dataset loaded above.<jupyter_code>class LeNet(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)
        
        for m in self.modules():
            if type(m) in [nn.Linear, nn.Conv2d]:
                nn.init.kaiming_normal_(m.weight)
                m.bias.data.fill_(0.)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)<jupyter_output><empty_output><jupyter_text>The following is the convention for the training logic.<jupyter_code>epochs = 1
print_every = 1
model = LeNet().to(device)
optimizer = optim.Adam(model.parameters(), lr=0.001)<jupyter_output><empty_output><jupyter_text>Set model to training mode. The opposite is ```.eval()```. This part is necessary if the model includes layers that behave differently on evaluation phase, like batch normalization or dropout.<jupyter_code>model = model.train()
for epoch in range(epochs):
    for batch_ind, (input_data, target_data) in enumerate(train_data_loader):
        # Move data to device
        input_data, target_data = input_data.to(device), target_data.to(device)

        # Forward propagation
        output = model(input_data)

        # Calculate loss function
        loss = F.cross_entropy(output, target_data)

        # Backward propagation
        optimizer.zero_grad()    # This is equivalent to model.zero_grad()
        loss.backward()

        # Update parameters
        optimizer.step()

        # Print progress
        if batch_ind % print_every == 0:
            train_log = f'Epoch {epoch+1:2d}/{epochs:2d}\tLoss: {loss.cpu().item():.6f}\tTrain: [{batch_ind+1}/{len(train_data_loader)} ({100.*batch_ind/len(train_data_loader):.0f}%)]            '
            print(train_log, end='\r')
            sys.stdout.flush()
    print()<jupyter_output><empty_output><jupyter_text>The following is the convention for the testing logic.<jupyter_code>model = model.eval()
correct = 0
with torch.no_grad():
    with tqdm(total=len(test_data_loader)) as pbar:
        for batch_ind, (input_data, target_data) in enumerate(test_data_loader):
            # Move data to device
            input_data, target_data = input_data.to(device), target_data.to(device)
            
            # Inference
            output = model(input_data)
            pred = output.argmax(dim=1)
            
            # Count number of correct predictions
            correct += pred.eq(target_data.view_as(pred)).sum()
            
            # Progress bar update
            pbar.update(1)

print(f'Test accuracy: {100. * int(correct) / len(MNIST_test)}%')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11975950601234969, 'fraction_numerical': 0.013649658758531037, 'mean_word_length': 3.634789156626506, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '612381', 'license': 'MIT', 'n_tokens_mistral': 2014, 'n_tokens_neox': 1939, 'n_words': 509, 'path': 'Pytorch Tutorial/3. Training and Evaluating a Model.ipynb', 'repo_name': 'jaywonchung/Pytorch-Tutorial', 'size': '6154'}