<jupyter_start><jupyter_text>**Deep learning for image analysis with Python**  <NAME>, Systems Analyst I, Imaging Solutions, Research IT <EMAIL>    (slack) @fernando.cervantes Use ssh, or create a tunnel using MobaXTerm, or Putty to connect to the GCP**ssh -nNfg -L8888:computenodename:8080 student-@... **_To be used only during the workshop only. To login into JAX HPC use ssh as usual_ Run the singularity container using the following command:**singularity run --nv --env CUDA_VISIBLE_DEVICES=0 --bind /fastscratch/data/:/mnt/data/:ro,/fastscratch/models/:/mnt/models/:ro /fastscratch/pytorch_jupyter.sif -m jupyterlab --no-browser --ip=$(hostname -i)**- **--nv** tells Singularity to use the NVIDIA drivers and allows us to use the GPUs inside the container- **--env CUDA_VISIBLE_DEVICES=0** sets an environment variable that specifies what GPU device is going to be used by PyTorch- **--bind /fastscratch/data/:/mnt/data/:ro** bind the location of the datasets to be visible inside the container (under the path _/mnt/data/_) Copy the URL and paste into the search bar of your browser.If jupyter asks for a password setup, use the token from the URL that you copied and use as password: **student-**The token looks something like: http://some-ip:8888/lab?token= **A-long-alphanumeric-string**  5 Implement the training loop  5.1 _Data preparation_ Using the *torchvision* library, pull the CIFAR 100 dataset.We'll need to transform images from this dataset from PIL Images to PyTorch tensors.<jupyter_code>from torchvision.datasets import CIFAR100
from torchvision.transforms import ToTensor

cifar_data = CIFAR100(root='/home/cervaf/data', #'/mnt/data',
                             download=False,
                             train=True,
                             transform=ToTensor()
                            )
from torch.utils.data import DataLoader

cifar_loader = DataLoader(cifar_data,
                        batch_size=128,
                        shuffle=True,
                        pin_memory=True
                        )
print('Dataset size:', len(cifar_data))
print('DataLoader size:', len(cifar_loader))<jupyter_output>Dataset size: 50000
DataLoader size: 391<jupyter_text>Plot some of the CIFAR labeled examples. Check [here](https://huggingface.co/datasets/cifar100) the labels<jupyter_code>import matplotlib.pyplot as plt

x, t = cifar_data[10000]
print(x.size())
plt.imshow(x.permute(1, 2, 0))
plt.title(t)
plt.colorbar()
plt.show()<jupyter_output>torch.Size([3, 32, 32])<jupyter_text>*** 5.2 _Network preparation_ Implement a Residual Network ([ResNet](https://arxiv.org/pdf/1512.03385.pdf))The torchvision module from pytorch provides models that have been pre-trained with the ImageNet dataset (1.2 million images natural images for training, with 1000 different classes)This is usefull for *transfer the learning* to other applications, or use those as *warmed-up* models, instead of random initialization of the weights.However, the architecture is fixed to work with input images of $224\times224$ pixels with $3$ channels (RBG), and output $1000$ classes.For our example with the CIFAR 100 dataset, that is composed by images of $32\times32$ pixesl in RGB, and only has $100$ classes, thre pre-trained models won't be usefull.<jupyter_code>import torchvision.models as models
print(models.resnet18())<jupyter_output>ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
  [...]<jupyter_text>Lets use our LeNet network that we implemented as a custom pytorch Module<jupyter_code>import torch
import torch.nn as nn


class LeNet(nn.Module):
    def __init__(self, in_channels=1, num_classes=10):
        """
        Always call the initialization function from the nn.Module parent class.
        This way all parameters from the operations defined as members of *this* class are tracked for their optimization.
        """
        super(LeNet, self).__init__()
        
        self.conv_1 = nn.Conv2d(in_channels=in_channels, out_channels=6, kernel_size=5)
        self.sub_1 = nn.MaxPool2d(kernel_size=2, stride=2)
        
        self.conv_2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5)
        self.sub_2 = nn.MaxPool2d(kernel_size=2, stride=2)
        
        self.fc_1 = nn.Linear(in_features=5*5*16, out_features=120)
        self.fc_2 = nn.Linear(in_features=120, out_features=84)
        self.fc_3 = nn.Linear(in_features=84, out_features=num_classes)
        
        self.act_fn = nn.ReLU()

    def forward(self, x):
        # Apply convolution layers to extract feature maps with image context
        fx = self.act_fn(self.conv_1(x))
        fx = self.sub_1(fx)
        
        fx = self.act_fn(self.conv_2(fx))
        fx = self.sub_2(fx)
        
        # Flatten the feature maps to perform linear operations
        fx = fx.view(-1, 16*5*5)
        
        fx = self.act_fn(self.fc_1(fx))
        fx = self.act_fn(self.fc_2(fx))
        y = self.fc_3(fx)
        
        return y<jupyter_output><empty_output><jupyter_text>Alternatively, we can define the layers of our network using the Sequential function to reduce the code in the forward pass<jupyter_code>class LeNet(nn.Module):
    def __init__(self, in_channels=1, num_classes=10):
        """
        Always call the initialization function from the nn.Module parent class.
        This way all parameters from the operations defined as members of *this* class are tracked for their optimization.
        """
        super(LeNet, self).__init__()
        
        self.layer_1 = nn.Sequential(
            nn.Conv2d(in_channels=in_channels, out_channels=6, kernel_size=5, stride=1, padding=0, bias=False),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
        )
        
        self.layer_2 = nn.Sequential(
            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5, stride=1, padding=0, bias=False),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
        )
        
        self.layer_fc = nn.Sequential(
            nn.Linear(in_features=5*5*16, out_features=120, bias=True),
            nn.ReLU(),
            nn.Linear(in_features=120, out_features=84, bias=True),
            nn.ReLU(),        
            nn.Linear(in_features=84, out_features=num_classes, bias=True)
        )

    def forward(self, x):    
        fx = self.layer_1(x) 
        fx = self.layer_2(x)

        fx = fx.flatten()

        logits = self.layer_fc(x)
        
        return logits<jupyter_output><empty_output><jupyter_text>Or even have all layers as a single operation<jupyter_code>class LeNet(nn.Module):
    def __init__(self, in_channels=1, num_classes=10):
        """
        Always call the initialization function from the nn.Module parent class.
        This way all parameters from the operations defined as members of *this* class are tracked for their optimization.
        """
        super(LeNet, self).__init__()
        
        self.layers = nn.Sequential(
            nn.Conv2d(in_channels=in_channels, out_channels=6, kernel_size=5, stride=1, padding=0, bias=False),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0),            
            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5, stride=1, padding=0, bias=False),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0),
            nn.Flatten(),
            nn.Linear(in_features=5*5*16, out_features=120, bias=True),
            nn.ReLU(),
            nn.Linear(in_features=120, out_features=84, bias=True),
            nn.ReLU(),        
            nn.Linear(in_features=84, out_features=num_classes, bias=True)
        )

    def forward(self, x):
        logits = self.layers(x)
        
        return logits<jupyter_output><empty_output><jupyter_text>*** 5.3 _Definition of the optimization problem_ Define the loss function and the optimizer, and test it.<jupyter_code>import torch.optim as optim

net = LeNet(in_channels=3, num_classes=100)

criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(
    params=net.parameters(),
    lr=1e-3
)<jupyter_output><empty_output><jupyter_text>This is a single training step !<jupyter_code>%%time
optimizer.zero_grad()

x, t = cifar_data[0]

y = net(x.view(1, 3, 32, 32))

loss = criterion(y, torch.tensor([t]))

loss.backward()

optimizer.step()
print('Loss %f' % loss)<jupyter_output>Loss 3.525292<jupyter_text>*** Setup for training in GPU<jupyter_code>net = LeNet(in_channels=3, num_classes=100)
criterion = nn.CrossEntropyLoss()

# Move our network to the GPU memory, as well as our criterion
net.cuda()
criterion.cuda()

# Define the optimizer after we have moved the network to the GPU, so the update is perfomred in GPU momory
optimizer = optim.Adam(
    params=net.parameters(),
    lr=1e-3
)
%%time
x, t = cifar_data[1]

optimizer.zero_grad() # Clean the gradient accumulator

# Move the input to GPU memory before feeding the network with it
y = net(x.view(1, 3, 32, 32).cuda())

# Move the target as well to compute the loss on the GPU
loss = criterion(y, torch.tensor([t]).cuda())

loss.backward()

optimizer.step()<jupyter_output>CPU times: user 3.91 ms, sys: 2 µs, total: 3.91 ms
Wall time: 4.49 ms<jupyter_text>*** 5.4 _Implement the training loop_ The DataLoader class in PyTorch is usefull to split the dataset into batches.It also convert the individual targets to tensors that can be used directly by the loss function  Run one *epoch* of training<jupyter_code>%%time
trn_loss = []
for i, (x, t) in enumerate(cifar_loader):
    optimizer.zero_grad()
    
    x = x.cuda()
    t = t.cuda()
    
    y = net(x)
    
    loss = criterion(y, t)
    
    loss.backward()
    
    trn_loss.append(loss.item())
    
    optimizer.step()
    
    if i % 100 == 0:
        print('[%i] Loss %.8f' % (i, loss))
plt.plot(trn_loss)<jupyter_output><empty_output><jupyter_text>Run several *epochs*<jupyter_code>%%time
trn_loss = []
net.train()

for e in range(10):
    avg_loss = 0
    for i, (x, t) in enumerate(cifar_loader):
        optimizer.zero_grad()

        x = x.cuda()
        t = t.cuda()
        
        y = net(x)

        loss = criterion(y, t)

        loss.backward()
        
        avg_loss += loss.item()
        trn_loss.append(loss.item())

        optimizer.step()
    
    avg_loss = avg_loss / len(cifar_loader)
    print('[Epoch %02i] %.8f' % (e, avg_loss))
plt.plot(trn_loss)
%%time
trn_loss = []
trn_acc = []

net.train()

for e in range(10):
    avg_loss = 0
    avg_acc = 0
    
    for i, (x, t) in enumerate(cifar_loader):
        optimizer.zero_grad()

        x = x.cuda()
        t = t.cuda()
        
        y = net(x)

        loss = criterion(y, t)
        
        loss.backward()
        
        curr_acc = torch.sum(y.argmax(dim=1) == t).item()
        
        avg_loss += loss.item()
        avg_acc += curr_acc
        
        trn_loss.append(loss.item())
        trn_acc.append(curr_acc / x.size(0))

        optimizer.step()
    
    avg_loss = avg_loss / len(cifar_loader)
    avg_acc = avg_acc / len(cifar_data)
    print('[Epoch %02i] %.8f, acc %.4f %%' % (e, avg_loss, avg_acc))<jupyter_output>[Epoch 00] 2.57480830, acc 0.3429 %
[Epoch 01] 2.55356254, acc 0.3467 %
[Epoch 02] 2.53715809, acc 0.3512 %
[Epoch 03] 2.52499525, acc 0.3528 %
[Epoch 04] 2.50940348, acc 0.3584 %
[Epoch 05] 2.48814400, acc 0.3626 %
[Epoch 06] 2.47072009, acc 0.3648 %
[Epoch 07] 2.45626509, acc 0.3689 %
[Epoch 08] 2.44863286, acc 0.3690 %
[Epoch 09] 2.43216655, acc 0.3732 %
CPU times: user 4min 52s, sys: 918 ms, total: 4min 53s
Wall time: 2min 32s<jupyter_text>Save the trained model (checkpoint)<jupyter_code>torch.save(net.state_dict(), '/mnt/models/lenet_20220420.pth')<jupyter_output><empty_output><jupyter_text>*** Test the trained modelUse the test partition of the CIFAR 100 dataset<jupyter_code>cifar_tst_dataset = CIFAR100(root='/home/cervaf/data', download=False, train=False, transform=ToTensor())
cifar_tst_loader = DataLoader(cifar_tst_dataset, batch_size=128, shuffle=False, pin_memory=True)
import sklearn.metrics
import numpy as np
# Load the checkpoint
state_dict = torch.load('lenet_700epochs_20220519.pth')
for k in state_dict.keys():
    print(k)
# Create a new network object
test_net = LeNet(in_channels=3, num_classes=100)
# Load the trained weights
test_net.load_state_dict(state_dict)<jupyter_output><empty_output><jupyter_text>Test the accuracy of the trained network<jupyter_code>tst_avg_loss = 0
tst_avg_acc = 0
tst_avg_acc_5 = 0
tst_avg_acc_10 = 0

test_net.cuda()
test_net.eval()

for i, (x, t) in enumerate(cifar_tst_loader):
    with torch.no_grad():
        x = x.cuda()
        t = t.cuda()
        
        y = test_net(x)
        loss = criterion(y, t)
    
    tst_avg_loss += loss.item()

    tst_avg_acc += sklearn.metrics.accuracy_score(t.cpu().numpy(), y.cpu().argmax(dim=1).numpy(), normalize=False)
    tst_avg_acc_5 += sklearn.metrics.top_k_accuracy_score(t.cpu().numpy(), y.cpu().numpy(), k=5, labels=np.arange(100), normalize=False)
    tst_avg_acc_10 += sklearn.metrics.top_k_accuracy_score(t.cpu().numpy(), y.cpu().numpy(), k=10, labels=np.arange(100), normalize=False)

tst_avg_loss = tst_avg_loss / len(cifar_tst_loader)
tst_avg_acc = tst_avg_acc / len(cifar_tst_dataset) * 100
tst_avg_acc_5 = tst_avg_acc_5 / len(cifar_tst_dataset) * 100
tst_avg_acc_10 = tst_avg_acc_10 / len(cifar_tst_dataset) * 100
print('Average test loss %.8f, average accuracy %.2f %%, average accuracy@5 %.2f %%, average accuracy@10 %.2f %%' % (tst_avg_loss, tst_avg_acc, tst_avg_acc_5, tst_avg_acc_10))<jupyter_output>Average test loss 3.76036327, average accuracy 27.34 %, average accuracy@5 55.42 %, average accuracy@10 68.21 %<jupyter_text>Check with a sample image from the training set<jupyter_code>x, t = cifar_data[100]
x = x.cuda()
y = test_net(x.view(1, 3, 32, 32))
plt.imshow(x.cpu().permute(1, 2, 0))
plt.title('Real %i (%.2f %%), Predicted %i (%.2f %%)' % (t, torch.softmax(y, dim=1)[0, t].item() * 100, y.cpu().argmax(dim=1)[0].item(), torch.softmax(y, dim=1).max().item() * 100))
plt.show()<jupyter_output><empty_output><jupyter_text>Check the labels [here](https://huggingface.co/datasets/cifar100) Check one image from the test set<jupyter_code>x, t = cifar_tst_dataset[5550]
x = x.cuda()
y = test_net(x.view(1, 3, 32, 32))
plt.imshow(x.cpu().permute(1, 2, 0))
plt.title('Real %i (%.2f %%), Predicted %i (%.2f %%)' % (t, torch.softmax(y, dim=1)[0, t].item() * 100, y.cpu().argmax(dim=1)[0].item(), torch.softmax(y, dim=1).max().item() * 100))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12749645664218529, 'fraction_numerical': 0.04554825409096766, 'mean_word_length': 3.3887475261521063, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '341485', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5995, 'n_tokens_neox': 5482, 'n_words': 1358, 'path': 'code/5-Implement_the_training_loop.ipynb', 'repo_name': 'TheJacksonLaboratory/2022-05-26-deep-learning-image-analysis-pytorch', 'size': '15553'}