<jupyter_start><jupyter_text>References:* Paper for Part3.: <NAME>, <NAME>, <NAME>, and <NAME>. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385,2015.* https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py* https://stackoverflow.com/questions/7691514/extracting-text-from-xml-using-python* https://docs.python.org/2/library/xml.etree.elementtree.html* https://stackoverflow.com/questions/45113245/how-to-get-mini-batches-in-pytorch-in-a-clean-and-efficient-way* https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial<jupyter_code># downloading the whole dataset and unzipping it, I have used the PASCAL VOC 2012 dataset for both training and testing
! wget https://data.deepai.org/PascalVOC2012.zip
! unzip -qq "PascalVOC2012.zip"
# necessary imports here
import time
import os
import cv2
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
from xml.etree import cElementTree as ET
import gc
from tqdm import tqdm
import torch
import torchvision
import torchvision.transforms as transforms
#VOC directory
voc_dir_images = "VOC2012/JPEGImages/"
voc_dir_annotations = "VOC2012/Annotations/"
images = os.listdir(voc_dir_images)
annotations = os.listdir(voc_dir_annotations)
# sorting images and annotations so they align
images = sorted(images)
annotations = sorted(annotations)
# image0 = cv2.imread(voc_dir_images+images[0])[:,:,::-1]
# ann0 = ET.parse(voc_dir_annotations+annotations[0])
# labels[ann0.find("object/name").text]
# encoding the labels
labels = {"aeroplane":0, "bicycle":1, "bird":2, "boat":3, "bottle":4,
          "bus":5, "car":6, "cat":7, "chair":8, "cow":9, "diningtable":10,
          "dog":11, "horse":12, "motorbike":13, "person":14, "pottedplant":15,
          "sheep":16, "sofa":17, "train":18, "tvmonitor":19}
# reverse encoding of labels
rev_labels = {v: k for k, v in labels.items()}
# defining the resize scale
resize = 32<jupyter_output><empty_output><jupyter_text>Defining Train set split<jupyter_code># splitting the train and test set
len(images)   # lets keep 12000 for training and rest for testing
train_list = []
test_list = []
for i, item in enumerate(images):
  if i<12000:
    train_list.append(cv2.resize(cv2.imread(voc_dir_images+item)[:,:,::-1], (resize,resize)))
  else:
    test_list.append(cv2.resize(cv2.imread(voc_dir_images+item)[:,:,::-1], (resize,resize)))
train_class_list = []
test_class_list = []
for i, item in enumerate(annotations):
  ann = ET.parse(voc_dir_annotations+item)
  # print(ann.find("object/name").text)
  if i<12000:
    train_class_list.append(labels[ann.find("object/name").text])
  else:
    test_class_list.append(labels[ann.find("object/name").text])
# creating the train and test tensor for corresponding images and labels
train_X = torch.tensor(train_list)
train_Y = torch.tensor(train_class_list).reshape(-1,1)
test_X = torch.tensor(test_list)
test_Y = torch.tensor(test_class_list).reshape(-1,1)
print(train_X.shape, train_Y.shape) # checking for the input dimensions for training set
print(test_X.shape, test_Y.shape) # checking for the input dimensions for testing set<jupyter_output>torch.Size([12000, 32, 32, 3]) torch.Size([12000, 1])
torch.Size([5125, 32, 32, 3]) torch.Size([5125, 1])<jupyter_text>Visualizing some of the data<jupyter_code># visualizing an intermediate layer output of first 8 channels
plt.figure(figsize=(15,5))
for i in range(5):
  plt.subplot(1,5,i+1)
  plt.imshow(train_X[i])
  plt.title(rev_labels[train_Y[i].item()])
plt.show()<jupyter_output><empty_output><jupyter_text>PART 1. Defining the fully connected network<jupyter_code>import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
torch.manual_seed(0)
np.random.seed(0)
class FlatNetwork(nn.Module):
    def __init__(self):
        super(FlatNetwork, self).__init__()
        self.fc1 = nn.Linear(resize * resize * 3, 1500)
        self.fc2 = nn.Linear(1500, 840)
        self.fc3 = nn.Linear(840, 300)
        self.fc4 = nn.Linear(300, 100)
        self.fc5 = nn.Linear(100, 20)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x) # not using softmax() because the loss function F.cross_entropy(), implicitly performs the softmax()
        return x
FN = FlatNetwork()
# definig a loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(FN.parameters(), lr=0.001, momentum=0.9)
# divind into minibatches
batch_size = 128
st = time.time()
for epoch in tqdm(range(6)):  # loop over the dataset multiple times

    permutation = torch.randperm(train_X.size()[0])
    running_loss = 0.0
    print()
    for i in range(0, train_X.size()[0], batch_size):
      ind = permutation[i:i+batch_size]
      batch_x = train_X[ind]
      batch_y = train_Y[ind]

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = FN(batch_x.reshape(-1,resize*resize*3).float())
      # outputs = outputs.argmax(dim=1).reshape(-1,1)
      # print(outputs.shape,batch_ y.shape)
      loss = criterion(outputs, batch_y.flatten())
      loss.backward()
      optimizer.step()

      # print statistics
      running_loss += loss.item()
      if i>0 and (i // batch_size)%25 == 0:    # print after every 25th mini-batch
          print(epoch + 1, i + 1, running_loss / 25, loss.item())
          running_loss = 0.0

print('Finished Training')
et = time.time()
print("Total time taken to train Simple Neural Network: ", et-st)
# saving the trained model
torch.save(FN.state_dict(), "FlatNet_model")<jupyter_output><empty_output><jupyter_text>Prediction on Test set, using the Flat network<jupyter_code># comment this cell block if training from scratch again
# also, to run this cell first drop all the trained model present in the shared drive folder Q1_models
FN = FlatNetwork()
FN.load_state_dict(torch.load("FlatNet_model"))
print(FN)   # checking the flat network summary
# https://deeplizard.com/learn/video/6vweQjouLEE
FN_preds = FN(test_X.reshape(-1,resize*resize*3).float())
# calculating the accuracy on the test set
# for different runs of the model, getting accuracy between 59% to 85%
print("Flat NN model accuracy on test: ", (FN_preds.argmax(dim=1).reshape(-1,1) == test_Y).sum().item() / test_Y.shape[0] *100)<jupyter_output>Flat NN model accuracy on test:  61.44390243902439<jupyter_text>------  Part 2. Designing a simple CNN<jupyter_code>class SCNN(nn.Module):
    def __init__(self):
        super(SCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 8, 3)
        self.fc1 = nn.Linear(15*15* 8, 1200)  # 15 is the mutated image dimension
        self.fc2 = nn.Linear(1200, 860)
        self.fc3 = nn.Linear(860, 300)
        self.fc4 = nn.Linear(300, 100)
        self.fc5 = nn.Linear(100, 20)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x) # not using softmax() because the loss function F.cross_entropy(), implicitly performs the softmax()
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_feats = 1
        for sz in size:
            num_feats *= sz
        return num_feats
      
    def intermediary_layer_visualization(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        return x
SimpleCNN = SCNN()
# definig a loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(SimpleCNN.parameters(), lr=0.001, momentum=0.9)
batch_size = 128
# using the same training regime as FlatNetwork
st = time.time()
for epoch in tqdm(range(6)):  # loop over the dataset multiple times

    permutation = torch.randperm(train_X.size()[0])
    running_loss = 0.0
    print()
    for i in range(0, train_X.size()[0], batch_size):
      ind = permutation[i:i+batch_size]
      batch_x = train_X[ind]
      batch_y = train_Y[ind]

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = SimpleCNN(batch_x.permute(0,3,1,2).float())
      # outputs = outputs.argmax(dim=1).reshape(-1,1)
      # print(outputs.shape,batch_ y.shape)
      loss = criterion(outputs, batch_y.flatten())
      loss.backward()
      optimizer.step()

      # print statistics
      running_loss += loss.item()
      if i>0 and (i // batch_size)%25 == 0:    # print after every 25th mini-batch
          print(epoch + 1, i + 1, running_loss / 25, loss.item())
          running_loss = 0.0

print('Finished Training')
et = time.time()
print("Total time taken to train Simple CNN: ", et-st)
# saving the trained model
torch.save(SimpleCNN.state_dict(), "SimpleCNN_model")<jupyter_output><empty_output><jupyter_text>Prediction on Test set, using the Flat network<jupyter_code># comment this cell block if training from scratch again
# also, to run this cell first drop all the trained model present in the shared drive folder Q1_models
SimpleCNN = SCNN()
SimpleCNN.load_state_dict(torch.load("SimpleCNN_model"))
print(SimpleCNN)
SCNN_preds = SimpleCNN(test_X.permute(0,3,1,2).float())
print("Simple CNN model accuracy on test: ", (SCNN_preds.argmax(dim=1).reshape(-1,1) == test_Y).sum().item() / test_Y.shape[0] *100)<jupyter_output>Simple CNN model accuracy on test:  65.60000000000001<jupyter_text>Visualization<jupyter_code># visualizing an intermediate layer output
plt.figure(figsize=(8,25))
for i in range(8):
  plt.subplot(8,1,i+1)
  plt.imshow((SimpleCNN.intermediary_layer_visualization(test_X[0].permute(2,0,1).float().reshape(1,3,32,32)))[0][i].detach().numpy())
plt.show()
plt.imshow(test_X[0])
print("Label: ", rev_labels[test_Y[0].item()])
plt.show()<jupyter_output><empty_output><jupyter_text>------  Part 3. Designing a Deep CNN<jupyter_code># https://missinglink.ai/guides/pytorch/pytorch-resnet-building-training-scaling-residual-networks-pytorch/
# first creating a basic block module
class BasicBlock(nn.Module):
    expansion = 1
    def __init__(self, in_c, out_c, stride=1):
      super(BasicBlock, self).__init__()
      self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, stride=stride, padding=1, bias=False)
      self.bn1 = nn.BatchNorm2d(out_c)
      self.conv2 = nn.Conv2d(out_c, out_c, kernel_size=3, stride=1, padding=1, bias=False)
      self.bn2 = nn.BatchNorm2d(out_c)

      self.skip = nn.Sequential()
      if stride != 1 or in_c != self.expansion*out_c:
          self.skip = nn.Sequential(nn.Conv2d(in_c, self.expansion*out_c, kernel_size=1, stride=stride, bias=False),
                                    nn.BatchNorm2d(self.expansion*out_c))
    
    def forward(self, x):
      out = F.relu(self.bn1(self.conv1(x)))
      out = self.bn2(self.conv2(out))
      out += self.skip(x)
      out = F.relu(out)
      return out
# the resnet module
class ResNet(nn.Module):
  def __init__(self, block, num_blocks, num_classes=20):
      super(ResNet, self).__init__()
      self.in_c = 64
      self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
      self.bn1 = nn.BatchNorm2d(64)
      self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
      self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
      self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
      self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
      self.linear = nn.Linear(512*block.expansion, num_classes)

  def _make_layer(self, block, out_c, num_blocks, stride):
      strides = [stride] + [1]*(num_blocks-1)
      layers = []
      for stride in strides:
          layers.append(block(self.in_c, out_c, stride))
          self.in_c = out_c * block.expansion
      return nn.Sequential(*layers)

  def forward(self, x):
      out = F.relu(self.bn1(self.conv1(x)))
      out = self.layer1(out)
      out = self.layer2(out)
      out = self.layer3(out)
      out = self.layer4(out)
      out = F.avg_pool2d(out, 4)
      out = out.view(out.size(0), -1)
      out = self.linear(out)
      return out

  def intermediary_layer_visualization(self, x):
      out = F.relu(self.bn1(self.conv1(x)))
      out = self.layer1(out)
      out = self.layer2(out)
      return out
resnet18 = ResNet(BasicBlock, [2, 2, 2, 2])
# definig a loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(resnet18.parameters(), lr=0.001, momentum=0.9)
batch_size = 128
# using the same training regime as FlatNetwork
st = time.time()
for epoch in tqdm(range(1)):  # loop over the dataset multiple times
    permutation = torch.randperm(train_X.size()[0])
    running_loss = 0.0
    for i in range(0, train_X.size()[0], batch_size):
      ind = permutation[i:i+batch_size]
      batch_x = train_X[ind]
      batch_y = train_Y[ind]

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = resnet18(batch_x.permute(0,3,1,2).float())
      # outputs = outputs.argmax(dim=1).reshape(-1,1)
      # print(outputs.shape,batch_ y.shape)
      loss = criterion(outputs, batch_y.flatten())
      loss.backward()
      optimizer.step()

      # print statistics
      running_loss += loss.item()
      if i>0 and (i // batch_size)%2 == 0:    # print after every 3rd mini-batch
          print(epoch + 1, i + 1, running_loss / 25, loss.item())
          running_loss = 0.0

print('Finished Training')
et = time.time()
print("Total time taken to train Resnet34: ", et-st)
# saving the trained model
torch.save(resnet18.state_dict(), "resnet18_model")<jupyter_output><empty_output><jupyter_text>Prediction on Test set, using the resnet18 network<jupyter_code># comment this cell block if training from scratch again
# also, to run this cell first drop all the trained model present in the shared drive folder Q1_models
resnet18 = ResNet(BasicBlock, [2, 2, 2, 2])
resnet18.load_state_dict(torch.load("resnet18_model"))
print(resnet18)
# Since the architechture is deeper, we want to test predictions in batches
test_batch_size = 125
acc_sum = 0
for bat,i in enumerate(range(0, test_X.size()[0], test_batch_size)):
      #ind = [i:i+test_batch_size]
      batch_x = test_X[i:i+test_batch_size]
      batch_y = test_Y[i:i+test_batch_size]
      res_preds = resnet18(batch_x.permute(0,3,1,2).float())
      acc_sum += (res_preds.argmax(dim=1).reshape(-1,1) == batch_y).sum().item() / batch_y.shape[0] *100
      print("RESNET CNN model accuracy on test batch", bat+1,": ", (res_preds.argmax(dim=1).reshape(-1,1) == batch_y).sum().item() / batch_y.shape[0] *100)
print("RESNET CNN model overall accuracy on test set is: ", acc_sum/(test_Y.shape[0]/test_batch_size))<jupyter_output>RESNET CNN model accuracy on test batch 1 :  25.6
RESNET CNN model accuracy on test batch 2 :  16.8
RESNET CNN model accuracy on test batch 3 :  18.4
RESNET CNN model accuracy on test batch 4 :  24.0
RESNET CNN model accuracy on test batch 5 :  26.400000000000002
RESNET CNN model accuracy on test batch 6 :  28.000000000000004
RESNET CNN model accuracy on test batch 7 :  17.599999999999998
RESNET CNN model accuracy on test batch 8 :  22.400000000000002
RESNET CNN model accuracy on test batch 9 :  88.0
RESNET CNN model accuracy on test batch 10 :  81.6
RESNET CNN model accuracy on test batch 11 :  82.39999999999999
RESNET CNN model accuracy on test batch 12 :  86.4
RESNET CNN model accuracy on test batch 13 :  80.0
RESNET CNN model accuracy on test batch 14 :  83.2
RESNET CNN model accuracy on test batch 15 :  82.39999999999999
RESNET CNN model accuracy on test batch 16 :  84.0
RESNET CNN model accuracy on test batch 17 :  87.2
RESNET CNN model accuracy on test batch 18 :  84.0
RESNET CN[...]<jupyter_text>Visulization<jupyter_code># visualizing an intermediate layer output of first 8 channels
plt.figure(figsize=(8,25))
for i in range(8):
  plt.subplot(8,1,i+1)
  plt.imshow((resnet18.intermediary_layer_visualization(test_X[0].permute(2,0,1).float().reshape(1,3,32,32)))[0][i].detach().numpy())
plt.show()
plt.imshow(test_X[0])
print("Label: ", rev_labels[test_Y[0].item()])
plt.show()<jupyter_output><empty_output><jupyter_text>Analysis and comparision **Time Comparision*** The Flat Neural Network took **48.05s** to complete training for 6 epochs on 12000 images of size 32x32x3 (i.e. 3 channels RGB)* The Simple CNN with single convolution and pooling layer took **44.73s** to complete the training procedure for 6 epochs* The Deep CNN i.e. Resnet18 architecture took **917.18s** to complete the training process for a single epoch. **Accuracy on test set*** Flat NN model accuracy on test:  **61.44390243902439 %*** Simple CNN model accuracy on test:  **65.60000000000001 %*** RESNET CNN model overall accuracy on test set is:  **71.2 %****Final Thoughts*** Since the overall dataset size is roughly 17000, deep networks like resnet gives average performance as this network has a huge hunger for data, which the current dataset does'nt provide much.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1296037296037296, 'fraction_numerical': 0.048193473193473196, 'mean_word_length': 3.987213019471084, 'pattern_counts': {'":': 21, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '415299', 'license': 'MIT', 'n_tokens_mistral': 6462, 'n_tokens_neox': 5945, 'n_words': 1507, 'path': 'main.ipynb', 'repo_name': 'jaideepbankoti/voc2012-classification', 'size': '17161'}