<jupyter_start><jupyter_text>DL4CV Mid-Term Project (Spring, 2022) <NAME> RequirementsThis project has been developed to run on Google Colab as an Jupyter notebook. No dependencies beyond what Colab supplies are required. It is recommended that the Colab runtime be set to 'GPU' and 'High Ram' but it will execute fine if these are unset. If a library is missing it may be installed into a Colab session with the following:  ```bash!pip install ```If a runtime error occurs it is likely related to connection issues between Colab and Google Drive, or to input data not being available in the required form. The instructors have provided no guidance regarding how the GTSRB dataset should be made available for this assignment to run in a different environment; this has been left to the student to determine.  Data LoadingFirst, if running Colab with Drive, create the directory 'DL4CV-2022' in your home directory and clone [this Github repository](<EMAIL>:geraldmc/DL4CV-2022.git) into DL4CV-2022. Your gdrive_path to the project should then look like the following:      gdrive_path = '/content/drive//MyDrive/DL4CV-2022/project-I/'The notebook will attempt to load three zipped data files.     zip_path_train = gdrive_path + 'data/Final_Training.zip'    zip_path_val = gdrive_path + 'data/Final_Validation.zip'    zip_path_test = gdrive_path + 'data/Final_Test.zip'Two of these (Final_Training.zip, Final_Test.zip) have been created by downloading from the GTSRB download site. The files used for the assignment are:[GTSRB_Final_Training_Images.zip](https://sid.erda.dk/public/archives/daaeac0d7ce1152aea9b61d9f1e19370/GTSRB_Final_Training_Images.zip)[GTSRB_Final_Test_Images.zip](https://sid.erda.dk/public/archives/daaeac0d7ce1152aea9b61d9f1e19370/GTSRB_Final_Test_Images.zip)The directory structure for the unzipped files is the following:```- Final_Training  - Annotations    - GT_Training.csv (this file is required)    - GT-00000.csv    - GT-00001.csv  - Images    - 00000      - 00000_00000.ppm      - 00000_00001.ppm    - 00001      - 00000_00000.ppm      - 00000_00001.ppm    - ...- Final_Test  - Images    - 01089.ppm    - 01090.ppm    - ...```The file 'Final_Validation.zip' was created by using images 00000\*, 00001\* and 00002\* from each class in the training set. There are 23,220 files in the augmented validation set and 3,900 in the unaugmented set.    Running Execute the Jupyter notebook `DL4CV-2022/project-I/main.ipynb`. It will run to completion using the `Run All` command in Colab. IssuesIf there are any questions please do not hesitate to text me at <EMAIL>.<jupyter_code>import os
import argparse
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import datasets, transforms
from torchsummary import summary
# Automatic reloading and inline plotting
%reload_ext autoreload
%autoreload 2
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
import sys

COLAB = 'google.colab' in str(get_ipython())
HAS_GPU = torch.cuda.is_available()
is_local = not COLAB

if COLAB: # running on Colab
    
    from google.colab import drive
    drive.flush_and_unmount()
    
    drive.mount('/content/drive', force_remount=True)
    gdrive_path = '/content/drive//MyDrive/DL4CV-2022/project-I/'
    sys.path.append(gdrive_path)

    zip_path_train = gdrive_path + 'data/Final_Training.zip'
    zip_path_val = gdrive_path + 'data/Final_Validation.zip'
    zip_path_test = gdrive_path + 'data/Final_Test.zip'
    !unzip -q "{zip_path_train}"
    !unzip -q "{zip_path_val}"
    !unzip -q "{zip_path_test}"
    
    if HAS_GPU:
        print("Using GPU")
else:
    HAS_GPU = False
    print("Using CPU")

# If we run cuda then accomodate these datatypes.
FloatTensor = torch.cuda.FloatTensor if HAS_GPU else torch.FloatTensor
LongTensor = torch.cuda.LongTensor if HAS_GPU else torch.LongTensor
ByteTensor = torch.cuda.ByteTensor if HAS_GPU else torch.ByteTensor
Tensor = FloatTensor
import loader.args as load_args
args = load_args.get_args(is_local)

TRAIN_DATA_PATH = args.data_dir_train
VAL_DATA_PATH = args.data_dir_val
TEST_DATA_PATH = args.data_dir_test
BATCH_SIZE = args.batch_size
EPOCHS = args.num_epochs
BATCH_SIZE = args.batch_size
LEARNING_RATE = args.learning_rate
from loader import transforms as tfs
from torch.utils.data import Dataset
from loader.gtsrb_data import GTSRB_Test
from utils.tensor2image import imshow

train_data = datasets.ImageFolder(
    root=TRAIN_DATA_PATH, 
    transform=tfs.base_transform)
train_loader = data.DataLoader(train_data, 
    batch_size=BATCH_SIZE, shuffle=True,  
    num_workers=2)

val_data = datasets.ImageFolder(
    root=VAL_DATA_PATH, 
    transform=tfs.base_transform)
val_loader  = data.DataLoader(val_data, 
    batch_size=BATCH_SIZE, shuffle=False, 
    num_workers=2)
if COLAB:
    GTSRB_test_dataset = GTSRB_Test('/content/')
    
else:
    GTSRB_test_dataset = GTSRB_Test('data/')

assert len(GTSRB_test_dataset) == 12630

test_loader = data.DataLoader(GTSRB_test_dataset, 
    batch_size=BATCH_SIZE, shuffle=False, 
    num_workers=2)
# Without augmentation
assert len(train_loader)*50 == 39250
assert len(val_loader)*50 == 3900
assert len(test_loader)*50 == 12650
print()
print('Unaugmented dataset: %d training samples & %d validation samples\n' % (
len(train_loader)*args.batch_size, len(val_loader)*args.batch_size))
from model.models import Net1

# Reset everything
model = None
optimizer = None
scheduler = None

model = Net1()
if HAS_GPU:
    model.cuda()

# Optimizer is updated to *not* include non-gradient weights.    
optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()),
                                                      lr=args.learning_rate)
# Reduce learning rate when a metric has stopped improving. 
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min',
                                                 patience=5,
                                                 factor=0.5,
                                                 verbose=True)
summary(model,(3,32,32))
from utils.meter import AvgMeter, calc_accuracy

train_losses = []
train_accuracies = []

'''
    .                       o8o
  .o8                       `"'
.o888oo oooo d8b  .oooo.   oooo  ooo. .oo.
  888   `888""8P `P  )88b  `888  `888P"Y88b
  888    888      .oP"888   888   888   888
  888 .  888     d8(  888   888   888   888
  "888" d888b    `Y888""8o o888o o888o o888o
'''

def train(epoch):
    print()
    print('Epoch {} -------------------------------------------------------->'.format(epoch) )
    print()
    losses = AvgMeter()
    accuracies = AvgMeter()
    model.train()
    correct = 0
    training_loss = 0
    
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data), Variable(target)
        if HAS_GPU:
            data = data.cuda()
            target = target.cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        
        losses.update(loss.item(), data.size(0))
        accuracy = calc_accuracy(output, target)[0]
        train_accuracies.append(int(accuracy))
        accuracies.update(accuracy, data.size(0))
        
        loss.backward()
        optimizer.step()
        max_index = output.max(dim = 1)[1] # _, predicted
        correct += (max_index == target).sum()
        training_loss += loss
        
        if batch_idx % args.log_interval == 0:
            print('Training set: Average loss: {:.4f}, Accuracy: {}/{} ({:.3f}%)\n'.format(
            losses.avg, int(accuracies.sum/100), accuracies.count, accuracies.avg))
    
    return train_accuracies
val_losses = []
val_accuracies = []

'''
                      oooo
                      `888
oooo    ooo  .oooo.    888
 `88.  .8'  `P  )88b   888
  `88..8'    .oP"888   888
   `888'    d8(  888   888
    `8'     `Y888""8o o888o
'''

def validate(epoch):
    losses = AvgMeter()
    accuracies = AvgMeter()
    error_cases = []
    
    model.eval() # changes the forward() behaviour of the module for val/test.
    
    with torch.no_grad(): # disable tracking of gradients in autograd.
        for data, target in val_loader:
            if HAS_GPU:
                data = data.cuda()
                target = target.cuda()
            output = model(data)
            loss = F.nll_loss(output, target)

            losses.update(loss.item(), data.size(0))
            accuracy = calc_accuracy(output, target)[0]
            val_accuracies.append(accuracy)
            accuracies.update(accuracy, data.size(0))

            _, pred = output.topk(1, 1, True, True)
            pred = pred.t()

        print('\nValidation set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n'.format(
            losses.avg, int(accuracies.sum/100), accuracies.count, accuracies.avg))

    return val_accuracies
import numpy as np
import pandas as pd
import sklearn.metrics as sm

'''
    .                          .
  .o8                        .o8
888oo  .ooooo.   .oooo.o .o888oo
  888   d88' `88b d88(  "8   888
  888   888ooo888 `"Y88b.    888
  888 . 888    .o o.  )88b   888 .
  "888" `Y8bod8P' 8""888P'   "888"
'''

def test(test_loader):
    model.eval()
    correct = 0
    targets, preds = [], []
    
    with torch.no_grad():
        for data, target in test_loader:
            if HAS_GPU:
                data = data.cuda()
                target = target.cuda()
            output = model(data)
            pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

            targets += list(target.cpu().numpy())
            preds += list(pred.cpu().numpy())

    test_acc = 100. * correct / len(test_loader.dataset)
    confusion_mtx = sm.confusion_matrix(targets, preds)
    return test_acc, confusion_mtx
training_accuracy = []
validation_accuracy = []
test_accuracy = []
confusion = [] 

for epoch in range(1, args.num_epochs + 1):
    training_accuracy.append(train(epoch))
    validation_accuracy.append(validate(epoch))
    test_acc, confusion_mtx = test(test_loader)
    test_accuracy.append(test_acc)
    confusion.append(confusion_mtx)
    print()
    print('\r --> Testing set: Accuracy (Best) = %2.2f%%' % max(test_accuracy), 
          end='', flush=True)
    print()

#model_file = 'model_' + str(epoch) + '.pth'
#torch.save(model.state_dict(), model_file)
import matplotlib.pyplot as plt

# Plot accuracy
plt.plot(training_accuracy[0],'-o', markersize=.5)
plt.plot(validation_accuracy[0],'-o', markersize=.5)
plt.xlabel('Sample # per Epoch')
plt.ylabel('Accuracy')
plt.legend(['Train','Valid'])
plt.title('Train vs Valid Accuracy per Epoch (Best)')
None
# Reset
training_accuracy = None
validation_accuracy = None
import matplotlib.pyplot as plt
print('Dataset: %d training samples & %d testing samples\n' % (
    len(train_loader.dataset), len(test_loader.dataset)))
 
print('Distribution of classes in training dataset:')
fig, ax = plt.subplots()
labels = [label for _, label in train_loader.dataset.imgs]
class_labels, counts = np.unique(labels, return_counts=True)
ax.bar(class_labels, counts)
ax.set_xticks(class_labels)
plt.show()
idx2class = {v: k for k, v in train_data.class_to_idx.items()}

def get_class_distribution(dataset_obj):
    count_dict = {k:0 for k,v in dataset_obj.class_to_idx.items()}
    
    for element in dataset_obj:
        y_lbl = element[1]
        y_lbl = idx2class[y_lbl]
        count_dict[y_lbl] += 1
            
    return count_dict

class_distro = get_class_distribution(train_data)
print("Distribution of classes ranges from {} to {}.".format
      (min(class_distro.values()), max(class_distro.values())))

from utils.visualize import visual

visual(test_accuracy, confusion, class_labels)
from loader.imbalanced import ImbalancedDatasetSampler
from loader import transforms as tfs
from torch.utils.data import Dataset
from loader.gtsrb_data import GTSRB_Test

train_data = None
val_data = None
train_loader = None
val_loader = None
GTSRB_test_dataset = None
test_loader = None

# Reload everything
train_data = datasets.ImageFolder(
    root=TRAIN_DATA_PATH, 
    transform=tfs.base_transform)
train_loader = data.DataLoader(train_data, 
    batch_size=BATCH_SIZE, 
    sampler=ImbalancedDatasetSampler(train_data), 
    num_workers=2)

val_data = datasets.ImageFolder(
    root=VAL_DATA_PATH, 
    transform=tfs.base_transform)
val_loader = data.DataLoader(val_data, 
    batch_size=BATCH_SIZE, 
    sampler=ImbalancedDatasetSampler(val_data), # <----- data sampler
    num_workers=2)

if COLAB:
    GTSRB_test_dataset = GTSRB_Test('/content/')
    
else:
    GTSRB_test_dataset = GTSRB_Test('data/')

assert len(GTSRB_test_dataset) == 12630

test_loader = data.DataLoader(GTSRB_test_dataset, 
    batch_size=BATCH_SIZE, shuffle=False, 
    num_workers=2)

from model.models import Net1

# Reset
model = None
optimizer = None
scheduler = None

model = Net1()
if HAS_GPU:
    model.cuda()

# Optimizer is updated to *not* include non-gradient weights.    
optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()),
                                                      lr=args.learning_rate)
# Reduce learning rate when a metric has stopped improving. 
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min',
                                                 patience=5,
                                                 factor=0.5,
                                                 verbose=True)
# Reset
training_accuracy = []
validation_accuracy = []
test_accuracy = []
confusion = [] 
test_acc = None
confusion_mtx = None
epoch = None

for epoch in range(1, args.num_epochs + 1):
    training_accuracy.append(train(epoch))
    validation_accuracy.append(validate(epoch))
    test_acc, confusion_mtx = test(test_loader)
    test_accuracy.append(test_acc)
    confusion.append(confusion_mtx)
    print()
    print('\r --> Testing set: Accuracy (Best) = %2.2f%%' % max(test_accuracy), 
          end='', flush=True)
    print()
    
#model_file = 'model_' + str(epoch) + '.pth'
#torch.save(model.state_dict(), model_file)
from utils.visualize import visual

visual(test_accuracy, confusion, class_labels)
train_loader = None
val_loader = None

train_loader = torch.utils.data.DataLoader(
 torch.utils.data.ConcatDataset([datasets.ImageFolder(
                                    TRAIN_DATA_PATH, 
                                    transform=tfs.base_transform),
                               datasets.ImageFolder(
                                    TRAIN_DATA_PATH,
                                    transform=tfs.data_translate),
                               datasets.ImageFolder(
                                    TRAIN_DATA_PATH,
                                    transform=tfs.data_grayscale),
                               datasets.ImageFolder(
                                    TRAIN_DATA_PATH,
                                    transform=tfs.data_center),
                               datasets.ImageFolder(
                                    TRAIN_DATA_PATH,
                                    transform=tfs.data_rotate),
                               datasets.ImageFolder(
                                    TRAIN_DATA_PATH,
                                    transform=tfs.data_jitter_hue)]), 
                                    batch_size=args.batch_size, 
                                    shuffle=True, num_workers=2, 
                                    pin_memory=HAS_GPU)

val_loader = torch.utils.data.DataLoader(
 torch.utils.data.ConcatDataset([datasets.ImageFolder(
                                    VAL_DATA_PATH, 
                                    transform=tfs.base_transform),
                               datasets.ImageFolder(
                                    VAL_DATA_PATH,
                                    transform=tfs.data_translate),
                               datasets.ImageFolder(
                                    VAL_DATA_PATH,
                                    transform=tfs.data_grayscale),
                               datasets.ImageFolder(
                                    VAL_DATA_PATH,
                                    transform=tfs.data_center),
                               datasets.ImageFolder(
                                    VAL_DATA_PATH,
                                    transform=tfs.data_rotate),
                               datasets.ImageFolder(
                                    VAL_DATA_PATH,
                                    transform=tfs.data_jitter_hue)]), 
                                    batch_size=args.batch_size, 
                                    shuffle=False, num_workers=2, 
                                    pin_memory=HAS_GPU)

if COLAB:
    GTSRB_test_dataset = GTSRB_Test('/content/')
    
else:
    GTSRB_test_dataset = GTSRB_Test('data/')

assert len(GTSRB_test_dataset) == 12630

test_loader = data.DataLoader(GTSRB_test_dataset, 
    batch_size=BATCH_SIZE, shuffle=False, 
    num_workers=2)
print('Augmented dataset: %d training samples & %d validation samples\n' % (
    len(train_loader.dataset), len(val_loader.dataset)))

# Get labels from concatenated datasets.
train_dataset_list = train_loader.dataset.datasets
train_concat_labels = []
for ds in train_dataset_list:
    train_concat_labels.extend(ds.targets)

val_dataset_list = val_loader.dataset.datasets
val_concat_labels = []
for ds in val_dataset_list:
    val_concat_labels.extend(ds.targets)

class_labels = range(43)

print('Distribution of classes in augmented train dataset:')
fig, ax = plt.subplots()
_, counts = np.unique(train_concat_labels, return_counts=True)
ax.bar(class_labels, counts)
ax.set_xticks(class_labels, minor=True)
plt.show()
from model.models import Net1

# Reset
model = None
optimizer = None
scheduler = None

model = Net1()
if HAS_GPU:
    model.cuda()

# Optimizer is updated to *not* include non-gradient weights.    
optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()),
                                                      lr=args.learning_rate)
# Reduce learning rate when a metric has stopped improving. 
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min',
                                                 patience=5,
                                                 factor=0.5,
                                                 verbose=True)
# Reset these
training_accuracy = []
validation_accuracy = []
train_losses = []
train_accuracies = []
val_losses = []
val_accuracies = []

test_accuracy = []
confusion = [] 
test_acc = None
confusion_mtx = None
epoch = None

for epoch in range(1, args.num_epochs + 1):
    training_accuracy.append(train(epoch))
    validation_accuracy.append(validate(epoch))
    test_acc, confusion_mtx = test(test_loader)
    test_accuracy.append(test_acc)
    confusion.append(confusion_mtx)
    print()
    print('\r --> Testing set: Accuracy (Best) = %2.2f%%' % max(test_accuracy), 
          end='', flush=True)
    print()
import matplotlib.pyplot as plt

# Plot accuracy
plt.plot(training_accuracy[0],'-o', markersize=.5)
plt.plot(validation_accuracy[0],'-o', markersize=.5)
plt.xlabel('Sample # per Epoch')
plt.ylabel('Accuracy')
plt.legend(['Train','Valid'])
plt.title('Train vs Valid Accuracy per Epoch (Best)')
None
# Reset
training_accuracy = None
validation_accuracy = None
visual(test_accuracy, confusion, class_labels)
# ---------------------------------------------------------------------
# Experiment Demonstrating Precision For Multi-Class Predictions
# ---------------------------------------------------------------------

# Calculate precision for a 1:1:100 dataset (50TP, 20FP, 99TP, 51FP)
from sklearn.metrics import precision_score

# Define actuals
actual_positive1 = [1 for _ in range(100)]
actual_positive2 = [2 for _ in range(100)]
actual_negative = [0 for _ in range(10000)]
y_true = actual_positive1 + actual_positive2 + actual_negative

# Define predictions
predicted_positive1 = [0 for _ in range(50)] + [1 for _ in range(50)]
predicted_positive2 = [0 for _ in range(1)] + [2 for _ in range(99)]
predicted_negative = [1 for _ in range(20)] + [2 for _ in range(51)] + [0 for _ in range(9929)] 
y_pred = predicted_positive1 + predicted_positive2 + predicted_negative

# calculate prediction
prec = precision_score(y_true, y_pred, labels=[1,2], average='micro') 
print('Precision: %.3f' % prec)
# fin<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1059285091543156, 'fraction_numerical': 0.025234912331686524, 'mean_word_length': 2.4683352931295146, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 12, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 56, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '69491', 'license': 'MIT', 'n_tokens_mistral': 6708, 'n_tokens_neox': 6241, 'n_words': 1493, 'path': 'project-I/main.ipynb', 'repo_name': 'geraldmc/DL4CV-2022', 'size': '20680'}