<jupyter_start><jupyter_text>DATA20001 Deep Learning - Group Project Image project**Due Thursday, December 13, before 23:59.**The task is to learn to assign the correct labels to a set of images.  The images are originally from a photo-sharing site and released under Creative Commons-licenses allowing sharing.  The training set contains 20 000 images. We have resized them and cropped them to 128x128 to make the task a bit more manageable.We're only giving you the code for downloading the data. The rest you'll have to do yourselves.Some comments and hints particular to the image project:- One image may belong to many classes in this problem, i.e., it's a multi-label classification problem. In fact there are images that don't belong to any of our classes, and you should also be able to handle these correctly. Pay careful attention to how you design the outputs of the network (e.g., what activation to use) and what loss function should be used.- As the dataset is pretty imbalanced, don't focus too strictly on the outputs being probabilistic. (Meaning that the right threshold for selecting the label might not be 0.5.)- Image files can be loaded as numpy matrices for example using `imread` from `matplotlib.pyplot`. Most images are color, but a few grayscale. You need to handle the grayscale ones somehow as they would have a different number of color channels (depth) than the color ones.- In the exercises we used e.g., `torchvision.datasets.MNIST` to handle the loading of the data in suitable batches. Here, you need to handle the dataloading yourself.  The easiest way is probably to create a custom `Dataset`. [See for example here for a tutorial](https://github.com/utkuozbulak/pytorch-custom-dataset-examples).<jupyter_code>! pip install torch torchvision torchsummary sklearn
!pip install pillow==4.1.1
%reload_ext autoreload
%autoreload<jupyter_output><empty_output><jupyter_text>Download the data<jupyter_code>import os
import torch
import torchvision
from torchvision.datasets.utils import download_url
import zipfile
train_path = 'train'
dl_file = 'dl2018-image-proj.zip'
dl_url = 'https://users.aalto.fi/mvsjober/misc/'

zip_path = os.path.join(train_path, dl_file)
if not os.path.isfile(zip_path):
    download_url(dl_url + dl_file, root=train_path, filename=dl_file, md5=None)

with zipfile.ZipFile(zip_path) as zip_f:
    zip_f.extractall(train_path)
    #os.unlink(zip_path)
if torch.cuda.is_available():
    print('Using GPU!')
    device = torch.device('cuda')
else:
    print('Using CPU')
    device = torch.device('cpu')<jupyter_output>Using GPU!<jupyter_text>The above command downloaded and extracted the data files into the `train` subdirectory.The images can be found in `train/images`, and are named as `im1.jpg`, `im2.jpg` and so on until `im20000.jpg`.The class labels, or annotations, can be found in `train/annotations` as `CLASSNAME.txt`, where CLASSNAME is one of the fourteen classes: *baby, bird, car, clouds, dog, female, flower, male, night, people, portrait, river, sea,* and *tree*.Each annotation file is a simple text file that lists the images that depict that class, one per line. The images are listed with their number, not the full filename. For example `5969` refers to the image `im5969.jpg`.  Your stuff goes here ...<jupyter_code>import os
from os import listdir
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import random as rd
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import numpy as np
img_path = "./train/images/"
labels_path = "./train/annotations/"<jupyter_output><empty_output><jupyter_text>Data preprocessing<jupyter_code>images_path = "train/images/" 
image_files = [(images_path+f) for f in os.listdir(images_path) \
                if os.path.isfile(os.path.join(images_path, f))]
images = []
for current_img_path in image_files:
    current_img_vector =  plt.imread( current_img_path )
    filename_without_extension = os.path.basename(current_img_path)[:-4] # :-4 to drop the extension
    img_num = filename_without_extension[2:] # drop the heading "im"
    images += [img_num]
del image_files
data=images
labels = [[] for _ in range(len(data))] # Empty list of labels
annotations_path = "train/annotations/"
annotations_files = [(annotations_path+f) for f in os.listdir(annotations_path) \
        if os.path.isfile(os.path.join(annotations_path, f))]
for file in annotations_files:
    label = os.path.basename(file)[:-4] # :-4 to drop the extension
    with open(file) as f: # file is the name of the label currently read
        img_nums = f.read().split()
    for img_num in img_nums:
        if img_num not in data:
            continue
        labels[data.index(img_num)].append(label)
class OneHotTransformer:
    def __init__(self):
        annotations_files = [(annotations_path+f) for f in os.listdir(annotations_path) \
                    if os.path.isfile(os.path.join(annotations_path, f))]
        self.vocabulary = []
        for file in annotations_files:
            label = os.path.basename(file)[:-4] # :-4 to drop the extension
            self.vocabulary.append(label)
    def toOneHot(self, labels):
        res = [0]*len(self.vocabulary)
        for i, voc in enumerate(self.vocabulary):
            if voc in labels:
                res[i] = 1
        return np.array(res)
    def toVocabulary(self, onehot):
        res = []
        for i, n in enumerate(onehot):
            if n: res.append(self.vocabulary[i])
        return res
oneHotTransformer = OneHotTransformer()
# Files to pytorch dataset
# Inspired from https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py
from torch.utils.data.dataset import Dataset
from torchvision import datasets, transforms
from PIL import Image

#merging_method : 
# 0 : Convert  a colored image to grey image
# 1 : Complete a grey    image with zeros   to a have colored image
# 2 : Complete a grey    image with himself to a have colored image


class imageDataset(Dataset):
    def __init__(self, data_basenames, merging_method = 2, channels = 3, height = 128, width = 128, transforms = transforms.Compose([
        #transforms.Resize(256),
        #transforms.CenterCrop(224),
        transforms.ToTensor(),
    #    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
                train=True):
        """
        data_basenames is a list of id of image, such as ["9885", "9886"]
        transforms is a list of transformation. it last element should be transforms.ToTensor()
        """
        def parse_labels(data_paths):
            """ Parse to one hot """
            labels = [[] for _ in range(len(data_paths))] # Empty list of labels
            annotations_path = "train/annotations/"
            annotations_files = [(annotations_path+f) for f in os.listdir(annotations_path) \
                    if os.path.isfile(os.path.join(annotations_path, f))]
            vocabulary = []
            for file in annotations_files:
                label = os.path.basename(file)[:-4] # :-4 to drop the extension
                vocabulary.append(label)
                with open(file) as f: # file is the name of the label currently read
                    img_nums = f.read().split()
                for img_num in img_nums:
                    if img_num not in data_paths:
                        continue
                    labels[data_paths.index(img_num)].append(label)
            return labels
          
        self.data = data_basenames
        self.labels = parse_labels(data_basenames)
        self.channels = channels
        self.height = height
        self.width = width
        self.transforms = transforms
        self.data_len = len(data_basenames)
        self.merging_method = merging_method
        self.train = train
        
    def __getitem__(self, index):
        if self.train:
          img_path = "train/images/im" + self.data[index] + ".jpg"
        else:
          img_path = "to_test/images/im" + self.data[index] + ".jpg"
        data = Image.open(img_path)
        
        if self.transforms is not None:
            img = self.transforms(data)
        #labels = np.array(self.labels[index])
        #print(self.labels[index])
        
        if(self.merging_method == 0 ):
            
            if img.shape[0] == 3:
              img = img[0] + img[1] + img[2]
              img /= 3
              img = img.reshape((1, 128, 128))
              
        elif(self.merging_method == 1):
          
            if img.shape[0] == 1:  # change graysclae images to RGB
                zeros_torser = torch.zeros([1, 128, 128])
                img = torch.cat((img,zeros_torser, zeros_torser),0)
                
        elif(self.merging_method == 2):  
            if img.shape[0] == 1:  # change graysclae images to RGB
                img = torch.cat((img,img, img),0)
        
            
        labels = self.labels[index]
        one_hot_labels = oneHotTransformer.toOneHot(labels)
        return img, one_hot_labels

    def __len__(self):
        return self.data_len

import random
random.shuffle(images) # So we make sure test data has all kinds of label, in case
                                 # the repartition is not even according to label names

batch_size = 512

# 80% train, 20% test
index_cut =  int(len(images)*0.8)
merge_method = 2

def build_augmented_dataset(images_basenames):
  train_dataset_orig = imageDataset(images_basenames[:index_cut], merging_method = merge_method)
  train_dataset_augmentation = imageDataset(images_basenames[:index_cut],
                                         merging_method = merge_method,
                                         transforms = transforms.Compose([
                                             transforms.RandomRotation(40),
                                             transforms.ToTensor(),
                                         ])
                                        )
  train_dataset_augmentation2 = imageDataset(images_basenames[:index_cut],
                                         merging_method = merge_method,
                                         transforms = transforms.Compose([
                                             transforms.RandomRotation(20),
                                             transforms.ToTensor(),
                                         ])
                                        )


  train_dataset_augmentation3 = imageDataset(images_basenames[:index_cut],
                                         merging_method = merge_method,
                                         transforms = transforms.Compose([
                                             transforms.RandomCrop([128, 128]),
                                             transforms.ToTensor(),
                                         ])
                                        )
  
  train_dataset = torch.utils.data.ConcatDataset([train_dataset_orig,
                                                  train_dataset_augmentation,
                                                 train_dataset_augmentation2,
                                                 train_dataset_augmentation3])
  return train_dataset

train_dataset = build_augmented_dataset(images[:index_cut])
  
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                          batch_size=batch_size,
                                          shuffle=True)

test_dataset = imageDataset(images[index_cut:], merging_method = merge_method)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=True)
# 3 channel show
data, target = next(train_loader.__iter__()) # Iteration works!

d0 = data[0]
print(d0)
t0 = target[0]

f, axs = plt.subplots(1,3)
axs[0].imshow(d0[0])
axs[1].imshow(d0[1])
axs[2].imshow(d0[2])
plt.show()

arr_ = d0.transpose(0,-1)
arr_ = np.rot90(arr_, 3)
arr_ = np.flip(arr_, axis=1)
plt.imshow(arr_)
plt.grid(None)
plt.show()
print(t0)
print("labels:", oneHotTransformer.toVocabulary(t0))<jupyter_output>tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]],

        [[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]],

        [[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]])<jupyter_text>Final Models  a) Customized model<jupyter_code>class deep_cnn_v5(nn.Module):
    def __init__(self, input_channels=1):
        super().__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(input_channels, 16, kernel_size=3),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(5, stride=3)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(5, stride=3)
        )
        self.conv3 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=7),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(5, stride=3)
        )
        
        self.fc1 = nn.Linear(64, 32)
        self.fc2 = nn.Sequential(
   #         nn.Dropout(p=0.5),
            nn.Linear(32, 14)
        )
        
    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.conv3(out)
        out = out.view(out.size(0),-1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
      
from torchsummary import summary
summary(model, (3, 128, 128))<jupyter_output>----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 16, 126, 126]             448
       BatchNorm2d-2         [-1, 16, 126, 126]              32
              ReLU-3         [-1, 16, 126, 126]               0
         MaxPool2d-4           [-1, 16, 41, 41]               0
            Conv2d-5           [-1, 32, 37, 37]          12,832
       BatchNorm2d-6           [-1, 32, 37, 37]              64
              ReLU-7           [-1, 32, 37, 37]               0
         MaxPool2d-8           [-1, 32, 11, 11]               0
            Conv2d-9             [-1, 64, 5, 5]         100,416
      BatchNorm2d-10             [-1, 64, 5, 5]             128
             ReLU-11             [-1, 64, 5, 5]               0
        MaxPool2d-12             [-1, 64, 1, 1]               0
           Linear-13                  [...]<jupyter_text>b)  Pretrained models<jupyter_code>import torchvision.models as models

resnet18 = models.resnet18(pretrained=True)
num_input = resnet18.fc.in_features
resnet18.fc = nn.Linear(num_input, 14)
model = resnet18.to(device)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>def train_net(model, epoch, trainLoss, interval = 10):
    # set model to train
    model.train()
    loss = 0
    #acc = 0
    
    for batch_idx, (data, target) in enumerate(train_loader):
        
        data = data.to(device) # to GPU
        target = target.float().to(device) # to GPU, ".float()" is improtant, otherwise got error.
        
        # set zero gradient buffer
        optimizer.zero_grad()
        
        # compute the output and Loss
        output = model(data)
        Loss = criterion(output, target)
        
        # back prop
        Loss.backward()
        
        # update weights
        optimizer.step()
        
        # print information
        if batch_idx % interval == 0:
            print("Train Epoch: {} [{}/{} {:.0f}%]\tLoss: {:.6f}".format(epoch, len(data)*batch_idx, len(train_loader)*len(data), batch_idx/len(train_loader)*100, Loss.data.item()))   
       
        loss += Loss
    loss/=len(train_loader)
    trainLoss.append(loss)

def validate(model, lossv, verbose=True):
  loss, acc, prec, rec = 0,0,0,0
  for i, sample in enumerate(test_loader):
    inputv, targetv = sample
    inputv = inputv.to(device)
    output = model(inputv)
   
    loss += criterion(output, targetv.data.to(device).float()).data.item()
  
  loss /= len(test_loader)
  
  lossv.append(loss)
  
  if verbose:
    print('Validation set: Average loss: {:.4f}\n'.format(
            loss))
%%time
#model = one_layer_cnn().to(device)
#model = sim_cnn_v1().to(device)
model = deep_cnn_v5(input_channels=3).to(device)
#model = lstm()
epoch = 15

#learning_rate = 0.000001 # for sim_cnn_v1
learning_rate = 0.0001 # for deep_cnn
optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate) #Adam
criterion = nn.MultiLabelSoftMarginLoss()
#criterion = nn.BCELoss()
train_loss, val_loss = [], []
for i in range(1,1+epoch):
    train_net(model, i, train_loss)
    validate(model, val_loss)
plt.title("Model loss")
plt.plot(train_loss, label="train")
plt.plot(val_loss, label="validation")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend()
plt.show()
# The schema allows to see if the model was properly fitted
# that allows us to learn if the initial learning rate was appropriate
plt.title("Model loss")
plt.plot(train_loss, label="train")
plt.plot(val_loss, label="validation")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Treshold optimizationDISCLAIMER : Does not work as intended.In that part we strive to optimize the treshold of activation of each label, in order to maximize F1 score<jupyter_code># Concatenating all the batches _label_ output in order to not compute it several 
# times for F1 score optimization.
import sklearn.metrics
def make_pred(bounded_output_all, tresholds):
  return bounded_output_all.data.gt(tresholds) # Activation

def compute_f1(tresholds_as_tensor):
  f1 = 0 # Will be the average f1 over all batches (according to their sizes)
  for i, (inputv, targetv) in enumerate(test_loader):
    inputv = inputv.to(device)
    targetv = targetv.to(device)
    
    batch_size = len(inputv)
    bounded_output = torch.sigmoid(model(inputv))
    
    pred = bounded_output.data.gt(tresholds_as_tensor)
    curr_f1 = sklearn.metrics.f1_score(targetv.cpu(), pred.cpu(), average="micro") # same result
    f1 += curr_f1 * (batch_size)
  return f1/len(test_dataset)
ounded_output = torch.sigmoid(model(inputv))
    plt.xlabel("treshold")
    plt.ylabel("F1 score")
    for label in range(0,14):
      plt.plot(tresholds , tab[label], label=oneHotTransformer.vocabulary[label])
      imax = np.argmax(tab[label])
      plt.scatter(tresholds[imax], tab[label][imax], marker="+")
    plt.legend()
    plt.show()
# Optimizing treshold for each label : Iterating over labels
# Dropout layers have the effect of making a network not returning the same output for a same output at different runs
# This means that F1 score function will be "noisy", ie it won't be a smooth curve.
# This makes optimization much more complicate than simply a SGD descent. To be tolerant of that case, 
# we check all values in order to not be tricked by noise, and choose the observed maximum.

    
class tresholdOptimizer():
  def __init__(self, paces=10, max=1): # High pace means more precision but slower computation
    self.tresholds = np.linspace(0,max,paces)
    for i in range(14): # label
      current_tresholds = [1 for _ in range(14)] # 1 so other labels are never activated,
                                                 # and only the currently studied treshold make the f1 vary
      f1 = []
      print("label", i)
      for t in self.tresholds:
        current_tresholds[i] = t
        tresholds_as_tensor = torch.Tensor(current_tresholds).to(device) 
        f1_curr = compute_f1(tresholds_as_tensor)
        f1.append(f1_curr)
        #print("  t=", t, "\t f1 =", f1_curr)
      if i==0:
        self.tab = [f1]
      else:
        self.tab.append(f1)
        
  def view(self):
    plt.xlabel("treshold")
    plt.ylabel("F1 score")
    for label in range(0,14):
      plt.plot(self.tresholds , self.tab[label], label=oneHotTransformer.vocabulary[label])
      imax = np.argmax(self.tab[label])
      plt.scatter(self.tresholds[imax], self.tab[label][imax], marker="+")
    plt.legend()
    plt.show()

  def get_best_tresholds(self):
    res = []
    for label in range(0,14):
      imax=np.argmax(self.tab[label])
      res.append(self.tresholds[imax])
    return res

treshold_opt = tresholdOptimizer(max=0.4) 
treshold_opt.view()
print("tresholds:", treshold_opt.get_best_tresholds())
# CAREFUL! It appears that my implementation does not work. A constant treshold of 0.2
# appear to yield much better f1 than my implementation. Some fields are detected as performing best with 0, 
# with our optimizer which is wrong as shown below.

# tresholds = treshold_opt.get_best_tresholds() # Is actually not the best..!


tresholds = treshold_opt.get_best_tresholds() # Is actually not the best..!)      
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
f1 = compute_f1(tresholds_as_tensor)
#f1 = sklearn.metrics.f1_score(truth_all.cpu(), pred_all.cpu(), average="micro") # same result
print("tresholds", tresholds_as_tensor)
print("F1 score : {:.2f}%\n".format(f1*100))<jupyter_output>tresholds tensor([0.0000, 0.0000, 0.0444, 0.0444, 0.0889, 0.0000, 0.0889, 0.0000, 0.0444,
        0.0000, 0.0444, 0.1333, 0.0000, 0.0000], device='cuda:0')
F1 score : 15.01%<jupyter_text>F1 score<jupyter_code>import sklearn.metrics

def compute_f1_acc(tresholds_as_tensor):
  f1 = 0 # Will be the average f1 over all batches (according to their sizes)
  for i, (inputv, targetv) in enumerate(test_loader):
    inputv = inputv.to(device)
    targetv = targetv.to(device)
    
    batch_size = len(inputv)
    bounded_output = torch.sigmoid(model(inputv))
    
    pred = bounded_output.data.gt(tresholds_as_tensor)
    curr_f1 = sklearn.metrics.f1_score(targetv.cpu(), pred.cpu(), average="micro") # same result
    f1 += curr_f1 * (batch_size)
    return 
#tresholds = [0.2 for _ in range(14)]
tresholds = [0.2, 0.2,
             0.2, 0.2,
             0.2, 0.2,
             0.2, 0.2,
             0.2, 0.2,
             0.2, 0.2,
             0.2, 0.2
          ]
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
f1 = compute_f1(tresholds_as_tensor)
print("tresholds", tresholds_as_tensor)
print("F1 score : {:.2f}%\n".format(f1*100))
#tresholds = [0.2 for _ in range(14)]
tresholds = [0.3, 0.2,
             0.2, 0.2,
             0.225, 0.25,
             0.2, 0.25,
             0.225, 0.35,
             0.2, 0.3,
             0.25, 0.2
          ]
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
f1 = compute_f1(tresholds_as_tensor)
f1s = 0
for _ in range(5):
  f1s += compute_f1(tresholds_as_tensor)
f1s /= 5
print("tresholds", tresholds_as_tensor)
print("F1 avg score : {:.2f}%\n".format(f1s*100))
#tresholds = [0.2 for _ in range(14)]
tresholds = [0.3, 0.2,
             0.2, 0.2,
             0.225, 0.25,
             0.2, 0.25,
             0.225, 0.35,
             0.2, 0.3,
             0.25, 0.25
          ]
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
f1 = compute_f1(tresholds_as_tensor)
f1s = 0
for _ in range(5):
  f1s += compute_f1(tresholds_as_tensor)
f1s /= 5
print("tresholds", tresholds_as_tensor)
print("F1 avg score : {:.2f}%\n".format(f1s*100))
# Deprecated cell!
# Result with """ Optimized """ tresholds
from sklearn.metrics import classification_report
tresholds = treshold_opt.get_best_tresholds() # Is actually not the best..!)      
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
pred_all = make_pred(bounded_output_all, tresholds_as_tensor)

target_names = oneHotTransformer.vocabulary
#report = classification_report(truth_all.cpu(), pred_all.cpu(), target_names=target_names, output_dict=True)
report = classification_report(truth_all.cpu(), pred_all.cpu(), target_names=target_names)
print("with tresholds", list(zip(target_names, tresholds)))
print(report)
# Apparently, the optimization is good for all f1-score avg _EXCEPT_ micro (compare with the cell below)
# Deprecated cell!
# Result with rubbish tresholds
from sklearn.metrics import classification_report
tresholds = [0.2 for _ in range(14)]
tresholds_as_tensor = torch.Tensor(tresholds).to(device)
pred_all = make_pred(bounded_output_all, tresholds_as_tensor)

target_names = oneHotTransformer.vocabulary
#report = classification_report(truth_all.cpu(), pred_all.cpu(), target_names=target_names, output_dict=True)
report = classification_report(truth_all.cpu(), pred_all.cpu(), target_names=target_names)
print("with tresholds", list(zip(target_names, tresholds)))
print(report)
# Have a look at results
# So we can know what error is common
for i, sample in enumerate(test_loader):
    inputv, targetv = sample
    inputv = inputv.to(device)
    output = model(inputv)
    
    #print("Prediction\n", output)
    tresholds = [0.3, 0.2,
             0.2, 0.2,
             0.225, 0.25,
             0.2, 0.25,
             0.225, 0.35,
             0.2, 0.3,
             0.25, 0.25
          ]
    tresholds_as_tensor = torch.Tensor(tresholds).to(device) 
    pred = output.data.gt(tresholds_as_tensor) # Activated output
    print("Prediction\n", pred)
    print("Truth\n", targetv)
    break<jupyter_output>Prediction
 tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0]], device='cuda:0', dtype=torch.uint8)
Truth
 tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 1, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 1, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0]])<jupyter_text>Save your modelIt might be useful to save your model if you want to continue your work later, or use it for inference later.<jupyter_code>torch.save(model.state_dict(), 'model.pkl')<jupyter_output><empty_output><jupyter_text>The model file should now be visible in the "Home" screen of the jupyter notebooks interface.  There you should be able to select it and press "download".  [See more here on how to load the model back](https://github.com/pytorch/pytorch/blob/761d6799beb3afa03657a71776412a2171ee7533/docs/source/notes/serialization.rst) if you want to continue training later.  Download test setThe testset will be made available during the last week before the deadline and can be downloaded in the same way as the training set.<jupyter_code>to_test_path = 'to_test'
dl_file = 'dl2018-image-test.zip'
dl_url = 'https://users.aalto.fi/~mvsjober/misc/'

zip_path = os.path.join(to_test_path, dl_file)
if not os.path.isfile(zip_path):
    download_url(dl_url + dl_file, root=to_test_path, filename=dl_file, md5=None)

with zipfile.ZipFile(zip_path) as zip_f:
    zip_f.extractall(to_test_path)<jupyter_output>Downloading https://users.aalto.fi/~mvsjober/misc/dl2018-image-test.zip to to_test/dl2018-image-test.zip<jupyter_text>Predict for test setYou should return your predictions for the test set in a plain text file.  The text file contains one row for each test set image.  Each row contains a binary prediction for each label (separated by a single space), 1 if it's present in the image, and 0 if not. The order of the labels is as follows (alphabetic order of the label names):    baby bird car clouds dog female flower male night people portrait river sea treeAn example row could like like this if your system predicts the presense of a bird and clouds:    0 1 0 1 0 0 0 0 0 0 0 0 0 0    The order of the rows should be according to the numeric order of the image numbers.  In the test set, this means that the first row refers to image `im20001.jpg`, the second to `im20002.jpg`, and so on. If you have the prediction output matrix prepared in `y` you can use the following function to save it to a text file.<jupyter_code>order = ['baby', # given order, most likely different from the order constructed
 'bird',         # automatically in the preprocessing part.
 'car',
 'clouds',
 'dog',
 'female',
 'flower',
 'male',
 'night',
 'people',
 'portrait',
 'river',
 'sea',
 'tree']
# Permutation table (arrays' indices to permute) 
switch = [0 for _ in range(14)]
for i in range(14):
  searched =  order[i]
  switch[i] = oneHotTransformer.vocabulary.index(searched)

def batch_output_in_right_order(batch_pred, order_switch):
  return batch_pred[:, order_switch]
  

# Example for one batch
inputv, targetv = next(test_loader.__iter__())
inputv = inputv.to(device)
best_tresholds_as_tensor = torch.Tensor([0.2]*14).to(device)
batch_pred = torch.sigmoid(model(inputv).to(device)) > best_tresholds_as_tensor

batch_right_order = batch_output_in_right_order(batch_pred, switch)

# Double check it works
print("Truth: ", targetv[0])
print("Labelled Truth: ", oneHotTransformer.toVocabulary(targetv[0]))

ex0 = batch_right_order[0]


print("-"*10)
print("new order", order)
#print(inputv.cpu().numpy()[0][0])


targetv_good_order = batch_output_in_right_order(targetv, switch)
t0 = targetv_good_order[0]
plt.imshow(inputv.cpu().numpy()[0][0], cmap="gray")
#print(batch_output_in_right_order(batch_pred, switch))
print("re ordered", t0)
plt.show()


images_path_test = "to_test/images/" 
image_files_test = [(images_path_test+f) for f in os.listdir(images_path_test) \
                if os.path.isfile(os.path.join(images_path_test, f))]

image_files_test = sorted(image_files_test)
def to_basename(rel_path):
  """ 'to_test/images/im20001.jpg' to '20001' """
  return rel_path[17:-4] # stripping 17 first chars and last 4 ones (extension of the file)
basenames_test = [to_basename(path) for path in image_files_test] 

to_test_dataset = imageDataset(basenames_test, merging_method = merge_method, train=False)
to_test_loader = torch.utils.data.DataLoader(dataset=to_test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)
# load the model
#model = deep_cnn_v5(input_channels = 3).to(device)

resnet18 = models.resnet18(pretrained=True)
num_input = resnet18.fc.in_features
resnet18.fc = nn.Linear(num_input, 14)
model = resnet18.to(device)

model.load_state_dict(torch.load('/content/gdrive/My Drive/model.pkl'))
# Concatenate all predictions
tresholds = [0.3, 0.2,
             0.2, 0.2,
             0.225, 0.25,
             0.2, 0.25,
             0.225, 0.35,
             0.2, 0.3,
             0.25, 0.25
          ] # Optimized treshold
for i, (inputv, targetv) in enumerate(to_test_loader):
  inputv = inputv.to(device)
  best_tresholds_as_tensor = torch.Tensor(tresholds).to(device)
  batch_pred = torch.sigmoid(model(inputv).to(device)) > best_tresholds_as_tensor
  
  batch_right_order = batch_output_in_right_order(batch_pred, switch)
  print(i+1, "/", len(to_test_loader))
  if i==0:
    y = batch_right_order
  else:
    y = torch.cat((y, batch_right_order), 0) # Adding new batch to the list of predictions
np.savetxt('results.txt', y.cpu(), fmt='%d')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12221871049304678, 'fraction_numerical': 0.035176991150442476, 'mean_word_length': 2.5503815080789947, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 63, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '314592', 'license': 'MIT', 'n_tokens_mistral': 10954, 'n_tokens_neox': 10015, 'n_words': 2744, 'path': 'image_project.ipynb', 'repo_name': 'HappyBearDay/Deep_Learning_Image_Multi_label', 'size': '31640'}