<jupyter_start><jupyter_text>**CS 464****Introduction to Machine Learning****Fall 2021****Homework 3**Due: Jan 02, 2022 17:00 (GMT+3)  Instructions        This homework contains both written and programming questions about neural networks. You should implement programming questions on this notebook. Your plots should also be produced in this notebook. Each programming question has its own cell for your answer. You can implement your code directly in these cells, or you can call required functions which are defined in a different location for the given question.            For questions that you need to plot, your plot results have to be included in the cell output. For written questions, you may provide them either as comments in code cells or as seperate text cells.             You are NOT ALLOWED to use different libraries than given libraries in the code segments of this homework except for libraries inclueded in Python Standard Library (https://docs.python.org/3/library/).            You are NOT ALLOWED to use a different deep learning framework than PyTorch.            While submitting the homework file, please package notebook(".ipynb") and model (".pth") files as a gzipped TAR file or a ZIP file with the name cs464_hw3_section_Firstname_Lastname. Please do not use any Turkish letters for any of your files including code files and model files. Upload your homework to Moodle.            This is an individual assignment for each student. That is, you are NOT allowed to share your work with your classmates.         If you do not follow the submission routes, deadlines and specifications, it will lead to a significant grade deduction.             If you have any questions, please contact "<EMAIL>".      Environment Setup This homewrok is prepeared by using Google CoLab which already has required libraries. However, if you are using your own local Jupyter or any other Python notebook editor, you may use both anaconda or pip to install PyTorch to your own computer.  Anaconda Installation    Download anaconda from https://www.anaconda.com/download    Follow the instructions provided in https://conda.io/docs/user-guide/install/index.htmlregular-installation Creation of Virtual Environment    Create python3.7 virtual environment for your hw3 using follow command from the command line        > conda create -n HW3 python=3.7 anaconda    Activate your virtual environment        > source activate HW3    To install auxiliary libraries, replace the "package_name" in the following command and run it in activated "hw3" environment         > pip install "package_name"     When you create your virtual environment with "anaconda" metapackage, jupyter notebook should be installed. Try:         > jupyter notebook Pytorch Installation with AnacondaYou should install PyTorch to your virtual environment which is created for the hw3. Therefore, you should activate your homework virtual environment before to start PyTorch installation.> source activate HW3After you have activated the virtual environment, then use one of the following commands to install pytorch for CPU for your system. See https://pytorch.org/ for help.For MacOS:    > conda install pytorch torchvision -c pytorchFor Linux:    > conda install pytorch-cpu torchvision-cpu -c pytorchFor Windows:    > conda install pytorch-cpu torchvision-cpu -c pytorch Pip3 Installation    Download pip3 from https://pip.pypa.io/en/stable/installing/    If you are using Windows, you may need to add Python to your enviroment variables. You may use the following tutorial to install Python and pip.    https://phoenixnap.com/kb/how-to-install-python-3-windows PyTorch Installation with PipFor MacOS:    > pip3 install torch torchvisionFor Linux:    > pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.htmlFor Windows:    > pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html Question 1 [12 pts.]Answer the given questions with **at most a sentence**.  >a) Why do people use validation data?**Validation set is used for hyperparameter selection and model selection. If hyperparameter tuning will be done, it will not be possible to do it on test set, because we do not want our algorithm to see test set before the test time. On the other hand, if we use training set for hyperparameter tuning, then we will fit our hyperparameters on the data where we have trained our model on, so this will make our model overfit to the training data.**  >b) What is the difference between mean squared error and mean absolute error?  **They are mostly used in linear regression. Both metrics measure the difference between the actual and the predicted values of y. MSE takes the square of the differences to get rid of the minus results if there is, whereas MAE takes the absolute of the differences. MSE makes the errors big errors even bigger by taking the squares, on the other hand MAE is more robust and shows the real difference instead of squares.**  >c) What is the main problem of using sigmoid as activation function in an artificial neural network (ANN)?**The sigmoid function's output is between [0,1]. However, looking to the sigmoid curve, when our z value is lower than say -2 or higher than 2, the derivatives will be all 0. This will make it harder to backpropagate because the gradients will be 0 or very close to 0. This will lead to a problem called vanishing gradients. Another problem with the sigmoid activation is that, when dealing with neural networks we prefer activation functions that are zero centered. This will make unsmooth learning curve (loss vs iteration).**>d) What does it mean to overfit your data model?**When training our model on the data, if all the variance or more than enough variance from our training set comes to the learning and if our model becomes predicting nearly perfectly to a training set but fails dramatically when it encounters a dataset it never saw, we call this overfitting to the data. This problem is caused by small data set, too complex models or models that are too sensitive to the noise in data.**  >e) Your input image size is 3x64x64. If you apply 3x3 convolution with input_channel=3, output_channel=6, padding=0, stride=2, what would be the size of the output?**Output size = (image volume - filter size + 2 x padding) / stride + 1Output size = (64 - 3 + 2 x 0) / 2 + 1 = 31.5 (roundoff to 31)So the output will be (6x31x31)**  >f) In the previous question, how many trainable parameters are there? (you should also consider bias terms in addition to weights)  **The number of trainable parameters will be (3x3x3+1)x6 = 168** Question 2 [88 pts.] Computer vision (CV) is the field of study that deals with how computers can gain high-level understanding from digital images or videos. Your task for this question is to classify scenes according to their contexts by using simple machine learning algorithms developed for CV problems on scene images.Your dataset consist of scene images from 4 contexts. Images of each context is stored under separate folders in the compressed file given to you.  The dataset has been processed in such a way that each class has approximately 2500 samples.Download the dataset from the following link:https://drive.google.com/file/d/1l51t3aTY7B131fwq92ACI_b_D5Idq5In/view?usp=sharingLibraries that are required in this question is given in the following code cell.<jupyter_code># PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models

# To Read Data
from torch.utils.data import Dataset, DataLoader
import numpy as np
from PIL import Image
# To Interpret results & obtain plots
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score, accuracy_score
import matplotlib.pyplot as plt

# You could add your own libraries form Python Standard Library in this cell. Any other external libraries are not allowed.
import glob
import random
import itertools
import time<jupyter_output><empty_output><jupyter_text>Data Loader [6 pts.]An important part of such a task is to implement your own data loader. In this homework, a partial loader is provided to you. This loader is going to be based on a base class named "Dataset", provided in PyTorch library. You need to complete the code below to create your custom "SceneDataset" class which will be able to load your dataset. Implement the functions whose proptotypes are given. Follow the TODO notes below. You have to divide the files into three sets as train (70%), validation (10%) and **test (20%)** sets.  These non-overlapping splits, which are subsets of SceneDataset, should be retrieved using the "get_dataset" function.Hint: The dataset is not normalized and your results will heavily depend on your input.<jupyter_code>train_data_path = "/mnt/gunkaynar/project/hw3/data"

train_image_paths = [] 
classes = [] 

for data_path in glob.glob(train_data_path + '/*'):
    classes.append(data_path.split('/')[-1]) 
    train_image_paths.append(glob.glob(data_path + '/*'))
train_image_paths = np.array(train_image_paths,dtype=object)
train_image_paths_new = []
for lists in train_image_paths:
    for item in lists:
        train_image_paths_new.append(item)
train_image_paths_new = np.array(train_image_paths_new)
train_image_paths_new = (np.ndarray.flatten(train_image_paths_new))
random.shuffle(train_image_paths_new)

print('total number of images: ', np.shape(train_image_paths_new))


train_image_paths, valid_image_paths, test_image_paths = train_image_paths_new[:int(0.7*len(train_image_paths_new))], train_image_paths_new[int(0.7*len(train_image_paths_new)):int(0.8*len(train_image_paths_new))] , train_image_paths_new[int(0.8*len(train_image_paths_new)):]


print("Train set size: {}\nValidation set size: {}\nTest set size: {}".format(len(train_image_paths), len(valid_image_paths), len(test_image_paths)))

idx_to_class = {i:j for i, j in enumerate(classes)}
class_to_idx = {value:key for key,value in idx_to_class.items()}

print(idx_to_class)
class SceneDataset(Dataset):
    def __init__(self, image_paths, transform=False):
        self.image_paths = image_paths
        self.transform = transform
        
    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        image_filepath = self.image_paths[idx]
        image =  np.array(Image.open(image_filepath))
        import torchvision.transforms 
        transform = torchvision.transforms.Compose([
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],)
        ])
        image = transform(image)
        label = image_filepath.split('/')[-2]
        label = class_to_idx[label]
        return image, label
def get_dataset(root):
    train_data_path = "/mnt/gunkaynar/project/hw3/data"
    train_image_paths = [] 
    classes = [] 

    for data_path in glob.glob(train_data_path + '/*'):
        classes.append(data_path.split('/')[-1]) 
        train_image_paths.append(glob.glob(data_path + '/*'))
    train_image_paths = np.array(train_image_paths,dtype=object)
    train_image_paths_new = []
    for lists in train_image_paths:
        for item in lists:
            train_image_paths_new.append(item)
    train_image_paths_new = np.array(train_image_paths_new)
    train_image_paths_new = (np.ndarray.flatten(train_image_paths_new))
    random.shuffle(train_image_paths_new)



    train_image_paths, valid_image_paths, test_image_paths = train_image_paths_new[:int(0.7*len(train_image_paths_new))], train_image_paths_new[int(0.7*len(train_image_paths_new)):int(0.8*len(train_image_paths_new))] , train_image_paths_new[int(0.8*len(train_image_paths_new)):]

    train_dataset = SceneDataset(train_image_paths)
    valid_dataset = SceneDataset(valid_image_paths)
    test_dataset = SceneDataset(test_image_paths)  

    return train_dataset, valid_dataset, test_dataset
root_dir = "/mnt/gunkaynar/project/hw3/data"
train_dataset, valid_dataset, test_dataset = get_dataset(root_dir)
print(len(train_dataset))
print(len(valid_dataset))
print(len(test_dataset))<jupyter_output>7005
1001
2002<jupyter_text>Model Implementation [7 pts]Now implement your CNN. ConvNet class will represent your convolutional neural network. Implement 3 layers of convolution:     (1) 4 filters with size of 3 x 3 with stride 1 and padding 1, (2) ReLU     (3) 8 filters with size of 3 x 3 with stride 1 and padding 1, (4) ReLU and (5) MaxPool 2 x 2     (6) 16 filters with size of 3 x 3 with stride 1 and padding 1, (7) ReLU and (8) MaxPool 2 x 2  As the classifier layer, you need to add only one linear layer at the end of the network. You need to choose the appropriate input and output neuron sizes and the activation function for the dense layer.<jupyter_code>class ConvNet(nn.Module):
    def __init__(self, **kwargs): 
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=4, stride = 1, padding=1, kernel_size=(3,3))
        self.conv2 = nn.Conv2d(in_channels=4, out_channels=8, stride = 1, padding=1, kernel_size=(3,3))
        self.conv3 = nn.Conv2d(in_channels=8, out_channels=16, stride = 1, padding=1, kernel_size=(3,3))
        self.maxpool = nn.MaxPool2d(2)
        self.flatten = nn.Flatten(start_dim=1, end_dim=- 1)
        self.linear = nn.Linear(7744, 4)
        self.softmax = nn.LogSoftmax(dim=1)
    def forward(self, X): 
        X = F.relu(self.conv1(X))
        X = F.relu(self.conv2(X))
        X = self.maxpool(X)    
        X = F.relu(self.conv3(X))
        X = self.maxpool(X) 
        X = self.flatten(X)
        X = self.linear(X)
        X = self.softmax(X)
        return X<jupyter_output><empty_output><jupyter_text>Stochastic Gradient Descent [25 pts.] Training with SGD [15 pts.]Train your model up to 300 epochs with properly processed inputs, i.e. call your "get_dataset" function. Use SGD as your optimizer. Tune your learning rate, weight decay. Do not add additional parameters to SGD. Save your best model as "best_cnn_sgd.pth". The best model should be selected based on validation dataset. You could use any measurement and/or metric to decide on the best model. However, you must explain your reasoning in your choice.During training, you need to plot two figures:1. training loss and validation loss vs. epoch2. training accuracy and validation accuracy vs. epoch Name your axes and plots properly. **I have used validation accuracy as the metric for selecting the best model. Because I did not want to overfit the training set by selecting the model by using training accuracy. The validation set is not used for training, it is only used for hyperparameter selection and model selection.**<jupyter_code># HINT: note that your training time should not take more than 2 hours.

max_epoch = 120
train_batch = 512
validation_batch = 512
learning_rate = 0.05 # try learning rate from the interval [1e-1, 1e-4]

gpu_id = int(input("GPU index: "))
if torch.cuda.is_available():
    device = torch.device(f"cuda:{gpu_id}")
    print(f"GPU {gpu_id} is available")
else:
    device = torch.device("cpu")
    print("GPU is not available")

train_loader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)

valid_loader = DataLoader(valid_dataset, batch_size=validation_batch, shuffle=True)

test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
model = ConvNet().cuda()
criterion = nn.NLLLoss().cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=5e-04) # you can play with weight_decay as well but do not add additional parameters
steps = 0
running_loss = 0
train_losses, validation_losses = [], []
train_acc, validation_acc = [], []
best_acc = 0
train_accuracy = 0
epoc_acc_train_s,epoc_acc_valid_s = [], []
epoc_loss_train_s,epoc_loss_valid_s = [], []
process_start = time.time()
for epoch in range(max_epoch):
    batch_idx = 1
    for inputs, labels in train_loader:
        steps += 1
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        ps = torch.exp(logps)
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        train_accuracy = torch.mean(equals.type(torch.FloatTensor)).item()        
        running_loss += loss.item()
        validation_loss = 0
        accuracy = 0
        model.eval()
        with torch.no_grad():
            for inputs, labels in valid_loader:
                inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
                logps = model.forward(inputs)
                batch_loss = criterion(logps, labels)
                validation_loss += batch_loss.item()
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
        train_losses.append(running_loss)
        train_acc.append(train_accuracy)
        validation_losses.append(validation_loss/len(valid_loader))
        validation_acc.append(accuracy/len(valid_loader))                 
        print(f"Epoch {epoch+1}/{max_epoch},{batch_idx}/14 "
              f"Train loss: {running_loss:.3f}.. "
              f"Train accuracy: {train_accuracy:.3f}.. "
              f"Validation loss: {validation_loss/len(valid_loader):.3f}.. "
              f"Validation accuracy: {accuracy/len(valid_loader):.3f}")
        if accuracy/len(valid_loader) > best_acc:
            torch.save(model, '/mnt/gunkaynar/project/hw3/best_cnn_sgd.pth')
            best_acc = accuracy/len(valid_loader)
        running_loss = 0
        model.train()
        batch_idx +=1
    epoc_acc_train = np.mean(train_acc)
    epoc_acc_valid = np.mean(validation_acc) 
    epoc_loss_train = np.mean(train_losses)
    epoc_loss_valid = np.mean(validation_losses) 
    epoc_acc_train_s.append(epoc_acc_train)
    epoc_acc_valid_s.append(epoc_acc_valid)
    epoc_loss_train_s.append(epoc_loss_train)
    epoc_loss_valid_s.append(epoc_loss_valid)
process_end = time.time()
print(f"All training process took {process_end - process_start} secs.")
print("Best Acc:", best_acc)


x = [(i+1) for i in range(len(epoc_acc_train_s))]
plt.plot(x, epoc_acc_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_acc_valid_s, 'ro', label='Validation', color="red")
plt.title('Accuracies vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Accuracy', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()
plt.clf()
x = [(i+1) for i in range(len(epoc_loss_train_s))]
plt.plot(x, epoc_loss_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_loss_valid_s, 'ro', label='Validation', color="red")
plt.title('Losses vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Loss', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()<jupyter_output>GPU index: 0
GPU 0 is available
Epoch 1/120,1/14 Train loss: 1.392.. Train accuracy: 0.168.. Validation loss: 1.374.. Validation accuracy: 0.306
Epoch 1/120,2/14 Train loss: 1.376.. Train accuracy: 0.336.. Validation loss: 1.364.. Validation accuracy: 0.334
Epoch 1/120,3/14 Train loss: 1.366.. Train accuracy: 0.340.. Validation loss: 1.331.. Validation accuracy: 0.489
Epoch 1/120,4/14 Train loss: 1.341.. Train accuracy: 0.451.. Validation loss: 1.293.. Validation accuracy: 0.512
Epoch 1/120,5/14 Train loss: 1.306.. Train accuracy: 0.473.. Validation loss: 1.232.. Validation accuracy: 0.537
Epoch 1/120,6/14 Train loss: 1.239.. Train accuracy: 0.514.. Validation loss: 1.106.. Validation accuracy: 0.605
Epoch 1/120,7/14 Train loss: 1.142.. Train accuracy: 0.559.. Validation loss: 0.991.. Validation accuracy: 0.611
Epoch 1/120,8/14 Train loss: 1.041.. Train accuracy: 0.553.. Validation loss: 0.978.. Validation accuracy: 0.566
Epoch 1/120,9/14 Train loss: 0.999.. Train accuracy: 0.549.. Val[...]<jupyter_text>Test with SGD [10 pts.]Report the following for your best model on your test set which has not been seen by the model yet.1. A heatmap for confusion matrix2. Accuracy3. Macro Precision4. Macro Recall5. F1 Score<jupyter_code>best_path = '/mnt/gunkaynar/project/hw3/best_cnn_sgd.pth'
model = torch.load(best_path)
test_loss = 0
nb_classes = 4
test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
device = torch.device("cuda:0")
criterion = nn.NLLLoss().cuda()

model = model.eval()
with torch.no_grad():
    for inputs, labels in test_loader:
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        outputs = model(inputs)
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        test_loss += loss.item()
        ps = torch.exp(logps)
        _, preds = torch.max(outputs, 1)
        acc = accuracy_score(labels.cpu(), preds.cpu())
        f1 = f1_score(labels.cpu(), preds.cpu(),average='macro')
        pre = precision_score(labels.cpu(), preds.cpu(),average='macro')
        recall = recall_score(labels.cpu(), preds.cpu(),average='macro')
        cm =  confusion_matrix(labels.cpu(), preds.cpu())


losses = test_loss

print("Mean Loss:", losses, "\nMean Acc:", acc,"\nMean Macro Precision:", pre, "\nMean Macro Recall:", recall, "\nMean Macro F1 Score:", f1) 

import seaborn as sns
ax = sns.heatmap(cm,annot=True,fmt="",cmap="Blues",cbar=False,xticklabels=["forest","mountain","glacier","sea"],yticklabels=["forest","mountain","glacier","sea"])
ax.set_ylabel("Ground Truth")
ax.set_xlabel("Predicted")
plt.title("Heatmap for Confusion Matrix")
plt.show()<jupyter_output>Mean Loss: 0.6051817536354065 
Mean Acc: 0.7892107892107892 
Mean Macro Precision: 0.7912020746305897 
Mean Macro Recall: 0.7900233891785484 
Mean Macro F1 Score: 0.7892097936836774<jupyter_text>Adam Optimizer [25 pts.]Adam is an adaptive learning rate optimization algorithm that has been designed specifically for training deep neural networks. It was presented by <NAME> from OpenAI and <NAME> from the University of Toronto in their 2015 ICLR paper (poster) titled “Adam: A Method for Stochastic Optimization“.Nowadays, most of machine learning frameworks, including tensorflow, Pytorch, and Keras, choose Adam as the default optimizer. In this question, you will experiment with it and try to understand why it replaced SGD as the default optimizer. Training with ADAM [15 pts.]Train your model up to 300 epochs with properly processed inputs, i.e. call your "get_dataset". This time use Adam Optimizer as your optimizer. Tune your learning rate, weight decay. Save your best model as "best_cnn_adam.pth". The best model should be selected based on validation dataset. You could use any measurement and/or metric to decide on the best model for each network. However, you must explain your reasoning in your choice.During training, you need to plot:1. training loss and validation loss vs. epoch2. training accuracy and validation accuracy vs. epoch Name your axes and plots properly. **I have used validation accuracy as the metric for selecting the best model. Because I did not want to overfit the training set by selecting the model by using training accuracy.The validation set is not used for training, it is only used for hyperparameter selection and model selection.**<jupyter_code># HINT: note that your training time should not take more than 2 hours.

max_epoch = 120
train_batch = 2056
validation_batch = 256
learning_rate = 0.001 # try learning rate from the interval [1e-1, 1e-4]

gpu_id = int(input("GPU index: "))
if torch.cuda.is_available():
    device = torch.device(f"cuda:{gpu_id}")
    print(f"GPU {gpu_id} is available")
else:
    device = torch.device("cpu")
    print("GPU is not available")

train_loader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)

valid_loader = DataLoader(valid_dataset, batch_size=validation_batch, shuffle=True)

test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
model = ConvNet().cuda()
criterion = nn.NLLLoss().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=5e-04) # you can play with  weight_decay as well
steps = 0
running_loss = 0
train_losses, validation_losses = [], []
train_acc, validation_acc = [], []
best_acc = 0
train_accuracy = 0
epoc_acc_train_s,epoc_acc_valid_s = [], []
epoc_loss_train_s,epoc_loss_valid_s = [], []
process_start = time.time()
for epoch in range(max_epoch):
    batch_idx = 1
    for inputs, labels in train_loader:
        steps += 1
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        ps = torch.exp(logps)
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        train_accuracy = torch.mean(equals.type(torch.FloatTensor)).item()        
        running_loss += loss.item()
        validation_loss = 0
        accuracy = 0
        model.eval()
        with torch.no_grad():
            for inputs, labels in valid_loader:
                inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
                logps = model.forward(inputs)
                batch_loss = criterion(logps, labels)
                validation_loss += batch_loss.item()
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
        train_losses.append(running_loss)
        train_acc.append(train_accuracy)
        validation_losses.append(validation_loss/len(valid_loader))
        validation_acc.append(accuracy/len(valid_loader))                 
        print(f"Epoch {epoch+1}/{max_epoch},{batch_idx}/4 "
              f"Train loss: {running_loss:.3f}.. "
              f"Train accuracy: {train_accuracy:.3f}.. "
              f"Validation loss: {validation_loss/len(valid_loader):.3f}.. "
              f"Validation accuracy: {accuracy/len(valid_loader):.3f}")
        if accuracy/len(valid_loader) > best_acc:
            torch.save(model, '/mnt/gunkaynar/project/hw3/best_cnn_adam.pth')
            best_acc = accuracy/len(valid_loader)
        running_loss = 0
        model.train()
        batch_idx +=1
    epoc_acc_train = np.mean(train_acc)
    epoc_acc_valid = np.mean(validation_acc) 
    epoc_loss_train = np.mean(train_losses)
    epoc_loss_valid = np.mean(validation_losses) 
    epoc_acc_train_s.append(epoc_acc_train)
    epoc_acc_valid_s.append(epoc_acc_valid)
    epoc_loss_train_s.append(epoc_loss_train)
    epoc_loss_valid_s.append(epoc_loss_valid)
process_end = time.time()
print(f"All training process took {process_end - process_start} secs.")
print("Best Acc:", best_acc)


x = [(i+1) for i in range(len(epoc_acc_train_s))]
plt.plot(x, epoc_acc_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_acc_valid_s, 'ro', label='Validation', color="red")
plt.title('Accuracies vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Accuracy', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()
plt.clf()
x = [(i+1) for i in range(len(epoc_loss_train_s))]
plt.plot(x, epoc_loss_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_loss_valid_s, 'ro', label='Validation', color="red")
plt.title('Losses vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Loss', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()<jupyter_output>GPU index: 0
GPU 0 is available
Epoch 1/120,1/4 Train loss: 1.381.. Train accuracy: 0.244.. Validation loss: 1.327.. Validation accuracy: 0.344
Epoch 1/120,2/4 Train loss: 1.338.. Train accuracy: 0.318.. Validation loss: 1.248.. Validation accuracy: 0.481
Epoch 1/120,3/4 Train loss: 1.258.. Train accuracy: 0.454.. Validation loss: 1.181.. Validation accuracy: 0.570
Epoch 1/120,4/4 Train loss: 1.209.. Train accuracy: 0.534.. Validation loss: 1.121.. Validation accuracy: 0.585
Epoch 2/120,1/4 Train loss: 1.157.. Train accuracy: 0.544.. Validation loss: 1.059.. Validation accuracy: 0.588
Epoch 2/120,2/4 Train loss: 1.095.. Train accuracy: 0.552.. Validation loss: 1.000.. Validation accuracy: 0.608
Epoch 2/120,3/4 Train loss: 1.040.. Train accuracy: 0.557.. Validation loss: 0.963.. Validation accuracy: 0.604
Epoch 2/120,4/4 Train loss: 0.961.. Train accuracy: 0.590.. Validation loss: 0.934.. Validation accuracy: 0.610
Epoch 3/120,1/4 Train loss: 0.997.. Train accuracy: 0.571.. Validation l[...]<jupyter_text>Test with ADAM [10 pts.]Report the following for your best model on your test set which has not been seen by the model yet.1. A heatmap for confusion matrix2. Accuracy3. Macro Precision4. Macro Recall5. F1 ScoreThen, discuss figures that you have plotted in the previous section, your test results and algorithm complexity with maximum 200 words. Compare two **optimizers**. Which one is more preferable? Why?<jupyter_code>best_path = '/mnt/gunkaynar/project/hw3/best_cnn_adam.pth'
model = torch.load(best_path)
test_loss = 0
nb_classes = 4
test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
device = torch.device("cuda:0")
criterion = nn.NLLLoss().cuda()

model = model.eval()
with torch.no_grad():
    for inputs, labels in test_loader:
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        outputs = model(inputs)
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        test_loss += loss.item()
        ps = torch.exp(logps)
        _, preds = torch.max(outputs, 1)
        acc = accuracy_score(labels.cpu(), preds.cpu())
        f1 = f1_score(labels.cpu(), preds.cpu(),average='macro')
        pre = precision_score(labels.cpu(), preds.cpu(),average='macro')
        recall = recall_score(labels.cpu(), preds.cpu(),average='macro')
        cm =  confusion_matrix(labels.cpu(), preds.cpu())


losses = test_loss

print("Mean Loss:", losses, "\nMean Acc:", acc,"\nMean Macro Precision:", pre, "\nMean Macro Recall:", recall, "\nMean Macro F1 Score:", f1) 

import seaborn as sns
ax = sns.heatmap(cm,annot=True,fmt="",cmap="Blues",cbar=False,xticklabels=["forest","mountain","glacier","sea"],yticklabels=["forest","mountain","glacier","sea"])
ax.set_ylabel("Ground Truth")
ax.set_xlabel("Predicted")
plt.title("Heatmap for Confusion Matrix")
plt.show()<jupyter_output>Mean Loss: 0.6756293177604675 
Mean Acc: 0.7757242757242757 
Mean Macro Precision: 0.7801834713247356 
Mean Macro Recall: 0.7757747389519813 
Mean Macro F1 Score: 0.7768020205675421<jupyter_text>**Then, discuss figures that you have plotted in the previous section, your test results and algorithm complexity with maximum 200 words.****In SGD, it can be seen from the accuracies plot that the training accuracy increased significantly leaving the validation accuracy lag behing. Likewise, the loss of training dropped dramatically but there has been a big gap between that and validation loss. The training accuracy increased to nearly 100%, whereas the validation accuracy cannot make it even to 80%. This is probably due to the high learning rate and the optimizer used. If I trained this model even more, it would probably overfit. In testing we saw that the model performed with 79% accuracy.****In ADAM optimizer, the validation and training accuracies were not that distinct. They increased together and there was no huge gap between them. As can be seen likewise from the loss plot that the model did not perform badly on  validation. Which is the consequence of decreased variance in our model thanks to ADAM optimizer. However, in the test set the ADAM optimized model performed with 78% accuracy which is 1% lower than SGD optimized model. This is probably due to the shuffle statement in our dataset loaders. I would prefer ADAM optimizer because of its convenience in training time, batch size/learning rate and that it reduces variance.****Compare two optimizers. Which one is more preferable? Why?****ADAM generalizes better than SGD optimizer. However, there are some difficulties one can encounter while using ADAM. That is, ADAM is very sensitive to its hyperparameters and requires fine tuning of the learning rate. It usually requires smaller learning rates than SGD. With ADAM optimizer, my training took less time than that of with SGD, but this is mostly because of the batch size. In my opinion, ADAM is more preferable when using bigger batch sizes and when there is time for hyperparameter tuning. But SGD generalizes better when batch size is smaller.****Additionally, ADAM is less prone to overfitting the training set, because as can be seen from the training-validation accuracy plots of both optimizers, with ADAM the accuracies change similarly whereas in SGD training accuracy surpasses the validation accuracy very significantly. I would prefer ADAM optimizer when the baseline accuracy in SGD is not enough for the model and I have time for cross validation and hyperparameter tuning. It means several training processes and/or using grid search.** Transfer Learning [25 pts.]Instead of training CNNs from scratch, you can use pretrained models and apply them to your task. Transfer learning is a machine learning technique where you can reuse a pretrained machine learning model as a starting point for your own task. In this question, you will experiment with it and try to understand why it is used. Training with Transfer Learning [15 pts.]Get pretrained ResNet18 model from torchvision.models and finetune your model up to 20 epochs with properly processed inputs, i.e. call your "get_dataset". This time use transfer learning. Tune your learning rate, weight decay. Save your best model as "best_cnn_transfer.pth". The best model should be selected based on validation dataset. You could use any measurement and/or metric to decide on the best model for each network. However, you must explain your reasoning in your choice.During training, you need to plot two figures:1. training loss and validation loss vs. epoch2. training accuracy and validation accuracy vs. epoch Name your axes and plots properly. **I have used validation accuracy as the metric for selecting the best model. Because I did not want to overfit the training set by selecting the model by using training accuracy.The validation set is not used for training, it is only used for hyperparameter selection and model selection.**<jupyter_code># HINT: note that your training time should not take more than 2 hours.

max_epoch = 20
train_batch = 1024
validation_batch = 64
learning_rate = 10**-4 # try learning rate from the interval [1e-1, 1e-4]

gpu_id = int(input("GPU index: "))
if torch.cuda.is_available():
    device = torch.device(f"cuda:{gpu_id}")
    print(f"GPU {gpu_id} is available")
else:
    device = torch.device("cpu")
    print("GPU is not available")

train_loader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)

valid_loader = DataLoader(valid_dataset, batch_size=validation_batch, shuffle=True)

test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
model = models.resnet18(pretrained=True).cuda()
num_features = model.fc.in_features
model.fc = nn.Sequential(nn.Linear(512, 4),nn.LogSoftmax(dim=1)).cuda()
criterion = nn.NLLLoss().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=5e-04) # you can play with  weight_decay as well
steps = 0
running_loss = 0
train_losses, validation_losses = [], []
train_acc, validation_acc = [], []
best_acc = 0
train_accuracy = 0
epoc_acc_train_s,epoc_acc_valid_s = [], []
epoc_loss_train_s,epoc_loss_valid_s = [], []
process_start = time.time()
for epoch in range(max_epoch):
    batch_idx = 1
    for inputs, labels in train_loader:
        steps += 1
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        ps = torch.exp(logps)
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        train_accuracy = torch.mean(equals.type(torch.FloatTensor)).item()        
        running_loss += loss.item()
        validation_loss = 0
        accuracy = 0
        model.eval()
        with torch.no_grad():
            for inputs, labels in valid_loader:
                inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
                logps = model.forward(inputs)
                batch_loss = criterion(logps, labels)
                validation_loss += batch_loss.item()
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
        train_losses.append(running_loss)
        train_acc.append(train_accuracy)
        validation_losses.append(validation_loss/len(valid_loader))
        validation_acc.append(accuracy/len(valid_loader))                 
        print(f"Epoch {epoch+1}/{max_epoch},{batch_idx}/7 "
              f"Train loss: {running_loss:.3f}.. "
              f"Train accuracy: {train_accuracy:.3f}.. "
              f"Validation loss: {validation_loss/len(valid_loader):.3f}.. "
              f"Validation accuracy: {accuracy/len(valid_loader):.3f}")
        if accuracy/len(valid_loader) > best_acc:
            torch.save(model, '/mnt/gunkaynar/project/hw3/best_cnn_transfer.pth')
            best_acc = accuracy/len(valid_loader)
        running_loss = 0
        model.train()
        batch_idx +=1
    epoc_acc_train = np.mean(train_acc)
    epoc_acc_valid = np.mean(validation_acc) 
    epoc_loss_train = np.mean(train_losses)
    epoc_loss_valid = np.mean(validation_losses) 
    epoc_acc_train_s.append(epoc_acc_train)
    epoc_acc_valid_s.append(epoc_acc_valid)
    epoc_loss_train_s.append(epoc_loss_train)
    epoc_loss_valid_s.append(epoc_loss_valid)
process_end = time.time()
print(f"All training process took {process_end - process_start} secs.")
print("Best Acc:", best_acc)


x = [(i+1) for i in range(len(epoc_acc_train_s))]
plt.plot(x, epoc_acc_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_acc_valid_s, 'ro', label='Validation', color="red")
plt.title('Accuracies vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Accuracy', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()
plt.clf()
x = [(i+1) for i in range(len(epoc_loss_train_s))]
plt.plot(x, epoc_loss_train_s, 'ro', label='Train', color="blue")
plt.plot(x, epoc_loss_valid_s, 'ro', label='Validation', color="red")
plt.title('Losses vs epoch')
plt.xlabel('Iteration', color='#1C2833')
plt.ylabel('Loss', color='#1C2833')
plt.legend(loc='upper left')
plt.grid()
plt.show()<jupyter_output>GPU index: 0
GPU 0 is available
Epoch 1/20,1/7 Train loss: 1.519.. Train accuracy: 0.238.. Validation loss: 1.235.. Validation accuracy: 0.332
Epoch 1/20,2/7 Train loss: 1.170.. Train accuracy: 0.486.. Validation loss: 0.998.. Validation accuracy: 0.532
Epoch 1/20,3/7 Train loss: 0.961.. Train accuracy: 0.636.. Validation loss: 0.833.. Validation accuracy: 0.680
Epoch 1/20,4/7 Train loss: 0.758.. Train accuracy: 0.743.. Validation loss: 0.706.. Validation accuracy: 0.744
Epoch 1/20,5/7 Train loss: 0.629.. Train accuracy: 0.794.. Validation loss: 0.611.. Validation accuracy: 0.783
Epoch 1/20,6/7 Train loss: 0.524.. Train accuracy: 0.827.. Validation loss: 0.523.. Validation accuracy: 0.810
Epoch 1/20,7/7 Train loss: 0.464.. Train accuracy: 0.862.. Validation loss: 0.464.. Validation accuracy: 0.833
Epoch 2/20,1/7 Train loss: 0.321.. Train accuracy: 0.902.. Validation loss: 0.424.. Validation accuracy: 0.846
Epoch 2/20,2/7 Train loss: 0.359.. Train accuracy: 0.876.. Validation loss: 0.40[...]<jupyter_text>Test for Transfer Learning [10 pts.]Report the following for your best model on your test set which has not been seen by the model yet.1. A heatmap for confusion matrix2. Accuracy3. Macro Precision4. Macro Recall5. F1 ScoreThen, discuss figures that you have plotted in the previous section, your test results and algorithm complexity with maximum 200 words. Explain the advantages of using transfer learning. Is it better to reuse a pretrained model instead of training a model from scratch? Why?<jupyter_code>best_path = '/mnt/gunkaynar/project/hw3/best_cnn_transfer.pth'
model = torch.load(best_path)
test_loss = 0
nb_classes = 4
test_loader = DataLoader(test_dataset, batch_size=2002, shuffle=False)
device = torch.device("cuda:0")
criterion = nn.NLLLoss().cuda()

model = model.eval()
with torch.no_grad():
    for inputs, labels in test_loader:
        inputs, labels = inputs.to(device=device,dtype=torch.float),labels.to(device=device).to(torch.long)
        outputs = model(inputs)
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        test_loss += loss.item()
        ps = torch.exp(logps)
        _, preds = torch.max(outputs, 1)
        acc = accuracy_score(labels.cpu(), preds.cpu())
        f1 = f1_score(labels.cpu(), preds.cpu(),average='macro')
        pre = precision_score(labels.cpu(), preds.cpu(),average='macro')
        recall = recall_score(labels.cpu(), preds.cpu(),average='macro')
        cm =  confusion_matrix(labels.cpu(), preds.cpu())


losses = test_loss

print("Mean Loss:", losses, "\nMean Acc:", acc,"\nMean Macro Precision:", pre, "\nMean Macro Recall:", recall, "\nMean Macro F1 Score:", f1) 

import seaborn as sns
ax = sns.heatmap(cm,annot=True,fmt="",cmap="Blues",cbar=False,xticklabels=["forest","mountain","glacier","sea"],yticklabels=["forest","mountain","glacier","sea"])
ax.set_ylabel("Ground Truth")
ax.set_xlabel("Predicted")
plt.title("Heatmap for Confusion Matrix")
plt.show()<jupyter_output>Mean Loss: 0.3316091299057007 
Mean Acc: 0.9040959040959041 
Mean Macro Precision: 0.905315793829547 
Mean Macro Recall: 0.9038820415661041 
Mean Macro F1 Score: 0.9040691133120188
{'fraction_non_alphanumeric': 0.10130022256061848, 'fraction_numerical': 0.033126391003865524, 'mean_word_length': 4.090150250417362, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 53, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '311168', 'license': 'MIT', 'n_tokens_mistral': 14013, 'n_tokens_neox': 12850, 'n_words': 4319, 'path': 'cs464_fall21_hw3.ipynb', 'repo_name': 'gunkaynar/cs464_hw3', 'size': '42709'}