<jupyter_start><jupyter_text>CSCI-GA.3033-​076 Vision Meets Machine Learning Assignment-2In this assignment, you will train the Resnet18 classifier on the CIFAR-10 dataset and do the statistical analysis of the classification.  Submission InfoYou should make a copy of this Colab (File->Save a copy in Drive). Please start the assignment early and don’t be afraid to ask for help from either the TAs or myself. You are allowed to collaborate with other students in terms discussing ideas and possible solutions. However you code up the solution yourself, i.e. you must write your own code. Copying your friends code and just changing all the names of the variables is not allowed! You are not allowed to use solutions from similar assignments in courses from other institutions, or those found elsewhere on the web. Your solutions should be submitted via the NYU classes system.  **Part 1- Training The Resnet classifier (40 Points)**<jupyter_code>import torch
import torchvision
import torchvision.transforms as transforms
from torchvision import datasets, models, transforms
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt
import numpy as np
import torch.optim as optim
# The CIFAR-10 dataset is available in the pytorch module.
#  http://www.cs.toronto.edu/~kriz/cifar.html
# Only normalization tranformation is applied. You can choose to apply data augmentation techniques as well.

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

#Loading the training data
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

#Loading the testing data
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')


# functions to show an image
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()<jupyter_output><empty_output><jupyter_text>**Question - 1. Show any 4 random images from the dataset.(10 Points)**<jupyter_code>#image = trainset(root = './data')
image = testloader
batch_x, batch_y = next(iter(testloader))
image = batch_x[0][0].reshape(32,32)
#image = image.fromarray(image)
next(iter(testloader))

#print(next(iter(testloader)))
for i in range(4):
  imshow(next(iter(testloader))[0][i])
# Using the GPU available on the google colab
if torch.cuda.device_count() > 0:
    current_device = 'cuda'
else:
    current_device = 'cpu'
print('Current Device: ' + current_device)
# Intiating the Resnet18 model from the torch module. Setting pretrained = False because we're training it from the scratch
net = models.resnet18(pretrained= False)
num_classes = 10<jupyter_output><empty_output><jupyter_text>**Question-2. The Resnet18 model intialized here contains 1000 output classes change it to 10 classes in the model.(10 Points)**<jupyter_code>torchvision.models.resnet18(pretrained = False).fc.num_classes = 10

#torchvision.models.resnet18(pretrained=False, progress=True, **kwargs)
##https://discuss.pytorch.org/t/changing-in-features-in-fc-layer-for-resnet18-in-torchvision/21445/2
#resnet18 = torchvision.models.resnet18(pretrained = False)
#resnet18.fc.out_features = 10
#torchvision.models.resnet18(pretrained = False).fc.out_features = 10
#Model training

net = net.to(current_device)


criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

num_epochs = 20

for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        
        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')<jupyter_output><empty_output><jupyter_text>**Question-3. (i).Plot the loss with epoch for lr = [10, 1, 0.1, 0.01, 0.001] and make comments about how high lr effects the training. (20 Points)** We train the model using different lr. And we repeat the program four times.<jupyter_code>#plot with lr=0.001

net = net.to(current_device)

criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

num_epochs = 20
y_1=[]
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #data is tensor format
        data_temp = data

        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        
        rs=running_loss/len(trainloader)

    y_1.append(rs)
    print("The loss {:.4f} of {:.1f}th epoch".format(rs,epoch))

print('Finished Training') 


print('y_1',y_1)
x_1=[i+1 for i in range(num_epochs)]
plt.plot(x_1, y_1, color = 'blue')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.title('Loss with epochs: lr={:.3f}'.format(0.001))
plt.show()

#plot with lr=0.01

net = net.to(current_device)

criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

num_epochs = 20
y_1=[]
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #data is tensor format
        data_temp = data

        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        
        rs=running_loss/len(trainloader)

    y_1.append(rs)
    print("The loss {:.4f} of {:.1f}th epoch".format(rs,epoch))

print('Finished Training') 


print('y_1',y_1)
x_1=[i+1 for i in range(num_epochs)]
plt.plot(x_1, y_1, color = 'blue')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.title('Loss with epochs: lr={:.3f}'.format(0.01))
plt.show()

#plot with lr=0.1

net = net.to(current_device)

criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9)

num_epochs = 20
y_1=[]
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #data is tensor format
        data_temp = data

        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        
        rs=running_loss/len(trainloader)

    y_1.append(rs)
    print("The loss {:.4f} of {:.1f}th epoch".format(rs,epoch))

print('Finished Training') 


print('y_1',y_1)
x_1=[i+1 for i in range(num_epochs)]
plt.plot(x_1, y_1, color = 'blue')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.title('Loss with epochs: lr={:.3f}'.format(10))
plt.show()

#plot with lr=1

net = net.to(current_device)

criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=1, momentum=0.9)

num_epochs = 20
y_1=[]
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #data is tensor format
        data_temp = data

        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        
        rs=running_loss/len(trainloader)

    y_1.append(rs)
    print("The loss {:.4f} of {:.1f}th epoch".format(rs,epoch))

print('Finished Training') 


print('y_1',y_1)
x_1=[i+1 for i in range(num_epochs)]
plt.plot(x_1, y_1, color = 'blue')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.title('Loss with epochs: lr={:.3f}'.format(1))
plt.show()

#plot with lr=10

net = net.to(current_device)

criterion = nn.CrossEntropyLoss()
#You can expermient with different lr and momentum
optimizer = optim.SGD(net.parameters(), lr=10, momentum=0.9)

num_epochs = 20
y_1=[]
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #data is tensor format
        data_temp = data

        inputs = inputs.to(current_device)
        labels = labels.to(current_device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        
        rs=running_loss/len(trainloader)

    y_1.append(rs)
    print("The loss {:.4f} of {:.1f}th epoch".format(rs,epoch))

print('Finished Training') 


print('y_1',y_1)
x_1=[i+1 for i in range(num_epochs)]
plt.plot(x_1, y_1, color = 'blue')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.title('Loss with epochs: lr={:.3f}'.format(10))
plt.show()<jupyter_output>The loss nan of 0.0th epoch
The loss nan of 1.0th epoch
The loss nan of 2.0th epoch
The loss nan of 3.0th epoch
The loss nan of 4.0th epoch
The loss nan of 5.0th epoch
The loss nan of 6.0th epoch
The loss nan of 7.0th epoch
The loss nan of 8.0th epoch
The loss nan of 9.0th epoch
The loss nan of 10.0th epoch
The loss nan of 11.0th epoch
The loss nan of 12.0th epoch
The loss nan of 13.0th epoch
The loss nan of 14.0th epoch
The loss nan of 15.0th epoch
The loss nan of 16.0th epoch
The loss nan of 17.0th epoch
The loss nan of 18.0th epoch
The loss nan of 19.0th epoch
Finished Training
y_1 [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]<jupyter_text>Lr has a great influence on loss function. When Lr = 0.01, loss decreased steadily with epoch. When Lr = 0.1, loss will increase in the middle of the process and then decrease steadily. If Lr is large, such as Lr = 10 and  Lr = 1, the convergence solution can not even be obtained.<jupyter_code>#Testing the model
correct = 0
total = 0
i=0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images.to(current_device))
        labels = labels.to(current_device)
        _, predicted = torch.max(outputs.data, 1)
        print(predicted,i+1)
        total += labels.size(0)
        correct += (predicted != labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))<jupyter_output><empty_output><jupyter_text>**Part - 2 (20 Points)** Question-1. Calculate the confusion matrix of the classifcation. (10 Points) Question-2. Calculate the Precision, Recall and F1-score for each class. (10 Points)(Hint - You can find this article useful - https://towardsdatascience.com/confusion-matrix-for-your-multi-class-machine-learning-model-ff9aa3bf7826)<jupyter_code>#Compute the confusion matrix of the classification and the Precision, Recall and F1-score for each class
nb_classes = 10
confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
    for i, (inputs, classes) in enumerate(testloader):
        inputs = inputs.to(current_device)
        classes = classes.to(current_device)
        outputs = net(inputs.to(current_device))
        _, preds = torch.max(outputs, 1)
        for t, p in zip(classes.view(-1), preds.view(-1)):
                confusion_matrix[t.long(), p.long()] += 1

print("The confusion maatrix:",confusion_matrix)


# To get the per-class recall:
recal=confusion_matrix.diag()/confusion_matrix.sum(0)
print("The per-call recall:",recal)

# To get the per-class accuracy:
accura=confusion_matrix.diag()/confusion_matrix.sum(1)
print("The per-class accuracy:", accura)

#To ger the F1-score
F1_score=2*recal* accura/(recal+accura)
print("The F1-scor:", F1_score)<jupyter_output>The confusion maatrix: tensor([[634.,  22.,  13.,   6.,  49.,   5.,  25.,  31., 167.,  48.],
        [ 57., 397.,   0.,   5.,  15.,   2.,  90.,  16., 154., 264.],
        [207.,  18., 120.,  56., 279.,  57., 123.,  89.,  27.,  24.],
        [ 96.,   8.,  24., 156., 136., 108., 297.,  93.,  43.,  39.],
        [104.,   5.,  36.,  33., 474.,  28., 150., 133.,  23.,  14.],
        [ 74.,   9.,  42., 158., 123., 210., 152., 177.,  35.,  20.],
        [ 36.,  10.,  11.,  36., 202.,  15., 595.,  34.,  21.,  40.],
        [ 69.,   5.,  14.,  27., 127.,  45.,  76., 577.,  11.,  49.],
        [191.,  38.,   4.,  12.,  28.,   6.,  29.,  17., 627.,  48.],
        [ 65.,  87.,   3.,  11.,  19.,  11.,  98.,  39., 189., 478.]])
The per-call recall: tensor([0.4136, 0.6628, 0.4494, 0.3120, 0.3264, 0.4312, 0.3639, 0.4784, 0.4834,
        0.4668])
The per-class accuracy: tensor([0.6340, 0.3970, 0.1200, 0.1560, 0.4740, 0.2100, 0.5950, 0.5770, 0.6270,
        0.4780])
The F1-scor: tensor([0.5006, 0.4966, [...]
{'fraction_non_alphanumeric': 0.12314948251015329, 'fraction_numerical': 0.04618105594130748, 'mean_word_length': 3.0506765720350226, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '390089', 'license': 'MIT', 'n_tokens_mistral': 5505, 'n_tokens_neox': 4982, 'n_words': 1524, 'path': 'Vision Meets ML/Vision_HW2 .ipynb', 'repo_name': 'mrwrongwrongwrong/Assignments', 'size': '15266'}