<jupyter_start><jupyter_text>Workshop 3 - Pytorch Model CreationDeepNeuron summer training 2020.Create a model using Pytorch which acts as a classifier for the CIFAR-10 dataset**Before starting:**1. **Don't edit this file, make a copy first:**  * Click on File -> Save a copy in Drive2. Also do the following:  * Click on Runtime -> Change runtime type -> Make sure hardware accelerator is set to GPU  ImportsDo all your imports here<jupyter_code>import torch
from torch import nn
from torch import optim
from torchvision import datasets, transforms

from tqdm.notebook import tqdm<jupyter_output><empty_output><jupyter_text>Model CreationCreate your model here. The model stub has already been created, you will need to define an `__init__` and a `forward()` method for your class.Your model will need:1. Convolutional Layer (3 input channels, 7 output channels, kernel size = 5)2. Max Pool (kernel size = 2, stride = 2)3. Convolutional Layer (7 input channels, 14 output channels, kernel size = 5)4. Fully Connected Layer (Figure out the input size, output size is 64)5. Fully Connected Layer (input size is 64, output size is the number of classes)Then, in your forward pass, the input should flow like so:1. Convolutional Layer 12. ReLU3. Max Pool4. Convolutional Layer 25. ReLU6. Max Pool (use the same max pool layer)7. Flatten the input, so it can be passed through the fully connected layers8. Fully Connected Layer 19. ReLU10. Fully Connected Layer 2 HintsAll layers are found in nn:* Fully Connected Layer: `Linear(num inputs, num outputs)`* Max Pooling: `MaxPool2d(kernel size, stride)`* Convolutional Layer: `Conv2d(input channels, output channels, kernel size, stride)`* Define your layers in the `__init__` method of your model* Flatten a PyTorch tensor using `.flatten()`The ReLU function can be found in `nn.functional.relu()`<jupyter_code>class MyCNN(nn.Module):
  def __init__(self, output_size):
      pass

  def forward(self, x):
    pass<jupyter_output><empty_output><jupyter_text>Training Using Your ModelIt's time to train your model!We use a basic PyTorch training loop, with standard built-in datasets, dataloaders and training loops<jupyter_code># Function for the training 

def train(model, train_loader, loss_fn, optimizer, device):
    model.train() # puts the model in training mode
    running_loss = 0
    with tqdm(total=len(train_loader)) as pbar:
        for i, data in enumerate(train_loader, 0): # loops through training data
            inputs, labels = data # separate inputs and labels (outputs)
            inputs, labels = inputs.to(device), labels.to(device) # puts the data on the GPU

            # forward + backward + optimize                                          
            optimizer.zero_grad() # clear the gradients in model parameters
            outputs = model(inputs) # forward pass and get predictions
            loss = loss_fn(outputs, labels) # calculate loss
            loss.backward() # calculates gradient w.r.t to loss for all parameters in model that have requires_grad=True
            optimizer.step() # iterate over all parameters in the model with requires_grad=True and update their weights.

            running_loss += loss.item() # sum total loss in current epoch for print later

            pbar.update(1) #increment our progress bar

    return running_loss/len(train_loader) # returns the total training loss for the epoch
# Function for the validation pass

def validation(model, val_loader, loss_fn, device):
    model.eval() # puts the model in validation mode
    running_loss = 0
    total = 0
    correct = 0
    
    with torch.no_grad(): # save memory by not saving gradients which we don't need 
        with tqdm(total=len(val_loader)) as pbar:
            for images, labels in iter(val_loader):
                images, labels = images.to(device), labels.to(device) # put the data on the GPU
                outputs = model(images) # passes image to the model, and gets a ouput which is the class probability prediction

                val_loss = loss_fn(outputs, labels) # calculates val_loss from model predictions and true labels
                running_loss += val_loss.item()
                _, predicted = torch.max(outputs, 1) # turns class probability predictions to class labels
                total += labels.size(0) # sums the number of predictions
                correct += (predicted == labels).sum().item() # sums the number of correct predictions
        
                pbar.update(1)

        return running_loss/len(val_loader), correct/total # return loss value, accuracy<jupyter_output><empty_output><jupyter_text>DatasetSet a path for the dataset downloads<jupyter_code>train_path = 'data/train'
valid_path = 'data/valid'
# Define transforms for the training and validation set
training_transforms = transforms.Compose([transforms.RandomRotation(30),
                                          transforms.RandomHorizontalFlip(),
                                          transforms.ToTensor(),
                                          transforms.Normalize([0.485, 0.456, 0.406], 
                                                               [0.229, 0.224, 0.225])])

validation_transforms = transforms.Compose([transforms.ToTensor(),
                                            transforms.Normalize([0.485, 0.456, 0.406], 
                                                                 [0.229, 0.224, 0.225])])
training_dataset = datasets.CIFAR10(train_path, train=True, transform=training_transforms, download=True)
validation_dataset = datasets.CIFAR10(valid_path, train=False, transform=validation_transforms, download=True)
training_loader = torch.utils.data.DataLoader(training_dataset, batch_size=32, shuffle=True)
validation_loader = torch.utils.data.DataLoader(validation_dataset, batch_size=32, shuffle=False)
# Check what classes are in our dataset

training_dataset.classes, validation_dataset.classes

num_classes = len(training_dataset.classes)<jupyter_output><empty_output><jupyter_text>Model Instantiation<jupyter_code>model = MyCNN(num_classes)<jupyter_output><empty_output><jupyter_text>We always need to keep track of where our PyTorch tensors are being kept i.e. whether or not they're on the GPU.<jupyter_code>device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") # Determine whether a GPU is available
model.to(device) # send model to GPU<jupyter_output><empty_output><jupyter_text>We now need to define our loss function and optimiser<jupyter_code>loss_fn = nn.CrossEntropyLoss() # We use Cross Entropy Loss, as this is a classification task
optimizer = optim.Adam(model.parameters(), lr=0.001) # If in doubt, we use Adam as our optimiser<jupyter_output><empty_output><jupyter_text>Training Time!<jupyter_code>total_epoch = 10 # Define how many epochs of training we want

# keep track of things we'd like to plot later
training_losses = []
validation_losses = []
accuracies = []

for epoch in range(total_epoch): # loops through number of epochs
  train_loss = train(model, training_loader, loss_fn, optimizer, device)  # train the model for one epoch
  val_loss, accuracy = validation(model, validation_loader, loss_fn, device) # after training for one epoch, run the validation() function to see how the model is doing on the validation dataset
  
  # keep track of interesting stuff
  training_losses.append(train_loss)
  validation_losses.append(val_loss)
  accuracies.append(accuracy)
  
  print("Epoch: {}/{}, Training Loss: {}, Val Loss: {}, Val Accuracy: {}".format(epoch+1, total_epoch, train_loss, val_loss, accuracy))
  print('-' * 20)

print("Finished Training")

# Save the queen
torch.save(model.state_dict(), 'finished')<jupyter_output><empty_output><jupyter_text>We now want to visualise our results:<jupyter_code>import matplotlib.pyplot as plt
plt.plot(training_losses, label="Training Losses")
plt.plot(validation_losses, label="Validation Losses")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.title("Loss vs Epoch")
plt.legend()
plt.show()

plt.plot(accuracies, label="Accuracy")
plt.xlabel("Epoch")
plt.ylabel("Accuracy")
plt.title("Accuracy vs Epoch")
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08669108669108669, 'fraction_numerical': 0.014774114774114774, 'mean_word_length': 3.6778983438035406, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '65801', 'license': 'MIT', 'n_tokens_mistral': 2415, 'n_tokens_neox': 2238, 'n_words': 803, 'path': 'Workshops/Workshop 3 - Pytorch Models/Workshop_3.ipynb', 'repo_name': 'traffichaos01/Training', 'size': '8190'}