<jupyter_start><jupyter_text>Team Convolution Workbook: AlexNet SubmissionWe have created a general structure for our notebooks. This structure is followed below and changes can be made to it, however, it provides the user with an easy starting point for testing different types of network architectures, data augmentation techniques, and hyperparameters. As is explained in the README.md, we have attempted to make our project as modular as possible by encapuslating any "significant" amount of code in .py files. This has allowed for the members of the team who are investigating the best network architectures and hyperparameters to quickly make use of any new functionality that is implemented and allows the notebook to be kept clean and simple.The pretrained AlexNet achieved Team Convolution's best results on the public leaderboards with an Area Under Receiver Operating Charactersitic Curve (AUC - ROC) score at 0.99256. However, it alexnet fell behind resnet when it came to looking at the larger private dataset. The final (AUC-ROC) score on the public leaderboard was 0.98146. This is a fairly significant difference. From insights provided by <NAME> about the nature of the public scoring it seems that our AlexNet was overfitting for the prediction of COVID-19 patients. The choice made to use AlexNet was largely based on its wide success as a CNN in classification of medical imaging.<jupyter_code>!pip install pycm livelossplot
!pip install wandb
%pylab inline
import os
import random
import copy
import numpy as np
import torch
from torchvision import models, datasets, transforms
import torchvision.models as models
from torch.utils.data import DataLoader, Subset, SubsetRandomSampler, TensorDataset
import torch.nn as nn
import torch.nn.functional as F
from sklearn.model_selection import StratifiedShuffleSplit
import wandb

# our modules
#----------------------------------------------------------------------------------
from utils.helper_functions import * # Importing Helper functions from helper_function.py
from utils.train_tools import *      # Training Functions
from utils.networks import *         # Import Network Architectures we would like to use
from utils.write_results import *
from utils.wandb import *           #Import Weights and Biases tracking tools
#----------------------------------------------------------------------------------

device = 'cpu'
if torch.cuda.device_count() > 0 and torch.cuda.is_available():
    print("Cuda installed! Running on GPU!")
    device = 'cuda'
else:
    print("No GPU available!")
    
wandb.login()<jupyter_output>Cuda installed! Running on GPU!<jupyter_text>Download data (google colab only, this may cause some path issues)<jupyter_code># Run line below if you want to download data directly from kaggle
# kaggle_download("kaggle.json")


#Run lines below if you want to mount the google drive to obtain data
#from google.colab import drive
#drive.mount('/content/gdrive/')

#if zip file manually uploaded run this line
##!unzip gdrive/MyDrive/acse4-ml-2020.zip #change to correct file directory<jupyter_output><empty_output><jupyter_text>Load ImagesHere data is loaded using ImageFolder - https://pytorch.org/vision/stable/datasets.htmlid30The user can set some transformations that they would like to apply by editing transform_train, and transform_test.A few pointers, the normalizing transform used for the transform_test should be be the same as the transform_train.Pre-processing functionalities need to be implemented i.e. data augmentation<jupyter_code># set normalization means and stdevss
means = [0.485, 0.456, 0.406] 
stdevs = [0.229, 0.224, 0.225] # <----- these are the ones for ImageNet, we use them as we are transfer learning

# set how many channels you would like for the data
channels = 3

full_data_set = get_data("xray-data/xray-data/train", transform=None, rgb=True)

# load in validation and trainset apply augmentations to the trainset, normalizations to both. Makes the split using stratified shuffle split
trainset, validset, _ = get_data_augmented("xray-data/xray-data/train", "xray-data/xray-data/testouter", 0.15, means, stdevs)<jupyter_output><empty_output><jupyter_text>Some initial plots to check the dataset has been loaded correctly<jupyter_code>visualise_rand(full_data_set) # this function visualises a set of random nine images from xray_train_dat<jupyter_output><empty_output><jupyter_text>---------------------------------------------------------- Network TrainingThis section of the notebook is where models are trained. The training can be split into three sections which the user may like to run:----------------------1) Training the model with the validation and training splits.    - User manually sets hyperparameters    - Doesn't preform a grid search i.e. runs only for one set of parameters    - Displays a LiveLoss plot of how the area under the curve (AUC-ROC) accuracy changes with epoch    - outputs a model trained test set  with the validation split----------------------2) Train the model using weights and biasis     - Allows the user to perform a grid search that is tracked from wandb    - Lots of useful metrics displayed    - Doesn't output a trained model    - Some grid parameters (optimizer and model) need to be set from within utils/wandb.py    - NOTE: this will take a while to run as it will preform a grid search!    ----------------------    3) Train the model on the full data set (produce submission model)    - User manually sets hyperparameters    - This can be seen as a "complete model" i.e. one from which we would make predictions for submission    ----------------------   Before you carry on with this lets set the Network. See next two cells.NOTE: if you would like to simply reproduce our submission model skip to part 3 in the notebook (after running the next two cells)<jupyter_code>"""
This cell shows how you can import Networks that are custom made.

The Network classes for networks that are user defined can be found in utils/networks.py

The example here shows how you would import the LeNet5() architecture from /networks.py

list of available options:

LeNet5()  <----- LeNet5 network with (1, 299, 299) input size

AlexNet() <----- AlexNet network with (1, 299, 299) input size
"""
#Net = LeNet5()
#print(Net) # <----- print the network to check it looks alright!
"""
This cell shows how you can import networks directly from pytorch

see https://pytorch.org/vision/stable/models.html for list of available models

Can import either a trained or pretrained network using model.<model name>(pretrained=True)

The user will need to change the final layer of the network such that it has 4 output features
as is shown below!
"""

Net = models.alexnet(pretrained=True)
Net.fc = nn.Linear(in_features=4096, out_features=4)
print(Net)<jupyter_output>AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
 [...]<jupyter_text>1) Training the model with the validation and test splits. Set training hyperparameters for training with validation split<jupyter_code># Set Training Hyperparameters for training with validation split
lr = 1e-4
momentum = 0.5
weight_decay = 0
batch_size = 32
test_batch_size = 1000
n_epochs = 12
seed = 42

# set data loaders
train_loader = DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=0)
validation_loader = DataLoader(validset, batch_size=test_batch_size, shuffle=False, num_workers=0)

# set seed
set_seed(seed)
# copy the model
model = copy.deepcopy(Net).to(device)
# pick an optimizer
optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0)
# pick a loss criterion
criterion = nn.CrossEntropyLoss()
model = train_model(model, optimizer, criterion, train_loader, validation_loader, channels, device, lr, momentum, batch_size, test_batch_size, n_epochs)<jupyter_output><empty_output><jupyter_text>2) Wandb.ai grid search run Set params for grid search<jupyter_code># Set Training Hyperparameters for training with validation split (these wont all be used in the grid search)
lr = 1e-4
momentum = 0.5
weight_decay = 0
batch_size = 32
test_batch_size = 1000
n_epochs = 20
seed = 42
#Hyperparameters that we mat want to vary during our run, can be extended upon as needed
hyperparameters = {
    "learning_rate": lr,
    "momentum": momentum,
    "batch_size": batch_size,
    "n_epochs": n_epochs,
    "test_batch_size": test_batch_size,
    "channels": channels
}
#Sweep configuration 1: value ranges for a grid search, program itself determines reliable hyperparameters to test over
bayes_sweep_config = {
    "name": "bayes",
    "method": "bayes",
    "metric": {
        "name": "validation_roc_auc",
        "goal": "maximize"
    },
    "parameters": {
        "learning_rate": {
            "min": 1e-4,
            "max": 1e-1
        },
        "momentum": {
            "min": 0.0,
            "max": 0.99999
        }
    },
    "early_terminate": {
        "type": "hyperband",
        "max_iter": hyperparameters["n_epochs"] // 2,
        "s": 5
    }
}
# sets the grid sweep configurations
# this particular example will run a grid sweep for learning rate and weight_decay using AdamW optimizer
grid_sweep_config = {
    "name": "grid_search",
    "method": "grid",
    "parameters": {
        "learning_rate": {            #grid search value is happening
            "values": [1e-4]  #grid search values
        },
        "weight_decay": {
            "values": [0, 0.1]
        }
    },
    "early_terminate": {
        "type": "hyperband",
        "max_iter": hyperparameters["n_epochs"] // 2, #stops earlier if a bad run
        "s": 5
    }
}<jupyter_output><empty_output><jupyter_text>Run the grid search<jupyter_code>model = copy.deepcopy(Net).to(device)

sweep_wandb(seed, model, device, trainset, validset, grid_sweep_config, hyperparameters, count=4, entity="4-2-convolution", project="AlexNet_augment_ph320")<jupyter_output><empty_output><jupyter_text>3) Start Training with validation and  training set (produce submission model) Set params for final training<jupyter_code>lr = 1e-4
momentum = 0.5
weight_decay = 0
batch_size = 32
test_batch_size = 1000
n_epochs = 12
seed = 42

model = copy.deepcopy(Net).to(device) 
optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0)                 
criterion = nn.CrossEntropyLoss()<jupyter_output><empty_output><jupyter_text>Start training on full dataset (i.e. train + validation)<jupyter_code>train_loader = DataLoader(full_data_set, batch_size=batch_size, shuffle=True, num_workers=0) # loads the full dataset i.e. validation + training
model = train_on_full(model, optimizer, criterion, train_loader, channels, 
                      device, lr, momentum, batch_size, test_batch_size, n_epochs)<jupyter_output><empty_output><jupyter_text>---------------------------------------------- This is where we evaluate and save the modelModels will be saved in the /saved_models folder. csv files containing our predictions for the test set will be saved in the /submissioncsv folderThe cell below is where the user can, and should, control the save file names from to avoid things being overwritten!  Set save names for model and prediciton/submission csvThese need to be changed by the individual<jupyter_code>model_save_name = "AlexNet_pretrain_final.pt"       # name to save model under 
submission_csv_name = "AlexNet_pretrain_final.csv"  # name to save submission csv under
# saving the trained model
torch.save(model.state_dict(), "saved_models/" + model_save_name)
from torch.utils.data import Dataset
from PIL import Image
class CustomDataSet(Dataset):
    """
    Custom data loader so that files are loaded and in the correct order
    This was only really necessary for the purposes of submitting the csv to kaggle in correct order
    """
    def __init__(self, main_dir, transform):
        self.main_dir = main_dir
        self.transform = transform
        all_imgs = os.listdir(main_dir)
        self.total_imgs = ["test_" + str(i) + ".png" for i in range(len(os.listdir(main_dir)))]

    def __len__(self):
        return len(self.total_imgs)

    def __getitem__(self, idx):
        img_loc = os.path.join(self.main_dir, self.total_imgs[idx])
        image = Image.open(img_loc).convert("RGB")
        tensor_image = self.transform(image)
        return tensor_image
my_dataset = CustomDataSet("xray-data/xray-data/testouter/test", transform=train_transform)
test_loader = DataLoader(my_dataset , batch_size=batch_size, shuffle=False, num_workers=4, drop_last=True)
# get predictions made by model on my_dataset
def get_predictions(model, X, device, channels=3):
    """
    Get predictions on a set of data X
    """
    model.eval()
    y_preds = []
    for x in X:
        with torch.no_grad():
            x = x.to(device)
            a2 = model(x.view(-1, 3, 299, 299))
            y_pred = F.log_softmax(a2, dim=1).max(1)[1]
            y_preds.append(y_pred.item())

    return y_preds
preds = get_predictions(model, my_dataset, device, channels=3)
print('done')
# write predictions to csv file, csv saved in /submission_csvs
write_file(submission_csv_name, preds)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11259406502910692, 'fraction_numerical': 0.02016186284253869, 'mean_word_length': 3.7736360555743818, 'pattern_counts': {'":': 33, '<': 49, '<?xml version=': 0, '>': 46, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '510473', 'license': 'MIT', 'n_tokens_mistral': 4354, 'n_tokens_neox': 4099, 'n_words': 1515, 'path': 'alexnet_submission.ipynb', 'repo_name': 'jamessoni/x-ray-ML-classification', 'size': '14092'}