<jupyter_start><jupyter_text>Convolutional Neural Network and Model Interpretation- Train a convolutional neural network- Visualize the last "feature" layer- Visualize filters in the first convolution layer- Visualize activation regions of the input image  Setup driveRun the following cell to mount your Drive onto Colab. Go to the given URL and once you login and copy and paste the authorization code, you should see "drive" pop up in the files tab on the left.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive<jupyter_text>Click the little triangle next to "drive" and navigate to the "AI4All Chest X-Ray Project" folder. Hover over the folder and click the 3 dots that appear on the right. Select "copy path" and replace `PASTE PATH HERE` with the path to your folder.<jupyter_code># cd "PASTE PATH HERE"
c
cd "/content/drive/My Drive/AI4All Project/AI4All Chest X-Ray Project"<jupyter_output>/content/drive/My Drive/AI4All Project/AI4All Chest X-Ray Project<jupyter_text>Import necessary libraries<jupyter_code>import math
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import seaborn as sns

import torch
from torch import nn, optim
from torch.utils.data import DataLoader, random_split

import torchvision
from torchvision import datasets, models, transforms

from utils.datahelper import get_random_image
from utils.plotting import imshow_dataset, imshow_filters, plot_accuracy_curves, plot_loss_curve
from utils.modelhelper import calc_accuracy, get_train_validate_sizes, predict, train, visualize_cam, GradCAM<jupyter_output><empty_output><jupyter_text>Setup paths<jupyter_code>path_to_dataset = os.path.join('data')

path_to_images = os.path.join(path_to_dataset, 'images')

metadata = pd.read_csv(os.path.join(path_to_dataset, 'metadata_train.csv'))<jupyter_output><empty_output><jupyter_text>Convolutional Neural Network (CNN)  Preprocess and split data The preprocessing and data splitting steps are the same as for the fully connected network**Preprocess data**Define preprocessing transforms<jupyter_code>DATA_MEAN = 0.544
DATA_STD = 0.237

RESIZE_SIZE = 235
CROP_SIZE = 224 
IM_SIZE = CROP_SIZE

# preprocess transforms
data_transforms = transforms.Compose([
            transforms.Grayscale(),
            transforms.Resize(RESIZE_SIZE),
            transforms.CenterCrop(CROP_SIZE),
            transforms.ToTensor(),
            transforms.Normalize(mean=DATA_MEAN, std=DATA_STD)])<jupyter_output><empty_output><jupyter_text>Sample data augmentation transforms (not used). Modify this code if you want to use augmentation!<jupyter_code># Framework for using data augmentation (not optimized)

DEGREE = 5
augmentation_transforms = [transforms.RandomAffine(degrees=DEGREE,
                                                   translate=((DEGREE / 100), (DEGREE/ 100)),
                                                   scale=(1 - (DEGREE / 100), 1 + (DEGREE / 100)),
                                                   shear=0),
                            transforms.RandomHorizontalFlip()]

data_transforms_augmented = transforms.Compose([
            transforms.Grayscale(),
            transforms.Resize(RESIZE_SIZE),
            *augmentation_transforms,
            transforms.CenterCrop(CROP_SIZE),
            transforms.ToTensor(),
            transforms.Normalize(mean=DATA_MEAN, std=DATA_STD)])<jupyter_output><empty_output><jupyter_text>Create image dataset using defined transforms<jupyter_code>dataset = datasets.ImageFolder(path_to_images, transform=data_transforms)<jupyter_output><empty_output><jupyter_text>**Split the data into training and validation set**Randomly split dataset into training and validation set. Use DataLoaders define how to load data from training and validation datasets.<jupyter_code>TRAIN_RATIO = 0.75
BATCH_SIZE = 64

# calculate training and validation set sizes
train_size, validate_size = get_train_validate_sizes(dataset, TRAIN_RATIO)

# split dataset into two training set and validation set
train_dataset, validate_dataset = random_split(dataset, [train_size, validate_size])

# create dataloaders
train_loader = DataLoader(dataset=train_dataset, shuffle=True, batch_size=BATCH_SIZE)
validate_loader = DataLoader(dataset=validate_dataset, shuffle=False, batch_size=len(validate_dataset))<jupyter_output><empty_output><jupyter_text>Ignore thisCalculate convolution output / linear input size for the `nn.Sequential` container. There may be a better way to do this (without using `nn.Module`).<jupyter_code>def calc_conv2d_dim(w, p=0, d=1, k=3, s=1):
    '''Calculate output dimension of a conv2d layer. Assumes input width=height 
    and kernels are square

    Args:
      w: input width/height (assumed equal)
      p: padding
      d: dilation
      k: kernel size (assumed equal)
      s: stride

    Returns: output dimension (width = height) 
    '''

    return math.floor((w + 2*p - d*(k-1) - 1) / s + 1)
def calc_maxpool2d_dim(w, p=0, d=1, k=2, s=2):
    '''Calculate output dimension of a maxpool2d layer. Assumes input width=height 
    and kernels are square

    Args:
      w: input width/height (assumed equal)
      p: padding
      d: dilation
      k: kernel size (assumed equal)
      s: stride

    Returns: output dimension (width = height) 
    '''
    return math.floor((w + 2*p - d*(k-1) - 1) / s + 1)<jupyter_output><empty_output><jupyter_text>Calculate the convolution block output size. To calculate the input size for the first linear layer in the fully connected block: k x w x hk: number of kernels in the last conv layerw = h: convolution block output size<jupyter_code># order of conv2d and maxpool2d layers as a string
layers = 'cmcm'

current_size = IM_SIZE

# calculate convolution block output size
# modify if you need to parameterize layers individually
for l in layers:
    if l == 'c':
        current_size = calc_conv2d_dim(current_size, k=3)
    elif l == 'm':
        current_size = calc_maxpool2d_dim(current_size)

cnn_out_size = current_size
print(f'Convolution block output size: {cnn_out_size}')<jupyter_output>Convolution block output size: 54<jupyter_text>Setup and train CNN**Define CNN**The CNN network consists of a convolution section followed by a fully connected section. The convolutin section is a sequence of convolution (`nn.Conv2d`) layers followed up by activation (`nn.ReLu`) and max pooling (`nn.MaxPool2d`). The output from the convolution section is then flattened (`nn.Flatten`) for the fully connected section.As before, the fully connected layer is a sequence of linear (`nn.Linear`) layers followed by activation (`nn.ReLu`). We end the network with an output linear layer followed by log softmax (`nn.LogSoftmax`). This outputs the log probability (likelihood) for each class label.Read about convolution layers in PyTorch [here](https://pytorch.org/docs/master/nn.htmlconvolution-layers)<jupyter_code>model = nn.Sequential(nn.Conv2d(1, 16, 3),
                      nn.ReLU(),
                      nn.MaxPool2d(2, 2),
                      nn.Conv2d(16, 16, 3),
                      nn.ReLU(),
                      nn.MaxPool2d(2, 2),
                      nn.Flatten(),
                      nn.Linear(16 * cnn_out_size * cnn_out_size, 16),
                      nn.ReLU(),
                      nn.Linear(16,  8),
                      nn.ReLU(),
                      nn.Linear(8, 2),
                      nn.LogSoftmax(dim=1))

print(model)<jupyter_output>Sequential(
  (0): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1))
  (1): ReLU()
  (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (3): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1))
  (4): ReLU()
  (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (6): Flatten()
  (7): Linear(in_features=46656, out_features=16, bias=True)
  (8): ReLU()
  (9): Linear(in_features=16, out_features=8, bias=True)
  (10): ReLU()
  (11): Linear(in_features=8, out_features=2, bias=True)
  (12): LogSoftmax()
)<jupyter_text>**Train CNN**Same function as for fully connected NN<jupyter_code>train_accuracy_log, validate_accuracy_log, loss_log = train(model, train_loader, validate_loader, epoch_num=1)
plot_accuracy_curves(train_accuracy_log, validate_accuracy_log)<jupyter_output><empty_output><jupyter_text>Model InterpretationThis section walks through three approaches for model interpretation, or understanding how a neural network makes its decisions. Don't worry about the code in this section as it's a bit more complicated. We will focus on the visualizations and interpretations as we go through the section together.  Visualize features in fully connected layerWe can visualize the feautures being learned in a fully connected layer by recording and plotting the layer activations for a given set of input images. The space of the features is also called the latent space. Here, we take a look at the last layer before the output layer.<jupyter_code>def save_output_hook(module, input, output):
    '''Hook for saving module output to module'''
    setattr(module, '_output_hook', output)
def get_feature_outputs(model, test_loader, layer_num=-3, inputs_num=300):
  '''Get outputs for the feauture layer. Default feature layer is the fully 
  connected layer right before the output layer

  Args:
    model: neural network
    test_loader: DataLoader for test images
    layer_num: layer number for activation (e.g. relu) layer of the feature layer
    inputs_num: number of test inputs to run

  Returns: layer_outputs, actual_labels
  '''

  # register a hook to save the output
  output_handle = model[layer_num].register_forward_hook(save_output_hook)

  layer_outputs = []
  labels = []

  for i in range(inputs_num):

    # get the next image
    input, label = next(iter(test_loader))

    # forward pass with an input image
    output = model(input)

    # get the activation value
    layer_output = model[layer_num]._output_hook.data
    layer_output = layer_output[0].numpy()

    layer_outputs.append(layer_output)
    labels.append(label.item())

  # remove hook
  output_handle.remove()

  layer_outputs = np.array(layer_outputs)

  return layer_outputs, labels

# define test data loader
dataset_test = datasets.ImageFolder(path_to_images, transform=data_transforms)
test_loader = DataLoader(dataset=dataset_test, shuffle=True, batch_size=1)
# get feature activations before training
activation_pretrain, labels_pretrain = get_feature_outputs(model, test_loader)

train_accuracy_log = []
validate_accuracy_log = []
loss_log = []

# get feature activations after training for one epoch
training_output = train(model, train_loader, validate_loader, epoch_num=1)

train_accuracy_log += training_output[0]
validate_accuracy_log += training_output[1]
loss_log += training_output[2]

activation_train1, labels_train1 = get_feature_outputs(model, test_loader)

# get feature activations after training for another 3 epochs
training_output = train(model, train_loader, validate_loader, epoch_num=3)

train_accuracy_log += training_output[0]
validate_accuracy_log += training_output[1]
loss_log += training_output[2]

activation_train2, labels_train2 = get_feature_outputs(model, test_loader)
# plot training curves
plot_accuracy_curves(train_accuracy_log, validate_accuracy_log)
plot_loss_curve(loss_log)
from sklearn.manifold import TSNE

def plot_tsne(activations, labels, ax):
  '''Compute and plot tsne plot'''

  tsne = TSNE(n_components=2, perplexity=40, n_iter=5000)
  feature_tsne = tsne.fit_transform(activations)

  df_tsne = pd.DataFrame({
      'tsne_x': feature_tsne[:,0],
      'tsne_y': feature_tsne[:,1],
      'target_labels': labels})

  sns.scatterplot(x='tsne_x', y='tsne_y', hue='target_labels', data=df_tsne, legend='full', ax=ax)
# plot tsne plots
fig, ax = plt.subplots(1, 3, figsize=(15,5))
ax = ax.ravel()

plot_tsne(activation_pretrain, labels_pretrain, ax=ax[0])
ax[0].set_title('Pre-training')
plot_tsne(activation_train1, labels_train1, ax=ax[1])
ax[1].set_title('After 1 epoch')
plot_tsne(activation_train2, labels_train2, ax=ax[2])
ax[2].set_title('After 3 epochs')

plt.suptitle('FC feature TSNE visualization')
plt.show()

from sklearn.decomposition import PCA

def plot_pca(activations, labels, ax):
  '''Compute and plot PCA plot'''

  pca = PCA(n_components=2)
  feature_pca = pca.fit_transform(activations)

  df_pca = pd.DataFrame({
      'pca1': feature_pca[:,0],
      'pca2': feature_pca[:,1],
      'target_labels': labels})

  sns.scatterplot(x='pca1', y='pca2', hue='target_labels', data=df_pca, legend='full', ax=ax)

# plot pca plots
fig, ax = plt.subplots(1, 3, figsize=(15,5))
ax = ax.ravel()

plot_pca(activation_pretrain, labels_pretrain, ax=ax[0])
ax[0].set_title('Pre-training')
plot_pca(activation_train1, labels_train1, ax=ax[1])
ax[1].set_title('After 1 epoch')
plot_pca(activation_train2, labels_train2, ax=ax[2])
ax[2].set_title('After 3 epochs')

plt.suptitle('FC feature PCA visualization')
plt.show()<jupyter_output><empty_output><jupyter_text>Visualize CNN filtersWe can also visualize the filters being learned in the first convolution layer of a CNN **Show filters for the CNN we just trained**<jupyter_code># get filters from the first convolution layer
filters = model[0].weight.data
imshow_filters(filters, single_channel=True)<jupyter_output><empty_output><jupyter_text>**Show filters for a pre-trained model (AlexNet)**<jupyter_code># load pre-trained alexnet
alexnet = models.alexnet(pretrained=True)

# get first layer weights
filters = alexnet.features[0].weight.data
imshow_filters(filters)<jupyter_output>Downloading: "https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth" to /root/.cache/torch/checkpoints/alexnet-owt-4df8aa71.pth<jupyter_text>**Show filters for a pre-trained model (VGG16)**<jupyter_code># load vgg
vgg16 = models.vgg16(pretrained=True)

# get first layer weights
filters = vgg16.features[0].weight.data
imshow_filters(filters)<jupyter_output>Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /root/.cache/torch/checkpoints/vgg16-397923af.pth<jupyter_text>Class activation mapsClass activation maps (CAMs) show which region of the image is being used (activated) to make the prediction.<jupyter_code>def show_random_gradcam(model, target_layer, dataset):
  # get a random image
  im = get_random_image(dataset)
  im_batch = torch.stack([im])

  # run gradient class activation mapping
  gradcam = GradCAM(model, target_layer)
  mask, logit = gradcam(im_batch)
  heatmap, cam_result = visualize_cam(mask, im_batch)

  # show results
  fig, ax = plt.subplots(1,3)
  plt.gray()
  ax[0].imshow(np.squeeze(im))
  ax[1].imshow(transforms.ToPILImage()(heatmap))
  ax[2].imshow(transforms.ToPILImage()(cam_result))
# find the last convolution layer and set as target layer
target_layer = model._modules['3']

show_random_gradcam(model, target_layer, dataset)
show_random_gradcam(model, target_layer, dataset)
show_random_gradcam(model, target_layer, dataset)
show_random_gradcam(model, target_layer, dataset)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09869515159268262, 'fraction_numerical': 0.020660099782525265, 'mean_word_length': 4.786454478164322, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '98973', 'license': 'MIT', 'n_tokens_mistral': 5242, 'n_tokens_neox': 4935, 'n_words': 1350, 'path': 'notebooks/Day 5 CNN and Interpretation.ipynb', 'repo_name': 'inachen/ai4all-computer-vision-project', 'size': '15634'}