<jupyter_start><jupyter_text>Exercise 2: Data set, augmentation, and baselineIn this exercise we are going to explore how augmented data can be used in training and testing.The code used in the exercise is based on the code already shown during the lecture. In this notebook you get the main code blocks to do the exercise. It helps to look at the code to get inspiration to solve the exercise.<jupyter_code>import numpy as np
import pandas as dp
import skimage.io as io
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load dataIn this execercise we will be working with a CT data set from lungs which are acompanied by a list of labels to tell if the image is healthy or malign. The image slices are stored in a (3D) numpy array and the labels in a pandas data frame. [Pandas](https://pandas.pydata.org/docs/) is a libray that helps you to store tables of data that are searchable and slicable depending on different criteria.__Note:__ Please uncompress the file ```ct_tiles.tif.zip``` before starting.<jupyter_code>img = io.imread('ct_tiles.tif')
labels = dp.read_csv('malignancy.csv')
print('Image size: {0},{1},{2}'.format(img.shape[0],img.shape[1],img.shape[2]))
print('Labels {0}'.format(labels.malignancy.count() ))<jupyter_output>Image size: 6691,64,64
Labels 6691<jupyter_text>Look at the first five labels<jupyter_code>labels.head()<jupyter_output><empty_output><jupyter_text>Look at the first five slices<jupyter_code>plt.figure(figsize=[15,5])
for idx,tile in enumerate(img[0:5]) :
    plt.subplot(1,5,idx+1)
    plt.imshow(tile) ;
    plt.title('Malignancy = {0}'.format(labels.malignancy[idx]))<jupyter_output><empty_output><jupyter_text>Look at the two casesIn our exercise we will work with images of healthy (imgA) and malign sample (imgB).<jupyter_code>imgA=img[labels[labels.malignancy==0.0].idx.iloc[3]]
imgB=img[labels[labels.malignancy==1.0].idx.iloc[3]]
fig,ax=plt.subplots(1,2,figsize=[12,7])

ax[0].imshow(imgA)
ax[0].set_title('Healthy')

ax[1].imshow(imgB)
ax[1].set_title('Malign');<jupyter_output><empty_output><jupyter_text>Augmenting the dataLet's create a data augmentator. Here, the values from the lecture are used, but you can play with different settings for parameters like:- rotation- translation- scaling- etc.<jupyter_code>from keras.preprocessing.image import ImageDataGenerator
img_aug = ImageDataGenerator(
    featurewise_center=False,
    samplewise_center=False,
    zca_whitening=False,
    zca_epsilon=1e-06,
    rotation_range=30.0,
    width_shift_range=0.25,
    height_shift_range=0.25,
    shear_range=0.25,
    zoom_range=0.5,
    fill_mode='nearest',
    horizontal_flip=False,
    vertical_flip=False
)<jupyter_output><empty_output><jupyter_text>This is an example using the lecture code with the lung data.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
%matplotlib inline

tiles = np.expand_dims(img[0:6500:450], -1)
tileLabels = labels.malignancy[0:6500:450]
fig, m_axs = plt.subplots(4, 10, figsize=(14, 9))
# setup augmentation
img_aug.fit(tiles)
real_aug = img_aug.flow(tiles[:10], tileLabels[:10], shuffle=False)
for c_axs, do_augmentation in zip(m_axs, [False, True, True, True]):
    if do_augmentation:
        img_batch, label_batch = next(real_aug)
    else:
        img_batch, label_batch = tiles, tileLabels
    for c_ax, c_img, c_label in zip(c_axs, img_batch, label_batch):
        c_ax.imshow(c_img[:, :, 0], cmap='viridis', vmin=0, vmax=255)
        c_ax.set_title('{}\n{}'.format(
            c_label, 'aug' if do_augmentation else ''))
        c_ax.axis('off')<jupyter_output><empty_output><jupyter_text>Testing baselines with augmented dataFrom the lectures you learned that in order to compare the performance of an algorithm, you need a baseline algorithm. Previously, you heard about two algorithms:- Dummy classifier which uses the majority as prediction for any input data.- Nearest neighbor that uses the nearest feature as prediction.Below you get examples on the lung data.  Dummy classifier<jupyter_code>from sklearn.dummy import DummyClassifier
dc = DummyClassifier(strategy='most_frequent')
dc.fit(tiles, tileLabels)
dc.predict(img[1:6000:1000])<jupyter_output><empty_output><jupyter_text>Nearest neighbor<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
neigh_class = KNeighborsClassifier(n_neighbors=1)
neigh_class.fit(tiles.reshape((15, -1)), tileLabels)
# predict on a few images
neigh_class.predict(tiles.reshape((15, -1)))<jupyter_output><empty_output><jupyter_text>Confusion matrixThis is some support code to produce the confusion matrix.<jupyter_code>import seaborn as sns
import pandas as pd
from sklearn.metrics import accuracy_score, confusion_matrix
def print_confusion_matrix(confusion_matrix, class_names, figsize = (10,7), fontsize=14):
    """Prints a confusion matrix, as returned by sklearn.metrics.confusion_matrix, as a heatmap.
    
    Stolen from: https://gist.github.com/shaypal5/94c53d765083101efc0240d776a23823
    
    Arguments
    ---------
    confusion_matrix: numpy.ndarray
        The numpy.ndarray object returned from a call to sklearn.metrics.confusion_matrix. 
        Similarly constructed ndarrays can also be used.
    class_names: list
        An ordered list of class names, in the order they index the given confusion matrix.
    figsize: tuple
        A 2-long tuple, the first value determining the horizontal size of the ouputted figure,
        the second determining the vertical size. Defaults to (10,7).
    fontsize: int
        Font size for axes labels. Defaults to 14.
        
    Returns
    -------
    matplotlib.figure.Figure
        The resulting confusion matrix figure
    """
    df_cm = pd.DataFrame(
        confusion_matrix, index=class_names, columns=class_names, 
    )
    fig, ax1 = plt.subplots(1, 1, figsize=figsize)
    try:
        heatmap = sns.heatmap(df_cm, annot=True, fmt="d")
    except ValueError:
        raise ValueError("Confusion matrix values must be integers.")
    heatmap.yaxis.set_ticklabels(heatmap.yaxis.get_ticklabels(), rotation=0, ha='right', fontsize=fontsize)
    heatmap.xaxis.set_ticklabels(heatmap.xaxis.get_ticklabels(), rotation=45, ha='right', fontsize=fontsize)
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    return ax1
pred_values = neigh_class.predict(tiles.reshape((15, -1)))
ax1 = print_confusion_matrix(confusion_matrix(tileLabels, pred_values), class_names=range(2))
ax1.set_title('Accuracy: {:2.2%}'.format(accuracy_score(tileLabels, pred_values)));<jupyter_output><empty_output><jupyter_text>Exercise1. Create an augmented data set with 110 images based on 11 images. Pay attention to obtaining a representative balance between healthy and malign samples.2. Train the baseline algorithms using     1. the original data of the same size as the augmented data set.    2. the augmented data.3. Compare the performance using the confusion matrix.  Task 1Create the original data set with 11 images. The data is fortunately sorted in a way that all malign and healty are grouped together. This makes it possible to use array slicing with a constant increment.The total number of images is 6691 which divided by 11 gives the step 608. Array slicing in python works like ```a[start:end:step]``` the index for _end_ is not included, so you will get elements from _start_ to _end-1_. In the example below, I adjusted the end value to $11\cdot{} 608=6688$.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
%matplotlib inline

tiles = np.expand_dims(img[0:6688:608], -1)
tileLabels = labels.malignancy[0:6688:608]<jupyter_output><empty_output><jupyter_text>Next, we configure the image augmentor. In this case I recycled the one from above, but feel free to play with the numbers.<jupyter_code>from keras.preprocessing.image import ImageDataGenerator
img_aug = ImageDataGenerator(
    featurewise_center=False,
    samplewise_center=False,
    zca_whitening=False,
    zca_epsilon=1e-06,
    rotation_range=30.0,
    width_shift_range=0.25,
    height_shift_range=0.25,
    shear_range=0.25,
    zoom_range=0.5,
    fill_mode='nearest',
    horizontal_flip=False,
    vertical_flip=False
)<jupyter_output><empty_output><jupyter_text>Provide the extracted images to the augmentor and create an interator.<jupyter_code># setup augmentation
img_aug.fit(tiles)
real_aug = img_aug.flow(tiles[:11], tileLabels[:11], shuffle=False)<jupyter_output><empty_output><jupyter_text>Show the created new tiles.<jupyter_code>fig, m_axs = plt.subplots(10, 11, figsize=(15,15))

aug_tiles  = tiles
aug_labels = tileLabels

for c_axs, do_augmentation in zip(m_axs, [False,True, True, True, True,True,True,True,True,True]):
    if do_augmentation:
        img_batch, label_batch = next(real_aug)
        aug_tiles=np.concatenate((aug_tiles,img_batch))
        aug_labels=np.concatenate((aug_labels,label_batch))
    else:
        img_batch, label_batch = tiles, tileLabels
    for c_ax, c_img, c_label in zip(c_axs, img_batch, label_batch):
        c_ax.imshow(c_img[:, :, 0], cmap='viridis', vmin=0, vmax=255)
        c_ax.set_title('{}\n{}'.format(c_label, 'aug' if do_augmentation else ''))
        c_ax.axis('off')<jupyter_output><empty_output><jupyter_text>Task 2Lets first create some test data<jupyter_code>testData = img[1:6000:200]
testLabels = labels.malignancy[1:6000:200].values<jupyter_output><empty_output><jupyter_text>DummyLet's train the dummy classifier.<jupyter_code>from sklearn.dummy import DummyClassifier
dc = DummyClassifier(strategy='most_frequent')
dc.fit(tiles, tileLabels)<jupyter_output><empty_output><jupyter_text>Test how it performs using unseen data. In this case, we start with index 1 instead of 0 for the training data.<jupyter_code>dc.predict(testData)
testLabels<jupyter_output><empty_output><jupyter_text>The training data contains more healthy tiles. Thus, with a most frequent classifier all images will be classified as healthy.  Nearest neighborLet's train and test the nearest neighbor classifier.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
neigh_class = KNeighborsClassifier(n_neighbors=1)
neigh_class.fit(aug_tiles.reshape((aug_tiles.shape[0], -1)), aug_labels)<jupyter_output><empty_output><jupyter_text>Now, lets test the classifier on the test data set.<jupyter_code># predict on a few images
neigh_class.predict(testData.reshape(testData.shape[0], -1))<jupyter_output><empty_output><jupyter_text>and compare to the labels of the selected images<jupyter_code>testLabels<jupyter_output><empty_output><jupyter_text>This is not an overwhelming performance! But may serve as baseline.  Task 3First we look at the performance on the training data.<jupyter_code>pred_values = neigh_class.predict(aug_tiles.reshape((aug_tiles.shape[0], -1)))
ax1 = print_confusion_matrix(confusion_matrix(aug_labels, pred_values), class_names=range(2))
ax1.set_title('Accuracy: {:2.2%}'.format(accuracy_score(aug_labels, pred_values)));
pred_values = neigh_class.predict(testData.reshape((testData.shape[0], -1)))
ax1 = print_confusion_matrix(confusion_matrix(testLabels, pred_values), class_names=range(2))
ax1.set_title('Accuracy: {:2.2%}'.format(accuracy_score(testLabels, pred_values)));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10073470832964504, 'fraction_numerical': 0.025139417544480836, 'mean_word_length': 5.530635838150289, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 84, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '412502', 'license': 'Apache-2.0', 'n_tokens_mistral': 3887, 'n_tokens_neox': 3686, 'n_words': 999, 'path': 'Exercises/02-Augmentation/02-AugmentationExercise-Solution.ipynb', 'repo_name': 'ImagingLectures/Quantitative-Big-Imaging-2022', 'size': '11297'}