<jupyter_start><jupyter_text>Malaria Project - Implementation Detecting Malaria cells using CNN and TF 2.0  Table of Contents:* [Introduction](Introduction)* [Brief discussion about existing kernels](Brief-discussion-about-existing-kernels)  * [Kernel 1](Kernel-1:-Detecting-Malaria-cells-using-Convolutional-Neural-Network)* [Getting Started](Getting-Started)   * [Imports](Imports)  * [Load Data](LoadData)* [Preprocess Data](Preprocess-Data)  * [Feature Engineering](Feature-Engineering)  * [Data Augmentation](Data-Augmentation)  * [Visualizing Augmented Data](Visualizing-Augmented-Data)  * [Generate Data and Labels](Generate-all-Data-and-Labels)  * [Save/Load Data](Save/Load-Data)  * [Data-Visualization](Data-Visualization)  * [Split Data](Split-Data)* [Setup CNN](Setup-CNN)* [Train Model](Train-Model)* [Evaluation and Predictions](Evaluation-and-Predictions)* Visualize Accuracy* Results, Confusion Matrix* Visualize Results* [Licenses](Licenses)* [Acknowledgements](Acknowledgements)  Introduction This notebook is the result of a course work I did for class that introduced Neural Networks at HTW Berlin.There is a second notebook summarizing the theory behind CNNs [here](https://github.com/lai-la/MalariaProject/blob/master/malaria_project_theoretical_background.ipynb).  Brief discussion about existing kernelsIn this section two kernels from the kaggle website dealing with the malaria dataset will be analized regarding data augmentation, data-split, cnn-architecture, used activation functions and obtained results (accuracy ...).They are being used as a guideline for our own implementation. Update: Only one Kernel in this Notebook, the other in Lukas's notebook (DenseNet)   Kernel 1: Detecting Malaria cells using Convolutional Neural Network By: <NAME>Source:https://www.kaggle.com/kushal1996/detecting-malaria-cnn Data Augmentation:- Done using the Python Image Libary (PIL) that provides support for opening, editing and saving many different data formats- The quantity of the "Parasitized" data is being quadrupled.- The quantity of the "Uninfected" data is being tripled. Data-SplitThe data in this kernel is being split into training, evaluation and test data:   - training-data: 80%   - evaluation-data: 10%   - test-data: 10% Used CNN-Architecture and activation functions:| Layer-Type   | Units | Nb_Filters | KernelSize PoolSize | Padding | Activation | Stride ||--------------|-------|------------|---------------------|---------|------------|--------|| Input        | _     | _          | _                   | _       | _          | _      ||  Conv        | _     | 50         | 7x7                 | same    | ReLU       | _      || Conv         | _     | 90         | 3x3                 | valid   | ReLU       | _      || Conv         | _     | 10         | 5x5                 | same    | ReLU       | _      || Pool         | _     | _          | 2x2                 | _       | _          | 2      || Conv         | _     | 5          | 3x3                 | same    | ReLU       | _      || Pool         | _     | _          | 2x2                 | same    | _          | 2      || Pool_flatten | _     | _          | _                   | _       | _          | _      || FC           | 2000  | _          | _                   | _       | ReLU       | _      || FC           | 1000  | _          | _                   | _       | ReLU       | _      || FC           | 500   | _          | _                   | _       | ReLU       | _      || Logits       | 2     | _          | _                   | _       | _          | _      | Obtained Results:| _           | precision |   recall | f1-score |  support ||-------------|-----------|----------|----------|----------||           0 |      0.90 |     0.93 |     0.91 |     4223 ||           1 |      0.95 |     0.92 |     0.93 |     5423 ||_            | _         | _        | _        | _        ||   micro avg |      0.92 |     0.92 |     0.92 |     9646 ||   macro avg |      0.92 |     0.92 |     0.92 |     9646 ||weighted avg |      0.92 |     0.92 |     0.92 |     9646 || _           | _         | _        | _        | _        ||0.9238026124818578  Getting Started  Imports<jupyter_code># Uncomment to install TF 2.0 if you don't have it
#!pip install tensorflow==2.0.0b1
# linear algebra
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, models

# Split arrays or matrices into random train and test subsets
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# computer vision library, image processing
import cv2

# statistical data visualization
import seaborn as sns

# used for data augmentation
from PIL import Image
from PIL import ImageOps
from PIL import ImageFilter

# to use operating system dependent functionality
import os

import pandas as pd<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code># Loads data by classes/categories (e.g Parasitized, Uninfected)
def load_data(path):
        '''
        Args:
            path: path to your data, relative to the location of this project
        Returns:
            Dict((class, label): list<data>)
        '''

        clazzes = os.listdir(path)
        clazzes.sort()
        clazz_data = dict()
        for clazz, label in zip(clazzes, range(len(clazzes))):
            clazz_data.update({(clazz, label) : (os.listdir(path + "/" + clazz))})
        return clazz_data
        
# Check whether you obtained the right results
check_data = load_data("data/cell_images")
print(check_data.keys())
for clazz in check_data:
    print(clazz[0] + " " + str(clazz[1]) + ": " + str(len(check_data.get(clazz))))<jupyter_output>dict_keys([('Parasitized', 0), ('Uninfected', 1)])
Parasitized 0: 13780
Uninfected 1: 13780<jupyter_text>Preprocess DataIn this section the data will be preprocessed in order to fulfil the requirements we have:- feature engineering: here we will eliminate files that are no images (e.g. my Uninfected and Parasitized folders both contain a temp file which we don't need and that would cause the program to fail)- resize the image to be of the input size we want to use- mode: decide on RGB, Greyscale...  Feature Engineering<jupyter_code># preprocesses the data (only images) and resizes them. The mode is set to RGB as default. It could also be Greyscale...
def preprocess_data(image_path, h_img, w_img, mode='RGB'):
    '''
    Args:
        image_path: image path, relative to the location of this project
        h_img: new height of image
        w_img: new width of image
        mode='RGB'
    Returns:
        preprocessed image
    '''
    image = cv2.imread(image_path)
    if image is None:
        return None
    img_as_array = Image.fromarray(image, mode)
    preprocessed_img = img_as_array.resize((h_img,w_img))
    return preprocessed_img<jupyter_output><empty_output><jupyter_text>Data Augmentation- form of regularization- useful for cells because impossible to predict position they will have<jupyter_code># augments data to get bigger training and testing sets
def augment_data(image, label, rotations=None, mirror=False, blur=False):
    '''
    Args:
        image: image that is to be aused for augmentation
        label: the label of image, so that the augmented data will have the same label
        rotations: rotates the image by angles in list, e.g rotations=[45,75,95]
        mirror: mirrors the image vertically if true
        blur: blurs the image if true
    Returns:
        img_data: List with augmented data
        img_labels: Respective labels to img_data  
    '''
    img_data = []
    img_labels = []
    if rotations:
        for rot in rotations:
            im = image.rotate(rot)
            img_data.append(np.array(im))
            img_labels.append(label)
    if mirror:
        im = ImageOps.mirror(image)
        img_data.append(np.array(im))
        img_labels.append(label)
    if blur:
        im = image.filter(ImageFilter.BLUR)
        img_data.append(np.array(im))
        img_labels.append(label)
    return img_data, img_labels<jupyter_output><empty_output><jupyter_text>Visualizing Augmented Data<jupyter_code>image = preprocess_data("data/cell_images/Parasitized/C99P60ThinF_IMG_20150918_141520_cell_107.png", 50,50)
augmented, auglabel = augment_data(image, 0 ,rotations=[45,75,95], mirror = True, blur = True)
images = []
images.append(np.array(image))
images.extend(augmented)
titles = ["original", "rotated 45°", "rotated 75°", "rotated 95°", "mirrored", "blured"]
plt.figure(1, figsize=(20,20))
n = 0
for img in images:
    n += 1
    plt.subplot(3,3,n)
    plt.subplots_adjust(hspace=0.3, wspace= 0.3)
    plt.imshow(img)
    plt.title(titles[n-1], fontsize=25)
    plt.xticks([]), plt.yticks([])

plt.show()<jupyter_output><empty_output><jupyter_text>Generate all Data and Labels<jupyter_code># Generates the dataset and the respective labels in a way it can be used as training and test data later on
def generate_data(path, h_img, w_img, mode='RGB', rotations=None, mirror=False, blur=False):
    '''
    Args:
        image_path: image path, relative to the location of this project
        h_img: new height of image
        w_img: new width of image
        mode='RGB'
        rotations: rotates the image by angles in list, e.g rotations=[45,75,95]
        mirror: mirrors the image vertically if true
        blur: blurs the image if true
    Returns:
        data: List of data/images
        labels: List of labels corresponding to the data
    '''
    data = []
    labels = []
    clazz_data = load_data(path)
    for clazz in clazz_data:
        for img in clazz_data.get(clazz):
            img_path = path + "/" + clazz[0] + "/" + img
            prep_img = preprocess_data(img_path, h_img, w_img)
            if prep_img is None:
                continue
            data.append(np.array(prep_img))
            labels.append(clazz[1])
            augm_data, augm_label = augment_data(prep_img,clazz[1], rotations, mirror, blur)
            data.extend(augm_data)
            labels.extend(augm_label)
    return data, labels
        
    
#checking whether data and label generation work properly
data, labels = generate_data("data/cell_images", 50, 50, rotations=[45,75], blur = True)

#There should be 110232 images in total
print('Number of data: {} \nNumber of labels: {}'.format(len(data), len(labels)))

# There should be 55116 images for each class
print('Expecting labels to be 0 : l[55114] = {}, l[55115] = {} \nExpecting labels to be 1 : l[55116] = {}, l[55117] = {}'.format(labels[55114],labels[55115],labels[55116],labels[55117]))
# some ideas for different data genareation examples

#dat_44, labs_44 = generate_data("data/cell_images", 44, 44, rotations=[45,75,120], mirror = True, blur = True)
#orig_d, orig_l =generate_data("data/cell_images", 50, 50)
#orig_d_44, orig_l_44 = generate_data("data/cell_images", 44, 44)
#print(len(orig_d_44))
#print(len(orig_l_44))<jupyter_output><empty_output><jupyter_text>Save/Load Data<jupyter_code># saves data and labels to .npy files
def save_data(cell_file_name, cells, label_file_name, labels):
    '''
    Args:
        cell_file_name: String -> name the file with the data should have
        label_file_name: String -> name the file with the label should have
    '''
    cellz = np.array(cells)                 
    labelz = np.array(labels)
    np.save(cell_file_name, cellz)
    np.save(label_file_name, labelz)

#uncomment if you want to save your results
#save_data("data_augm",data, "labels_augm",labels)

# some ideas for different data genareation examples

#save_data("data_augm_44",dat_44, "labels_augm_44",labs_44)
#save_data("orig_d", orig_d, "orig_l",orig_l)
#save_data("orig_d_44", orig_d_44, "orig_l_44", orig_l_44)
# loads data and label .npy file from current directory
def load_saved_data(data, labels):
    '''
    Args:
        data: name of data file without .npy
        labels: name of label file without .npy
    Returns:
        c: list containing cell-data
        l_ list containing respective labels
    '''
    c=np.load(data + ".npy")
    l=np.load(labels + ".npy")
    return c,l
c,l = load_saved_data("data_augm","labels_augm")
print('number of data c: {} \nnumber of labels l: {}'.format(len(c), len(l)))
# some ideas for different data genareation examples

#o_d, o_l = load_saved_data("orig_d", "orig_l")
#o_d_44, o_l_44 = load_saved_data("orig_d_44", "orig_l_44")
#ce,la = load_saved_data("data_augm_44", "labels_augm_44")
#print('number of data o_d: {} \nnumber of labels o_l: {}'.format(len(o_d), len(o_l)))
#print('number of data o_d_44: {} \nnumber of labels o_l_44: {}'.format(len(o_d_44), len(o_l_44)))
#print('number of data ce: {} \nnumber of labels la: {}'.format(len(ce), len(la)))<jupyter_output><empty_output><jupyter_text>Data-Visualization<jupyter_code>plt.figure(1, figsize=(15,9))
n=0
for i in range(49):
    n +=1
    r = np.random.randint(0, c.shape[0], 1)
    plt.subplot(7,7,n)
    plt.subplots_adjust(hspace=0.5, wspace= 0.5)
    plt.imshow(c[r[0]])
    plt.title('{} : {}'.format('Infected' if l[r[0]] == 0 else 'Uninfected', l[r[0]]))
    # Disable xticks:
    plt.xticks([]), plt.yticks([])
plt.show()
plt.figure(1, figsize=(15,7))
plt.subplot(1,2,1)
plt.imshow(c[0])
plt.title('Infected Cell')
plt.xticks([]), plt.yticks([])

plt.figure(1, figsize=(15,7))
plt.subplot(1,2,2)
plt.imshow(c[60000])
plt.title('Uninfected Cell')
plt.xticks([]), plt.yticks([])

plt.show()<jupyter_output><empty_output><jupyter_text>Shuffle and Normalize Data<jupyter_code># randomly shuffles the data and normalizes the image data 
def shuffle_and_normalize(data, labels):
    '''
    Args:
        data: data that is to be used for trainin, testing, evaluating the CNN
        labels: corresponding labels
    Returns:
        randomly shuffled data and labels (the indexes of data and it's corresponing label are shuffled in the same way)
    '''
    r = np.arange(data.shape[0])

    # set the seed for shuffle, to make results reproduceable
    np.random.seed(123)
    # shuffles all the values in r
    # Return evenly spaced values within a given interval(Here: 0-110231)
    # For integer arguments the function is equivalent to the Python built-in range function, but returns an
    # ndarray rather than a list.
    np.random.shuffle(r)
    # rearanges/shuffles order of cells
    cells = data[r]
    # rearanges/shuffles order of labels
    labelz = labels[r]
    # Copy of the array, cast to a specified type.
    # Pixel values of cells range from 0 to 255
    # To scale them to range from 0 to 1 -> divide by 255
    cells = cells.astype(np.float32)/255
    labelz = labelz.astype(np.int32)
    return cells, labelz
  
c_ells, l_abels = shuffle_and_normalize(c,l)

# some ideas for different data genareation examples

#ce_o,la_o = shuffle_and_normalize(o_d, o_l)
#ce_o_44, la_o_44 = shuffle_and_normalize(o_d_44, o_l_44)
#ce_lls,la_bels = shuffle_and_normalize(ce,le)<jupyter_output><empty_output><jupyter_text>Split Data<jupyter_code># test_size: represent the proportion of the dataset to include in the test split
# random_state: random_state is the random number generator; If None, the random number generator is the RandomState 
# instance used by np.random. -> optional (default:none)
train_x, test_x, train_y, test_y = train_test_split(c_ells, l_abels,
                                         test_size = 0.2,
                                         random_state = 123)

# some ideas for different data genareation examples
#train_x, test_x, train_y, test_y = train_test_split(ce_o, la_o,
#                                         test_size = 0.2,
#                                         random_state = 123)

print('train data shape : {}\ntest data shape : {}'.format(train_x.shape,test_x.shape))
# Plot split labels
# Visualize repartition of data
objects = ('train_0', 'train_1', 'test_0', 'test_1')
y_pos = np.arange(len(objects))
a = []
b = []
for i in train_y:
    if i == 0:
        a.append(i)
    else:
        b.append(i)
        
c = []
e = []
for j in test_y:
    if j == 0:
        c.append(j)
    else:
        e.append(j)

performance = [len(a),len(b),len(c),len(e)]

plt.barh(y_pos, performance, align='center', alpha=0.5)
plt.yticks(y_pos, objects)
plt.xlabel('Number')
plt.title('Number of cell images per training and testing categorie')

plt.show()<jupyter_output><empty_output><jupyter_text>Setup CNN<jupyter_code>model_1= models.Sequential()
model_1.add(layers.Conv2D(50, (5,5), activation='relu',padding='same', input_shape=(50,50,3)))
model_1.add(layers.Conv2D(90, (3,3), activation='relu',padding='valid'))
model_1.add(layers.Conv2D(10, (5,5), activation='relu', padding='same'))
model_1.add(layers.MaxPooling2D((2,2), 2))
model_1.add(layers.Conv2D(5, (3,3), activation='relu', padding='same'))
model_1.add(layers.MaxPooling2D((2,2), 2, padding='same'))
model_1.add(layers.Flatten())
model_1.add(layers.Dense(2000, activation='relu'))
model_1.add(layers.Dense(1000, activation='relu'))
model_1.add(layers.Dense(2, activation='softmax'))
model_1.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 50, 50, 50)        3800      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 48, 48, 90)        40590     
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 48, 48, 10)        22510     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 24, 24, 10)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 24, 24, 5)         455       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 12, 12, 5)         0         
________________________________________________________[...]<jupyter_text>Train Model<jupyter_code>model_1.compile(optimizer='adam',
             loss='sparse_categorical_crossentropy',
             metrics=['accuracy'])
history = model_1.fit(train_x, train_y, epochs=5)#,
           #validation_data = (eval))
# Save entire model to a HDF5 file
model_1.save('my_model.h5')

training_loss = history.history['loss']
epoch_count = range(1, len(training_loss) + 1)
plt.plot(epoch_count, training_loss, 'b--')
plt.legend(['Training Loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.show();<jupyter_output><empty_output><jupyter_text>Evaluation and Predictions<jupyter_code>#model_1.load_weights('03_my_model.h5')
model_1.compile(optimizer='adam',
             loss='sparse_categorical_crossentropy',
             metrics=['accuracy'])
eval_result = model_1.evaluate(test_x, test_y, verbose=1)
print(eval_result)
raw_pred = model_1.predict(test_x, verbose=1)
pred = [ 0 for p in raw_pred if p[0] < p[1] else 1 ]<jupyter_output>5512/5512 [==============================] - 24s 4ms/sample<jupyter_text>Confusion Matrix<jupyter_code>#print(pred)
res = []
for p in pred:
    res.append(0 if p[0] > p[1] else 1)
    
conf_matrix = tf.math.confusion_matrix(res, test_y)
print(conf_matrix)<jupyter_output>tf.Tensor(
[[2599   88]
 [ 184 2641]], shape=(2, 2), dtype=int32)
{'fraction_non_alphanumeric': 0.12364537103932294, 'fraction_numerical': 0.032149860666735475, 'mean_word_length': 3.0626834381551364, 'pattern_counts': {'":': 1, '<': 56, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '608882', 'license': 'MIT', 'n_tokens_mistral': 6599, 'n_tokens_neox': 6083, 'n_words': 1652, 'path': 'malaria_project_custom_implementation.ipynb', 'repo_name': 'lai-la/MalariaProject', 'size': '19390'}