<jupyter_start><jupyter_text>Malaria identification using Convolutional Neural Networks (CNN)  Necessary imports<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.image import imread
import os

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Conv2D,MaxPool2D,Flatten,Dropout
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.metrics import classification_report,confusion_matrix
from tensorflow.keras.preprocessing import image


import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Data Visualization and Pre-processing<jupyter_code># path of the directory where the images are stored (would change depending on where one stores the images)
data_dir='E:\DS_ML\cell_images'
# lists all the folders in that directory
os.listdir(data_dir)
# path for the train data
train_path=data_dir+'\\train'

# path for the test data
test_path=data_dir+'\\test'
os.listdir(train_path)
os.listdir(test_path)<jupyter_output><empty_output><jupyter_text>**train and test folders are divided into two categories, "parasitized" and "uninfected"**<jupyter_code>print("The number of uninfected images in the train set: ",len(os.listdir(train_path+'\\uninfected')))
print("The number of infected images in the train set: ",len(os.listdir(train_path+'\\parasitized')))
print("The number of uninfected images in the test set: ",len(os.listdir(test_path+'\\uninfected')))
print("The number of infected images in the test set: ",len(os.listdir(test_path+'\\parasitized')))<jupyter_output>The number of uninfected images in the train set:  12479
The number of infected images in the train set:  12479
The number of uninfected images in the test set:  1300
The number of infected images in the test set:  1300<jupyter_text>**The dataset is balanced with equal number of infected and uninfected cells in the train and test directories.**<jupyter_code># fetching the first image from the parasitized category
img_infected=os.listdir(train_path+'\\parasitized')[0]
img_infected
# visualizing the infected image
para_cell=train_path+'\\parasitized\\'+img_infected
para_img=imread(para_cell)
plt.imshow(para_img)
# shape of the image
para_img.shape<jupyter_output><empty_output><jupyter_text>**The shape of the image is of the form (height, width, number of color channels). The number of color channels for grey scale (black/white) images is 1 whereas that for colored (RGB) images is 3.**<jupyter_code># fetching the first image from the uninfected category
img_uninfected=os.listdir(train_path+'\\uninfected')[0]
img_uninfected
# visualizing the uninfected image
un_cell=train_path+'\\uninfected\\'+img_uninfected
uninfected_img=imread(un_cell)
plt.imshow(uninfected_img)
# shape of the image
uninfected_img.shape<jupyter_output><empty_output><jupyter_text>**We can clearly see a difference between the image of an infected cell and that of an uninfected cell. Now we need to train a model which would see an image and based on the difference classify it as infected or uninfected.**<jupyter_code># We would find the average dimension of the images so that we can scale all the images to the average dimensions

# lists for storing the dimensions
dim1=[]
dim2=[]

for img_file in os.listdir(train_path+'\\uninfected'):
    img=imread(train_path+'\\uninfected\\'+img_file)   # reading images one by one
    d1,d2,col=img.shape
    dim1.append(d1)
    dim2.append(d2)
# plotting the dimensions
sns.jointplot(dim1,dim2)
np.mean(dim1), np.mean(dim2)<jupyter_output><empty_output><jupyter_text>**The average size of the images is about 130 x 130 pixels. So we would reshape all images to the mean dimensions for the deep learning model.**<jupyter_code># The dimension of an image is of the form (width, height, number of color channels)
img_shape=(130,130,3)<jupyter_output><empty_output><jupyter_text>Image Manipulation There are about 25000 images in the training set and 2600 images in the test set, which is too much data to load into the model at a particular time. To handle large image data we can use the built in libraries provided by keras which can manipulate the data, process the data and generate a batch flow from the directory.  **ImageDataGenerator** provided by keras does exactly the same task specified above. It takes in a whole range of parameters such as width_shift_range (changing the width of the image), height_shift_range (changing the height of the image), rescaling etc. This way we can manipulate the images and pass it to our model making it more robust to all the different kinds of possible images which our not in our dataset. In order to use .flow_from_directory, you must organize the images in sub-directories. This is an absolute requirement, otherwise the method won't work. The directories should only contain images of one class, so one folder per class of images.Structure Needed:* Image Data Folder    * Class 1        * 0.jpg        * 1.jpg        * ...    * Class 2        * 0.jpg        * 1.jpg        * ...    * ...    * Class n<jupyter_code># uncomment the following command to get a full documentation for ImageDataGenerator

# help(ImageDataGenerator)
img_gen = ImageDataGenerator(rotation_range=20,      # rotate the image by a max of 20 degrees
                               width_shift_range=0.10, # Shift the pic width by a max of 5%
                               height_shift_range=0.10, # Shift the pic height by a max of 5%
                               rescale=1/255,        # Rescale the image by normalzing it.
                               shear_range=0.1,       # Shear means cutting away part of the image (max 10%)
                               zoom_range=0.1,        # Zoom in by 10% max
                               horizontal_flip=True,   # Allow horizontal flipping
                               fill_mode='nearest',     # Fill in missing pixels with the nearest filled value
                               vertical_flip=True
                              )
# original image
plt.imshow(para_img)
# image after transforming using the ImageDataGenerator
plt.imshow(img_gen.random_transform(para_img))
# creating a flow from the training directory
img_gen.flow_from_directory(train_path)
# creating a flow from the test directory
img_gen.flow_from_directory(test_path)<jupyter_output>Found 2600 images belonging to 2 classes.<jupyter_text>Model Creation We would be creating a sequential model and adding the convolution and pooling layers<jupyter_code>model=Sequential()

model.add(Conv2D(filters=32,kernel_size=(3,3),input_shape=img_shape,activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(filters=64,kernel_size=(3,3),input_shape=img_shape,activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(filters=128,kernel_size=(3,3),input_shape=img_shape,activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Flatten())

model.add(Dense(128,activation='relu'))
model.add(Dropout(0.5))

# final output layer 
model.add(Dense(1,activation='sigmoid'))

model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
# stopping the training if the val_loss increases which means we are overfitting the data
early_stop=EarlyStopping(monitor='val_loss',mode='min',patience=2)
# the size of the batch in which images flow from the directory
batch_size=32
train_img_gen=img_gen.flow_from_directory(directory=train_path,
                                          target_size=img_shape[:2],   # only requires the height and width
                                          color_mode='rgb',
                                          batch_size=batch_size,
                                          class_mode='binary')
test_img_gen=img_gen.flow_from_directory(directory=test_path,
                                         target_size=img_shape[:2],
                                         color_mode='rgb',
                                         batch_size=batch_size,
                                         class_mode='binary',
                                         shuffle=False)   

# we are not shuffling the test set because it would shuffle the labels (uninfected/infected) as well
# classes we are predicting
train_img_gen.class_indices<jupyter_output><empty_output><jupyter_text>**Training the model**<jupyter_code># fitting the model on the training data
results=model.fit(train_img_gen,epochs=20,callbacks=[early_stop],validation_data=test_img_gen)
# saving the model 
model.save('malaria_detection.h5')
# stores the metrics of the model (accuracy and losses)
metrics=pd.DataFrame(model.history.history)
metrics
# plot of loss and val_loss
metrics[['loss','val_loss']].plot()
# plot of accuracy and validation accuracy
metrics[['accuracy','val_accuracy']].plot()
# making predictions on the test set
pred_prob=model.predict(test_img_gen)
pred_prob
# since a value greater than 0.5 would mean 1 and less than 0.5 would mean 0
predictions=pred_prob > 0.5
predictions
# the classes of the test data
test_img_gen.classes
print(classification_report(test_img_gen.classes,predictions))
print(confusion_matrix(test_img_gen.classes,predictions))<jupyter_output>[[1217   83]
 [  45 1255]]<jupyter_text>**Checking our prediction for a particular image**<jupyter_code># loading the image of the infected cell
my_image=image.load_img(para_cell,target_size=img_shape)
# converting it into an array
my_image=image.img_to_array(my_image)
my_image.shape
# expanding dimensions so that it is of the form (batch_size, height, width, number of color channels)
my_image=np.expand_dims(my_image,axis=0)
my_image.shape
model.predict(my_image)>0.5
train_img_gen.class_indices<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0821472143147621, 'fraction_numerical': 0.015758438389589263, 'mean_word_length': 4.136814621409922, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '516667', 'license': 'MIT', 'n_tokens_mistral': 2987, 'n_tokens_neox': 2817, 'n_words': 924, 'path': 'Malaria-Identification.ipynb', 'repo_name': 'dhwanilms/Malaria-Diagnosis-using-CNN', 'size': '9836'}