<jupyter_start><jupyter_text>Detecting Pneumonia with Convolutional Neural Networks Release 1.0 - May, 2020 <NAME>  Foreword The 2020 COVID-19 outbreak has prompted for increased research on several aspects of the disease. The diagnosis of pneumonia (one of the primary complications resulting from COVID-19) through imaging techniques and the particular use of artificial intelligence as an ancillary tool has gained special traction as the disease progresses.This notebook is inspired by the current gravity of the COVID-19 pandemic. It is though not intended to serve as a medical reference in any sense. Instead, the main purpose of this exercise is to revisit traditional machine learning-based image processing and explore the level of confidence obtained in chest x-ray analysis as a pneumonia diagnosis instrument.Warning: computer vision machine learning exercises are time and resource consuming. This work was indeed developed in GPU-based runtime environments provided by Kaggle and Google Colab for code execution. Unless you have a very powerful machine to execute your code on a local host, consider a similar cloud-based approach.Logic enhancement and code forking are welcome and encouraged by the author, provided that this work is properly referenced. Thank you.  1. Introduction  1.1. The COVID-19 pandemic Corona Virus Disease 2019 (COVID-19) severely affected many countries globally over the  first half of 2020. COVID-19 is an infectious disease caused by the most recently discovered coronavirus, formally denominated 'Severe Acute Respiratory Syndrome Coronavirus 2' (SARS-CoV-2), unknown before the outbreak observed in Wuhan, China.COVID-19 is contagious in humans. SARS-CoV-2 primarily spreads among individuals through physical contact and via respiratory droplets originated from coughing or sneezing. The virus seizes human cells by binding to the angiotensin receptor converting enzyme 2 (ACE2).In humans, corona viruses are known to cause respiratory infections ranging from a common cold to more severe diseases such as the Middle East Respiratory Syndrome (MERS) and Severe Acute Respiratory Syndrome (SARS). In the case of COVID-19, usual symptoms include fever, cough, fatigue, shortness of breath, smell and taste disturbs. Illustration of a SARS-CoV-2 virion (Wikipedia) SARS-CoV-2 may spread through respiratory airways reaching the lungs, where it may lead to the development of severe viral pneumonia, identified as the final death cause in most lethal cases of COVID-19. Lung monitoring for pneumonia has gained greater importance over the COVID-19 outbreak.  1.2. Pneumonia Per Wikipedia,"Pneumonia is an inflammatory condition of the lung affecting primarily the small air sacs known as alveoli. Symptoms typically include some combination of productive or dry cough, chest pain, fever and difficulty breathing. (...) Pneumonia is usually caused by infection with viruses or bacteria and less commonly by other microorganisms, certain medications or conditions such as autoimmune diseases. Risk factors include cystic fibrosis, chronic obstructive pulmonary disease (COPD), sickle cell disease, asthma, diabetes, heart failure, a history of smoking, a poor ability to cough such as following a stroke and a weak immune system. Diagnosis is often based on symptoms and physical examination. Chest X-ray, blood tests, and culture of the sputum may help confirm the diagnosis. (...) Each year, pneumonia affects about 450 million people globally (7% of the population) and results in about 4 million deaths."Along with physical examination, imaging diagnosis plays a central role in the detection of pneumonia. Chest radiographs are frequently used in diagnosis procedures and represent a fast, cost-effective alternative to map the nature, features and extension of lung inflammations. X-ray radiograph image opacity areas are commonly correlated to pneumonia affected regions.  Normal (left) and abnormal (right) chest radiographs - Increased opacity suggests a pneumonia occurrence The identification of opacities in chest x-ray images is sometimes challenging, even for experienced practitioners. Artificial intelligence and computer vision techniques have been extensively used over the past few years as ancillary tools in pneumonia diagnosis.  2. Objectives of this deep learning exercise The primary goal of this exercise is educational. The role of computer vision  deep learning techniques - Convolutional Neural Networks (CNNs), in particular - is assessed in detail herein. Special attention is given to particular aspects of a CNN-based solution to chest radiograph opacity interpretation, including the choice and tuning of critical hyperparameters and the relevance of dataset augmentation procedures.  3. The dataset The dataset chosen for this deep learning exercise is a modified version of that submitted by <NAME> at Kaggle, 'Chest X-Ray Images (Pneumonia)'. Per the dataset author, "(...) Chest X-ray images (anterior-posterior) were selected from retrospective cohorts of pediatric patients of one to five years old from Guangzhou Women and Children’s Medical Center, Guangzhou. All chest X-ray imaging was performed as part of patients’ routine clinical care. (...)The original dataset contains 5,863 observations, split into training (1,341 normal cases, 3,875 pneumonia cases), validation (8 normal cases, 8 pneumonia cases) and testing (234 normal cases, 390 pneumonia cases) folders.The modified dataset contains the very same 5,863 observations, where a more balanced split between training  and validation images is proposed:    Training observations: 4,192 (1,082 normal cases, 3,110 lung opacity cases);    Validation observations: 1,040 (267 normal cases, 773 lung opacity cases);    Testing observations: 624 (234 normal cases, 390 lung opacity cases).This dataset is particularly interesting as:    it addresses a specific segment (children) critically susceptible to pneumonia;    the number of observations, though not small, is also not large, allowing for the exploration of augmentation techniques;    unlike other similar datasets, it contains images of several, non-standardized resolutions.  4. Initial setup  4.1. Importing required libraries Along with traditional libraries imported for tensor manipulation, mathematical operations and graphics, one scikit-learn module (confusion_matrix for performance metric assessment) and specific Keras image preprocessing (ImageDataGenerator) and deep learning objects (Sequential, Conv2D, MaxPooling2D, Flatten, Dense) are used in this exercise.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image

from sklearn.metrics import confusion_matrix

from keras.preprocessing.image import ImageDataGenerator, array_to_img
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout
import warnings
from datetime import datetime

warnings.filterwarnings('ignore')
start_time = datetime.now()<jupyter_output><empty_output><jupyter_text>4.2. Critical hyperparameters Machine learning hyperparameters are parameters whose value is set before the learning process starts. Several rounds of this exercise showed that specific hyperparameters play an important role when it comes to final prediction accuracy and execution runtime, reason why they have been segregated in the code session below:    'hyper_dimension': target image width and length in pixels considered when original images need to be rescaled for processing;    'hyper_epochs': number of epochs (leaning iterations through which the whole dataset is exposed to the machine for weight updates);    'hyper_batch_size': size of image batches;    'hyper_feature_maps': reference number of feature maps generated by convolutional layers;    'hyper_channels' and 'hyper_mode': number of channels utilized in the learning process. For colored RGB images, hyper_channels = 3 and hyper_mode = 'rgb', yet for grayscale images hyper_channels = 1 and hyper_mode = 'grayscale'.Specific considerations on hyperparameters are provided through the exercise.<jupyter_code>hyper_dimension = 500
hyper_epochs = 100
hyper_batch_size = 16
hyper_feature_maps = 32
hyper_channels = 1
hyper_mode = 'grayscale'<jupyter_output><empty_output><jupyter_text>5. Deep Learning - Convolutional Neural Network  5.1. Creating and compiling the deep learning model In this exercise a traditional convolutional neural network aimed at image processing and interpretation is adopted. The architecture, a Keras sequential model graphically depicted below, comprises:    Three pairs of convolution / pooling layers: first and second produce 32 feature maps processed by subsequent 2x2 pooling engines generating 32 pooled maps, while the third produces 64 feature maps also processed by a subsequent 2x2 pooling engine generating a 64 pooled map. Convolution is performed with 3x3xk kernels, where k = 1 for grayscale images and k = 3 for RGB images. The input shape is a 500x500 pixels image (1 or 3 channels, as described above). 'ReLU' (Rectifier Linear Unit) is the activation function of choice;    A flattening layer, responsible to provide the subsequent artificial neural network (ANN) with an flattened, unidimensional tensor with the output of the last pooling layer;    A full connection pair of layers with 64 and 1 neuron, respectively, responsible to perform binary classification ('normal' or 'pneumonia').  The model is compiled using 'adam' (stochastic gradient descent method based on adaptive estimation of first-order and second-order moments) as the optimization function. Since this is a classification problem ('normal' versus 'pneumonia' cases), 'binary_crossentropy' is elected as the loss function. Performance will be assessed via the 'accuracy' metric.<jupyter_code># Initializing the CNN
classifier = Sequential()

# Convolution & pooling - First convolution layer
classifier.add(Conv2D(hyper_feature_maps, (3, 3),
                      input_shape = (hyper_dimension,
                                     hyper_dimension,
                                     hyper_channels),
                      activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))

# Convolution & pooling - Second convolution layer (same as first layer)
classifier.add(Conv2D(hyper_feature_maps, (3, 3),
                      input_shape = (hyper_dimension,
                                     hyper_dimension,
                                     hyper_channels),
                      activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))

# Convolution & pooling - Third convolution layer
classifier.add(Conv2D(hyper_feature_maps * 2, (3, 3),
                      input_shape = (hyper_dimension,
                                     hyper_dimension,
                                     hyper_channels),
                      activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))

# Flattening
classifier.add(Flatten())

# Full connection
classifier.add(Dense(units = hyper_feature_maps * 2, activation = 'relu'))
classifier.add(Dense(units = 1, activation = 'sigmoid'))

# Compiling the CNN
classifier.compile(optimizer = 'adam',
                   loss = 'binary_crossentropy',
                   metrics = ['accuracy'])<jupyter_output><empty_output><jupyter_text>5.2. Creating training and validation sets through image flowing with augmentation Augmentation is an important tool to mitigate the risk of overfitting in machine learning exercises. When it comes to images, augmentation techniques provide modified versions of the original images through a series of specific procedures. As the chosen dataset is not as large as desired for an image processing experience, augmentation can fill this blank and help address the risk of overfitting.Keras 'ImageDataGenerator' object (class) is used in this exercise as the primary tool to generate augmented training and validation image sets. Two instances of this object are created:    'train_gen' will be utilized to generate the training set. As the original images stored in folders (more ahead) correspond to 500x500 pixel matrices of real values ranging from 0 to 255, rescaling is performed through the 'rescale' parameter. Also, augmentation is achieved with some image shearing, zooming and vertical flips.    'val_gen' will be utilized to generate the validation set. Only rescaling is performed in this case, as we intend to validate the learning process making use of pure images with no manipulation.The 'flow_from_directory' method allows for pulling from specific folders images for training and validation, resizing them, grouping them in batches and specifying the machine objective (a binary classification, in this case) and the color mode of choice. The use of 'flow_from_directory' assumes that the data is properly stored in appropriate folders with a predetermined structure:    train            normal        pneumonia        val            normal        pneumonia<jupyter_code># Creating training and validation image flows

train_gen = ImageDataGenerator(rescale = 1./255,
                               shear_range = 0.2,
                               zoom_range = 0.2,
                               vertical_flip = True)

val_gen = ImageDataGenerator(rescale = 1./255)

train_set = train_gen.flow_from_directory('../input/pneumonia-xray-images/train',
                                          target_size = (hyper_dimension,
                                                         hyper_dimension),
                                          batch_size = hyper_batch_size,
                                          class_mode = 'binary',
                                          color_mode = hyper_mode)

val_set = val_gen.flow_from_directory('../input/pneumonia-xray-images/val',
                                      target_size = (hyper_dimension,
                                                     hyper_dimension),
                                      batch_size = hyper_batch_size,
                                      class_mode = 'binary',
                                      color_mode = hyper_mode)<jupyter_output>Found 4192 images belonging to 2 classes.
Found 1040 images belonging to 2 classes.<jupyter_text>5.3. Image visualization prior to fitting Once training and testing sets are generated, it is important to investigate how images look like.Understanding the structure of the training and validation sets is important to manipulate them. Let's take the training set as an example.    'train_set[0]' corresponds to the first set of image batches flowed from the source folder;    'train_set[0][0]' corresponds to the first batch of 16 images included in the first set of batches;    Each of the 16 images in this first batch can be accessed via the 'array_to_img' method and properly plotted.The analysis of the first batch in the training set shows that augmentation has been successfully performed. Some images have been slightly shared, zoomed in and/or flipped vertically. Also, it is evident that we are now dealing with grayscale images, as expected.<jupyter_code># Visualizing images in the first training set batch 

image_batch = train_set[0][0]

plt.figure(figsize=(20,5))
for i in range(len(image_batch)):
    plt.subplot(2,8,i+1)
    pil_img = array_to_img(image_batch[i])
    plt.imshow(pil_img,cmap='gray')
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>5.4. Fitting the model As we are dealing with a relatively large dataset and augmentation has been implemented, 'fit_generator' is our method of choice for fitting purposes. (Note: 'fit' also supports data augmentation from TensorFlow 2.2.0, so it could also be used in this exercise. As the code may be forked by coders not equipped with TF 2.2.0, the original method, 'fit_generator', has been maintained herein.)The model will be trained in 100 epochs. Comments on this choice will be provided ahead.<jupyter_code># Fitting the model

results = classifier.fit_generator(train_set,
                                   steps_per_epoch = len(train_set),
                                   epochs = hyper_epochs,
                                   validation_data = val_set,
                                   validation_steps = len(val_set))<jupyter_output>Epoch 1/100
262/262 [==============================] - 225s 860ms/step - loss: 0.3998 - accuracy: 0.8292 - val_loss: 0.2235 - val_accuracy: 0.9144
Epoch 2/100
262/262 [==============================] - 197s 752ms/step - loss: 0.2256 - accuracy: 0.9115 - val_loss: 0.2355 - val_accuracy: 0.9298
Epoch 3/100
262/262 [==============================] - 199s 760ms/step - loss: 0.2021 - accuracy: 0.9220 - val_loss: 0.1959 - val_accuracy: 0.9471
Epoch 4/100
262/262 [==============================] - 196s 748ms/step - loss: 0.2000 - accuracy: 0.9187 - val_loss: 0.0635 - val_accuracy: 0.9413
Epoch 5/100
262/262 [==============================] - 199s 758ms/step - loss: 0.1773 - accuracy: 0.9363 - val_loss: 0.4701 - val_accuracy: 0.9510
Epoch 6/100
262/262 [==============================] - 202s 771ms/step - loss: 0.1720 - accuracy: 0.9339 - val_loss: 0.3772 - val_accuracy: 0.9500
Epoch 7/100
262/262 [==============================] - 201s 767ms/step - loss: 0.1613 - accuracy: 0.9408 - val_loss: 0[...]<jupyter_text>5.5. Predicting pneumonia The test set must be generated so that predictions on it can be made. The same 'flow_from_directory' method is applied, as images are segregated in different folders for different diagnosis ('normal' and 'pneumonia').As predictions will be confronted with the actual test image labels for performance assessment, it is assured that no shuffling will be executed. Also, test images will be flown from the corresponding folders individually (batches of 1 image each).The model will provide for each image a scalar output within the range [0,1] corresponding to the probability a given image represents a pneumonia case. In other words, the higher the output, the higher the probability of a pneumonia occurrence.As we intend to quantify true positive, true negative, false positive and false negative cases and visualize them in a confusion matrix, scalar results will be segregated in two subsets (below and above the 0.5 threshold), corresponding to 'normal' and 'pneumonia' diagnosis, respectively.<jupyter_code># Creating test image flow

test_gen = ImageDataGenerator(rescale = 1./255)

test_set = test_gen.flow_from_directory('../input/pneumonia-xray-images/test',
                                        target_size = (hyper_dimension,
                                                       hyper_dimension),
                                        batch_size = 1,
                                        class_mode = None,
                                        color_mode = hyper_mode,
                                        shuffle=False)

# Making predictions

predictions = classifier.predict_generator(test_set)
predictions[predictions <= 0.5] = 0
predictions[predictions > 0.5] = 1<jupyter_output>Found 624 images belonging to 2 classes.<jupyter_text>6. Results The segregation described in Section 5.5 allows for constructing a confusion matrix and calculating the overall prediction accuracy from it.<jupyter_code># Confusion matrix

cm = pd.DataFrame(data=confusion_matrix(test_set.classes, predictions, labels=[0, 1]),
                  index=["Actual Normal", "Actual Pneumonia"],
                  columns=["Predicted Normal", "Predicted Pneumonia"])
cm<jupyter_output><empty_output><jupyter_text>7. Discussion and Final Remarks Results show that very low losses and high accuracies (exceeding 96% in validation) are obtained with a traditional CNN approach, without any refinement based, for example, on the use of pretrained image classification models.In addition, when we deal with medical diagnosis, a false positive (i.e. predicting illness when the patient is healthy) is less critical than a false negative (predicting healthiness when the patient is sick). The number of false negatives obtained with the CNN presented here is extremely low, which positions the machine developed here as a reliable ancillary tool for pneumonia detection.Specific considerations on the model and the hyperparameters must be made:    As mentioned above, the approach presented here is very traditional. The CNN architecture adopted here follows those Yann LeCun (recognized as founding father of convolutional networks and their use in computer vision) promoted in the 1990s for image classification. No pretrained model was used here, and their employment could be taken as a suggestion for further improvements;    The model was trained several times with a variety of hyperparameter combinations. The final version depicted here was the one through which the best results were obtained. On hyperparameters:            Tested numbers of epochs were 10, 25, 50 and 100. Reasonable results (low losses and high accuracy on both validation AND testing) were achieved only with 50 or 100 epochs. The final training was performed over 100 epochs;        The model was fed with 400x400, 500x500 and 600x600 pixels images. Best performance was achieved with 500x500 resized input images;        Batches of 16 images proved to be the best choice over 8, 32 and 64 image alternatives. It must be emphasized though that much faster runtime could be achieved with the use of TPUs, case in which larger batches would be more appropriate. There is an ongoing discussion on how TPUs handle floating scalars and how large batches should be to extract from TPUs their maximum performance. As this exercise made use of GPUs only, this topic was not explored.        Finally, the model was trained with both 3-channel (RBG) and 1-channel (grayscale) versions of input images (which are originally 3-channel samples). As a higher number of channels carries in theory a larger amount of information, it was initially expected that 3-channel image trainings would lead to better machines. However, this was not observed - grayscale-based exercises led to higher accuracies and lower losses. Further investigation could be made, but in a first assessment of such results, considering that in pneumonia detection cases we look for opacities in the lungs, those might be more evidenced in grayscale samples than in colored ones. For now this is just an assumption demanding further analysis.<jupyter_code>end_time = datetime.now()

print('\nStart time', start_time)
print('End time', end_time)
print('Time elapsed', end_time - start_time)<jupyter_output>Start time 2020-10-09 14:04:39.194064
End time 2020-10-09 19:27:31.072125
Time elapsed 5:22:51.878061
{'fraction_non_alphanumeric': 0.06086956521739131, 'fraction_numerical': 0.02753623188405797, 'mean_word_length': 3.780810413604871, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 33, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '384293', 'license': 'MIT', 'n_tokens_mistral': 5934, 'n_tokens_neox': 5146, 'n_words': 2730, 'path': 'code/ipynb/detecting-pneumonia.ipynb', 'repo_name': 'pcbreviglieri/data-science-detecting-pneumonia', 'size': '22786'}