<jupyter_start><jupyter_text>Case 2 - Diabetic Retinopathy Classifier<NAME> Last Edited: 15.4.2018 Cognitive Systems for Health Technology Applications Helsinki Metropolia University of Applied Sciences  1. Objectives The objective of this case exercise was to train an image recognition system capable of recognizing diabetic retinopathy using a convolutional neural network trained with a dataset of approx. 2000 labeled examples of retinal images through supervised learning. The network in this case is a binary classifier, denoting retinal images as being either 'healthy' or 'unhealthy'.The background research for this case was jointly done by myself and <NAME>. It consisted of checking out competition winners for clues on how to build a network, as well as breaking the problem into small steps and reading recent studies related to convolutional neural networks to get answers for those steps. A compilation of useful links is listed under the final section of ideas on how to improve the network.The code implementation is a continuation of the template offered by our teachers at https://github.com/sakluk/cognitive-systems-for-health-technology/tree/Case-2. I merely tweaked the data preprocessing and network model.  2. Required libraries<jupyter_code># Import libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import time
import keras
import pickle
import sklearn

from keras import models
from keras import layers
from keras import optimizers
from keras import regularizers
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.applications import VGG16 

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.metrics import accuracy_score, precision_score, f1_score, confusion_matrix
from sklearn.metrics import classification_report, recall_score
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score

%matplotlib inline

# Versions used
print('numpy:', np.__version__)
print('pandas:', pd.__version__)
print('keras:', keras.__version__)<jupyter_output>numpy: 1.12.1
pandas: 0.20.3
keras: 2.1.3<jupyter_text>3. Data Generation The data used was the dataset given in the exercise (found in https://github.com/Nomikxyz/retinopathy-dataset).The accompanying 'Case_2_data_import_organization' file contains the breakdown of the dataset into train, validation and test sets (a 60/20/20 split). After the split the dataset was undersampled to bring the number of healthy examples down to the level of unhealthy ones. This was done so that the model wouldn't learn merely to call every picture it sees a healthy picture.<jupyter_code># Code, model and history filenames
my_code = 'CogSys_HealthTech_Case_2.ipynb'
model_filename = 'case_2_model.h5'
history_filename = 'case_2_history.p'

train_dir = "E:\\python_projects\\case_2_datasets\\train"
validation_dir = "E:\\python_projects\\case_2_datasets\\validation"
test_dir = "E:\\python_projects\\case_2_datasets\\test"

image_height = 200
image_width = 200 

def centering(x):
    r = x[:, :, 0]
    g = x[:, :, 1]
    b = x[:, :, 2]
    r = (r - np.mean(r))
    g = (g - np.mean(g))
    b = (b - np.mean(g))
    x[:, :, 0] = r
    x[:, :, 1] = g
    x[:, :, 2] = b
    return x

# Create datagenerators
train_datagen = ImageDataGenerator(
    rescale = 1./255,
    rotation_range = 360,
    zoom_range = 0.2,
    horizontal_flip = True,
    preprocessing_function = centering)

validation_datagen = ImageDataGenerator(
    rescale = 1./255,
    preprocessing_function = centering)

test_datagen = ImageDataGenerator(
    rescale = 1./255,
    preprocessing_function = centering)


# Generator for train dataset
train_generator = train_datagen.flow_from_directory(
    train_dir, 
    target_size = (image_height, image_width), 
    batch_size = batch_size, 
    class_mode = 'binary')

# Generator for validation dataset
validation_generator = validation_datagen.flow_from_directory(
    validation_dir, 
    target_size = (image_height, image_width), 
    batch_size = batch_size, 
    class_mode = 'binary')<jupyter_output>Found 714 images belonging to 2 classes.
Found 238 images belonging to 2 classes.<jupyter_text>4. Model Selection Starting out with a pre-trained VGG16, we add a simple linear classifier consisting of single Dense layer with 256 neurons, with weight regularization and dropout to combat overfitting, leading up to the output layer.<jupyter_code>conv_base = VGG16(
    weights='imagenet', 
    include_top=False, 
    input_shape=(image_width, image_height, 3))

model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(
    256, 
    kernel_initializer='random_normal',
    kernel_regularizer=regularizers.l2(0.001), 
    activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))

model.summary()
conv_base.trainable = False<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vgg16 (Model)                (None, 6, 6, 512)         14714688  
_________________________________________________________________
flatten_26 (Flatten)         (None, 18432)             0         
_________________________________________________________________
dense_45 (Dense)             (None, 256)               4718848   
_________________________________________________________________
dropout_7 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_46 (Dense)             (None, 1)                 257       
=================================================================
Total params: 19,433,793
Trainable params: 19,433,793
Non-trainable params: 0
________________________________________________________________[...]<jupyter_text>5. Model Compilation & Training I decided to try out the Adadelta optimizer (with default settings) as I found it converged rather quickly on an answer, cutting down training time.<jupyter_code># Training parameters
batch_size = 14
epochs = 20
steps_per_epoch = 51 
validation_steps = 5

oz = keras.optimizers.Adadelta(lr=1.0, rho=0.95, epsilon=None, decay=0.0)

model.compile(
    loss='binary_crossentropy',
    optimizer=oz,
    metrics=['acc'])

t1 = time.time()
h = model.fit_generator(
    train_generator,
    steps_per_epoch = steps_per_epoch,
    verbose = 0,
    epochs = epochs,
    validation_data = validation_generator,
    validation_steps = validation_steps)
t2 = time.time()

# Storing the elapsed time into history
h.history.update({'time_elapsed': t2 - t1})
print(' ')
print('Total elapsed time for training: {:.3f} minutes'.format((t2-t1)/60))
print(' ')
# Saving the model
fname = os.path.join('E:\\python_projects\\', 'case_2_results', model_filename)
model.save(fname)
 
# Saving the history
fname = os.path.join('E:\\python_projects\\', 'case_2_results', history_filename)
pickle.dump(h.history, open(fname, 'wb'))<jupyter_output><empty_output><jupyter_text>6. Evaluation<jupyter_code>acc = h.history['acc']
val_acc = h.history['val_acc']
loss = h.history['loss']
val_loss = h.history['val_loss']

epochs = range(len(acc))

plt.figure()

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.ylim([0, 1])
plt.xlabel('Epochs')
plt.grid()
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.ylim([0, 25])
plt.xlabel('Epochs')
plt.grid()
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>7. Results<jupyter_code># Testing the model

# Generator for test dataset
test_generator = test_datagen.flow_from_directory(
    test_dir, 
    target_size = (image_height, image_width), 
    batch_size = batch_size, 
    class_mode = 'binary')

r = model.evaluate_generator(test_generator, steps = 21)

# Loss and accuracy
r
# Predicting the score

# number of test examples
sample_count = 238
y_true = np.zeros(238)
y_score = np.zeros(238)

i = 0
for inputs_batch, labels_batch in test_generator:
        predicts_batch = model.predict(inputs_batch)
        L = labels_batch.shape[0]
        index = range(i, i + L)
        y_true[index] = labels_batch.ravel()
        y_score[index] = predicts_batch.ravel()
        i = i + L
        if i >= sample_count:
            break
# Calculating ROC and AUC

fpr, tpr, thresholds = roc_curve(y_true, y_score)
auc = roc_auc_score(y_true, y_score)

plt.figure()
plt.plot(fpr, tpr)
plt.plot([0, 1], [0, 1], '--')
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve   AUC = {:.3f}'.format(auc))
plt.show()

plt.figure()
plt.plot(thresholds, 1-fpr, label = 'specificity')
plt.plot(thresholds, tpr, label = 'sensitivity')
plt.legend()
plt.grid()
plt.xlim(0, 1)
plt.xlabel('Threshold value')
plt.show()
# Selecting the threshold to optimize for both specificity and sensitivity equally
th = 0.51

# Metrics
acc = accuracy_score(y_true, y_score > th)
prec = precision_score(y_true, y_score > th)
f1 = f1_score(y_true, y_score > th)
recall = recall_score(y_true, y_score > th)

print('Accuracy:  {:.3f}'.format(acc))
print('Precision: {:.3f}'.format(prec))
print('Recall:    {:.3f}'.format(recall))
print('F1:        {:.3f}'.format(f1))
print('Classification report')
print(classification_report(
    y_true, 
    y_score > threshold, 
    labels = [1.0, 0.0], 
    target_names = ['Disease', 'Healthy']))
tn, fp, fn, tp = confusion_matrix(y_true, y_score > th).ravel()

print('                      Confusion matrix')
print('                       True condition')
print('                      Positive Negative      Sum')
print('Predicted | Positive  {:8} {:8} {:8}'.format(tp, fp, tp + fp))
print('condition | Negative  {:8} {:8} {:8}'.format(fn, tn, fn + tn))
print('                 Sum  {:8} {:8} {:8}'.format(tp + fn, fp + tn, tp + fp + fn + tn))
print(' ')
print('Sensitivity: {:.3f}'.format(tp/(tp+fn)))
print('Specificity: {:.3f}'.format(tn/(tn+fp)))<jupyter_output>Confusion matrix
                       True condition
                      Positive Negative      Sum
Predicted | Positive        76       39      115
condition | Negative        45       78      123
                 Sum       121      117      238
 
Sensitivity: 0.628
Specificity: 0.667
{'fraction_non_alphanumeric': 0.11698326123753996, 'fraction_numerical': 0.025954485612187325, 'mean_word_length': 4.175182481751825, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '112391', 'license': 'MIT', 'n_tokens_mistral': 3535, 'n_tokens_neox': 3216, 'n_words': 948, 'path': 'CogSys_HealthTech_Case_2.ipynb', 'repo_name': 'rsykari/cognitive-systems-for-health-technology', 'size': '10645'}