<jupyter_start><jupyter_text>Deep Learning Project  The current project relies on using CNNs in order to process sperm data.The goal is to develop a classifier able to identify different morphologies of head spearm cells  Context :Due to our background in the fields of biology and health, it was in our interest to use this project,  to develop our knowledge of neural networks, in microscopic images.The chosen theme is a classification problem taking into account the morphology of the sperm cell heads. There are two datasets available, which we have chosen to aggregate into one.- SCIAN - MorphoSpermGS  - https://cimt.uchile.cl/gold10/ -  with 1132 image, each image has 35 x 35 pixels and has been classified by 3 experts. We will use majority vote result as target. - HuSHeM - Human Sperm head Morphology - https://easy.dans.knaw.nl/ui/datasets/id/easy-dataset:77214/tab/2 -  with 725 images, each image has 576×720 pixels and have the respective classification   The sperm cells heads will have the following classification: ![Title](Heads.png)* We don't include the "small" class from HuSHemRef: https://doi.org/10.1016/j.compbiomed.2019.103342 -> The process for adapting the data to use on CNN, for reasons of OS compatibility, has been removed. But can be found in [this document.][mylink][mylink]: DL_Project_LoadImg.html  In this pipeline, 4 models were developed:- The first model was produced by us, based on heuristics.- The second model was developed based on the Alex Net network architecture- The third model was based on the Resnet50 model.- The last model was based on an Efficient Net which, according to the current literature, has very promising results.<jupyter_code>'''#If you are able to run keras with GPU..

import os
os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices'
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
#
#Check Computer's available devices
#Will need to check in the future
import tensorflow as tf
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

#cfg 1
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)

    except RuntimeError as e:
        print(e)

#cfg2
config = tf.compat.v1.ConfigProto(gpu_options = tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8))
# device_count = {'GPU': 1})
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)
tf.compat.v1.keras.backend.set_session(session)'''
#base libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import random
import os
import matplotlib.image as mpimg
%matplotlib inline
from timeit import default_timer as timer
from tensorflow import keras


#image manipulation packages
import cv2
import os
from tqdm import tqdm
from glob import glob
from sklearn.metrics import confusion_matrix,classification_report


#Classification
from sklearn.metrics import classification_report, confusion_matrix, multilabel_confusion_matrix

sns.set()

import glob
import cv2
import shutil
import random
import Augmentor

#Data Augmentation
from keras.preprocessing.image import ImageDataGenerator

# Callbacks
from tensorflow.keras import callbacks
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.callbacks import ReduceLROnPlateau


import keras
from tensorflow.keras import models
from keras.models import Sequential
from tensorflow.keras import layers
from keras.layers import Dense, Activation, Dropout, Flatten, Conv2D, MaxPooling2D
from keras.layers.normalization import BatchNormalization



## RESNET50

from keras.applications.resnet50 import preprocess_input
from tensorflow.keras.applications.resnet50 import ResNet50


### Efficient
from tensorflow.keras.applications import EfficientNetB7<jupyter_output><empty_output><jupyter_text>1- Import Images and DataframeIn this case, we have a folder with 1132 image - SCIAN-MorphoSpermGS folder - https://cimt.uchile.cl/gold10/. Each image is 35 x 35 pixels and has been classified by 3 experts. We will use majority vote result as target. Each image will need to be loaded and the dataset will need to be created. This will yield a dataset with the picture name and the label.<jupyter_code>dataframe=pd.read_csv("Majority_Vote.csv")
os.chdir('path')
path = os.getcwd()
files = os.listdir(path)
path
#Count the number of images by class

def fileCount(folder):
    "count the number of files in a directory"

    count = 0

    for filename in os.listdir(folder):
        path = os.path.join(folder, filename)

        if os.path.isfile(path):
            count += 1
        elif os.path.isdir(path):
            count += fileCount(path)

    return count

count_0 = fileCount(path+'/class0')
count_1 = fileCount(path+'/class1')
count_2 = fileCount(path+'/class2')
count_3 = fileCount(path+'/class3')

print(f'Images Normal: {count_0}\n' +
     f'Images Tapered: {count_1}\n' +
     f'Images Pyriform: {count_2}\n' +
     f'Images Amorphous: {count_3}\n')
#ONLY WORKS IF TRAIN; TEST AND VAL FOLDERS ARE NOT IN PATH FOLDER

classes_dir = ['/class0', '/class1', '/class2', '/class3']

train_ratio = 0.6
val_ratio = 0.3
test_ratio = 0.1

if not os.path.isdir(path + '/train'):

    for cls in classes_dir:
    
        #creates train and test folders, with each class separated inside
        os.makedirs(path +'/train' + cls)
        os.makedirs(path +'/val' + cls)
        os.makedirs(path +'/test' + cls)


        # Creating partitions of the data after shuffeling
        src = path + cls # Folder to copy images from

        allFileNames = os.listdir(src)
        random.shuffle(allFileNames)
        
        train_FileNames, val_FileNames_test_FileNames = np.split(np.array(allFileNames),
                                                              [int(len(allFileNames)* (train_ratio))])
        
        
        random.shuffle(val_FileNames_test_FileNames)
        val_FileNames, test_FileNames = np.split(np.array(val_FileNames_test_FileNames),
                                                              [int(len(val_FileNames_test_FileNames)* 0.75)])
        
        train_FileNames = [src+'/'+ name for name in train_FileNames.tolist()]
        val_FileNames = [src+'/' + name for name in val_FileNames.tolist()]
        test_FileNames = [src+'/' + name for name in test_FileNames.tolist()]

    
        # Copy-pasting images
        for name in train_FileNames:
            shutil.copy(name, path +'/train' + cls)

        for name in val_FileNames:
            shutil.copy(name, path +'/val' + cls)

        for name in test_FileNames:
            shutil.copy(name, path +'/test' + cls)<jupyter_output><empty_output><jupyter_text>2- Preprocessing   2.1 DATA AUGMENTATION   2.1 Offline Data AugmentationOptions are limited since we're working with microscopy data<jupyter_code>#counts number of images in training data

train_count_0 = fileCount(path+'/train/class0')
train_count_1 = fileCount(path+'/train/class1')
train_count_2 = fileCount(path+'/train/class2')
train_count_3 = fileCount(path+'/train/class3')

print(f'Images Normal: {train_count_0}\n' +
     f'Images Tapered: {train_count_1}\n' +
     f'Images Pyriform: {train_count_2}\n' +
     f'Images Amorphous: {train_count_3}\n')
# Define augmentation pipelines
class_0 = Augmentor.Pipeline(path+'/train/class0')
class_1 = Augmentor.Pipeline(path+'/train/class1')
class_2 = Augmentor.Pipeline(path+'/train/class2')
class_3 = Augmentor.Pipeline(path+'/train/class3')


#rotate by a maximum of 90 degrees
class_0.rotate(probability=0.7, max_left_rotation=25, max_right_rotation=25)
class_1.rotate(probability=0.7, max_left_rotation=25, max_right_rotation=25)
class_2.rotate(probability=0.7, max_left_rotation=25, max_right_rotation=25)
class_3.rotate(probability=0.7, max_left_rotation=25, max_right_rotation=25)

#mirroring, vertical or horizontal, randomly
class_0.flip_random(probability=0.7)
class_1.flip_random(probability=0.7)
class_2.flip_random(probability=0.7)
class_3.flip_random(probability=0.7)


# Augment images to the same proportion as existing ones in class 4 (majority class - get to 1000 in each class)
class_0.sample(6400 - train_count_0)
class_1.sample(6400 - train_count_1)
class_2.sample(6400 - train_count_2)
class_3.sample(6400 - train_count_3)<jupyter_output>Initialised with 92 image(s) found.
Output directory set to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells\path/train/class0\output.Initialised with 168 image(s) found.
Output directory set to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells\path/train/class1\output.Initialised with 79 image(s) found.
Output directory set to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells\path/train/class2\output.<jupyter_text>2.2 Define Callbacks- Checkpoint- Early Stopping- Reduce Learning Rate- Timer<jupyter_code>#reset path to main folder
os.chdir('../')
path = os.getcwd()
checkpoint_filepath = path+'/Models/'

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5, min_lr=0, verbose = 1)

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=20, restore_best_weights = True)

mc = ModelCheckpoint(
    filepath=checkpoint_filepath,
    verbose = 1,
    monitor='val_accuracy',
    mode='max',
    save_best_only=True)


class TimingCallback(keras.callbacks.Callback):
    def __init__(self, logs={}):
        self.logs=[]
    def on_epoch_begin(self, epoch, logs={}):
        self.starttime = timer()
    def on_epoch_end(self, epoch, logs={}):
        self.logs.append(timer()-self.starttime)

cb = TimingCallback()<jupyter_output><empty_output><jupyter_text>Models:- First Model- Alex Net  ---- > Alex Net [Model Assessment.][mylink]- ResNet50- EfficientNet B7[mylink]: Alex_Model_Assessment_RGB.ipynb  3. First Model<jupyter_code>checkpoint_filepath = path+'/Models/model_HM.h5'<jupyter_output><empty_output><jupyter_text>3.1 Preprocess inputOnline Data Augmentation  in GrayScale<jupyter_code>cnn_train_datagen = ImageDataGenerator(rescale = 1./255,
                                   rotation_range = 5,
                                   width_shift_range = 0.1,
                                   height_shift_range = 0.1, 
                                   vertical_flip = True,
                                   horizontal_flip = True,
                                   brightness_range=[0.2,1.5], 
                                   fill_mode='nearest',
                                   zoom_range = 0.2,
                                   ) 

cnn_val_datagen = ImageDataGenerator(rescale = 1./255)
cnn_test_datagen = ImageDataGenerator(rescale = 1./255)

#test different color maps -  class modes and cross validation types
cnn_training = cnn_train_datagen.flow_from_directory(path+'/path/train',
                                                 target_size = (32, 32),
                                                 batch_size = 64,
                                                 shuffle = True,
                                                 class_mode="categorical",
                                                 color_mode = 'grayscale')

cnn_val = cnn_val_datagen.flow_from_directory(path+'/path/val',
                                            target_size = (32, 32),
                                            batch_size = 64,
                                            shuffle = True,
                                            class_mode="categorical",
                                            color_mode = 'grayscale')

cnn_test = cnn_test_datagen.flow_from_directory(path+'/path/test',
                                            target_size = (32, 32),
                                            batch_size = 1,
                                            shuffle = True,
                                            class_mode="categorical",
                                            color_mode = 'grayscale')<jupyter_output>Found 25600 images belonging to 4 classes.
Found 383 images belonging to 4 classes.
Found 130 images belonging to 4 classes.<jupyter_text>3.2 Build model<jupyter_code>#creating a model that will take inputs 32,32,1
cnn_model = Sequential()

#convolutional layer with 32 2x2 filters - output 14 x 14 active area
cnn_model.add(Conv2D(filters=32, kernel_size=(4,4), padding = 'same', activation='relu')) 
cnn_model.add(BatchNormalization())
        
#2nd convolution with 64 filters  
cnn_model.add(Conv2D(filters=64, kernel_size=(2,2), padding = 'same', activation= 'relu'))
cnn_model.add(BatchNormalization())  

#MaxPooling - takes the max value of each 2x2 pool in the feature map
cnn_model.add(MaxPooling2D(pool_size=(2,2)))
cnn_model.add(BatchNormalization())
      
#the result of kthe CNN is then flattened and placed into the 
cnn_model.add(Flatten())


cnn_model.add(Dense(576, activation='relu'))
cnn_model.add(BatchNormalization())

#widen layer
cnn_model.add(Dense(1024, activation='relu'))
cnn_model.add(BatchNormalization())
cnn_model.add(Dropout(0.4))


#shrink layer
cnn_model.add(Dense(256, activation='relu'))
cnn_model.add(BatchNormalization())
#Add Dropout
cnn_model.add(Dropout(0.4))

#final layer, is output, 1 out of 5 possible results
#0 Normal, 1 Tapered, 2 Pyriform, 3 Amorphous
cnn_model.add(Dense(4))
cnn_model.add(BatchNormalization())            
cnn_model.add(Activation('softmax'))<jupyter_output><empty_output><jupyter_text>3.3 Compile model<jupyter_code>cnn_model.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-3),
              loss='categorical_crossentropy',
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>3.4 Fit model<jupyter_code>history = cnn_model.fit(cnn_training,
        epochs=200, 
        validation_data=cnn_val,
        verbose = 1, 
        callbacks = [mc, reduce_lr, es,cb])<jupyter_output>Epoch 1/200
400/400 [==============================] - 126s 310ms/step - loss: 1.3916 - accuracy: 0.2991 - val_loss: 1.6001 - val_accuracy: 0.1462

Epoch 00001: val_accuracy improved from -inf to 0.14621, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\
INFO:tensorflow:Assets written to: C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\assets
Epoch 2/200
400/400 [==============================] - 19s 47ms/step - loss: 1.2605 - accuracy: 0.4101 - val_loss: 1.4010 - val_accuracy: 0.2559

Epoch 00002: val_accuracy improved from 0.14621 to 0.25587, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\
INFO:tensorflow:Assets written to: C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Model[...]<jupyter_text>3.5 Evaluate model<jupyter_code>_, train_acc = cnn_model.evaluate(cnn_training, verbose=1)
_, val_acc = cnn_model.evaluate(cnn_val, verbose=1)
print('Train: %.3f, Val: %.3f' % (train_acc, val_acc))

# plot training history
plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='val')
plt.legend()
plt.show()
cnn_model.summary()
test_loss, test_acc = cnn_model.evaluate(cnn_test)

print('Test loss:', test_loss)
print('Test accuracy:', test_acc)
print("\nTime:",sum(cb.logs)/60,"min")<jupyter_output>130/130 [==============================] - 1s 4ms/step - loss: 0.8901 - accuracy: 0.6692
Test loss: 0.8901058435440063
Test accuracy: 0.6692307591438293

Time: 71.24510394333325 min<jupyter_text>3.6 Make Predictions<jupyter_code>#get predicted probality
prediction = cnn_model.predict(cnn_test,verbose=1)

#Get class of prediction
predicted_class = np.argmax(prediction,axis=1)

#get trueclass
true_classes = cnn_test.classes

# get names of pictures
filenames = cnn_test.filenames

#store info in dataframe
df_predictions = pd.DataFrame({'Filename': filenames, 'Label': true_classes, 'CNN Model': predicted_class})
#check predictions
prediction
cnn_pred =  pd.DataFrame(prediction, columns = ['Class 0', 'Class 1', 'Class 2', 'Class 3'], index = filenames)

cnn_pred.to_csv(path+'/Models/Cnn_pred_prob.csv')

cnn_pred<jupyter_output><empty_output><jupyter_text>3.7  Confusion matrix and classification report<jupyter_code>#join prediction to dataframe
df_predictions = df_predictions.merge(cnn_model_predictions, on = 'Filename', how = 'inner')

del cnn_model_predictions
def metrics(y_train, pred_train , y_val, pred_val):
    print('___________________________________________________________________________________________________________')
    print('                                                     TRAIN                                                 ')
    print('----------------------------------------------------------------------------------------------------------')
    print(classification_report(y_train, pred_train))
    print(confusion_matrix(y_train, pred_train))


    print('___________________________________________________________________________________________________________')
    print('                                                VALIDATION                                                 ')
    print('-----------------------------------------------------------------------------------------------------------')
    print(classification_report(y_val, pred_val))
    print(confusion_matrix(y_val, pred_val))
train_labels, val_labels = cnn_training.classes, cnn_val.classes
pred_train, pred_val = np.argmax(cnn_model.predict(cnn_training), axis = 1), np.argmax(cnn_model.predict(cnn_val), axis = 1)

metrics(train_labels, pred_train , val_labels, pred_val)
class_labels = list(cnn_test.class_indices.keys())   

print(confusion_matrix(true_classes, predicted_class))

report = classification_report(true_classes, predicted_class, target_names=class_labels)
print(report)<jupyter_output>[[ 4  4  0  8]
 [ 8  3  8 10]
 [ 2  1  2  9]
 [12 16  9 34]]
              precision    recall  f1-score   support

      class0       0.15      0.25      0.19        16
      class1       0.12      0.10      0.11        29
      class2       0.11      0.14      0.12        14
      class3       0.56      0.48      0.52        71

    accuracy                           0.33       130
   macro avg       0.24      0.24      0.24       130
weighted avg       0.36      0.33      0.34       130<jupyter_text>4. ALEX NET<jupyter_code>checkpoint_filepath = path+'/Models/AlexNet.h5'
mc = ModelCheckpoint(
    filepath=checkpoint_filepath,
    verbose = 1,
    monitor='val_accuracy',
    mode='max',
    save_best_only=True)<jupyter_output><empty_output><jupyter_text>4.1 Preprocess inputOnline Data Augmentation  in RGB<jupyter_code>Alex_train_datagen = ImageDataGenerator(rescale = 1./255,
                                   rotation_range = 5,
                                   width_shift_range = 0.1,
                                   height_shift_range = 0.1, 
                                   vertical_flip = True,
                                   horizontal_flip = True,
                                   brightness_range=[0.2,1.5], 
                                   fill_mode='nearest',
                                   zoom_range = 0.2,
                                   ) 

Alex_val_datagen = ImageDataGenerator(rescale = 1./255)
Alex_test_datagen = ImageDataGenerator(rescale = 1./255)

#test different color maps -  class modes and cross validation types
Alex_training = Alex_train_datagen.flow_from_directory(path+'/path/train',
                                                 target_size = (32, 32),
                                                 batch_size = 64,
                                                 shuffle = True,
                                                 class_mode="categorical",
                                                 color_mode = 'rgb')

Alex_val = Alex_val_datagen.flow_from_directory(path+'/path/val',
                                            target_size = (32, 32),
                                            batch_size = 64,
                                            shuffle = True,
                                            class_mode="categorical",
                                            color_mode = 'rgb')

Alex_test = Alex_test_datagen.flow_from_directory(path+'/path/test',
                                            target_size = (32, 32),
                                            batch_size = 1,
                                            shuffle = True,
                                            class_mode="categorical",
                                            color_mode = 'rgb')<jupyter_output>Found 25600 images belonging to 4 classes.
Found 383 images belonging to 4 classes.
Found 130 images belonging to 4 classes.<jupyter_text>4.2 Build model<jupyter_code>#Instantiation
AlexNet = Sequential()

#1st Convolutional Layer - Alexnet took 224px imgs, which we do not have here, we will adjust layers accordingly 
#by reducing first first stride and max pool. All other layers will remain the same

AlexNet.add(Conv2D(filters=96, input_shape=(32,32,3), kernel_size=(4,4), padding='same'))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))
AlexNet.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

#2nd Convolutional Layer
AlexNet.add(Conv2D(filters=256, kernel_size=(5, 5), strides=(1,1), padding='same'))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))
AlexNet.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

#3rd Convolutional Layer
AlexNet.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='same'))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))

#4th Convolutional Layer
AlexNet.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='same'))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))

##5th Convolutional Layer
AlexNet.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='same'))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))
AlexNet.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

#Passing it to a Fully Connected layer
AlexNet.add(Flatten())

# 1st Fully Connected Layer
AlexNet.add(Dense(4096))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))

# Add Dropout to prevent overfitting
AlexNet.add(Dropout(0.4))

#2nd Fully Connected Layer
AlexNet.add(Dense(4096))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))

#Add Dropout
AlexNet.add(Dropout(0.4))

#3rd Fully Connected Layer
AlexNet.add(Dense(1024))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('relu'))
#Add Dropout
AlexNet.add(Dropout(0.4))

#Output Layer
AlexNet.add(Dense(4))
AlexNet.add(BatchNormalization())
AlexNet.add(Activation('softmax'))

#Model Summary
AlexNet.summary()<jupyter_output>Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_26 (Conv2D)           (None, 32, 32, 96)        4704      
_________________________________________________________________
batch_normalization_51 (Batc (None, 32, 32, 96)        384       
_________________________________________________________________
activation_45 (Activation)   (None, 32, 32, 96)        0         
_________________________________________________________________
max_pooling2d_14 (MaxPooling (None, 16, 16, 96)        0         
_________________________________________________________________
conv2d_27 (Conv2D)           (None, 16, 16, 256)       614656    
_________________________________________________________________
batch_normalization_52 (Batc (None, 16, 16, 256)       1024      
______________________________________________________[...]<jupyter_text>4.3 Compile model<jupyter_code>AlexNet.compile(loss = keras.losses.categorical_crossentropy, optimizer= 'adam', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>4.4 Fit model<jupyter_code>history = AlexNet.fit(Alex_training,
        epochs=200, 
        validation_data=Alex_val,
        verbose = 1, 
        callbacks = [mc, reduce_lr, es,cb])<jupyter_output>Epoch 1/200
400/400 [==============================] - 29s 68ms/step - loss: 1.3466 - accuracy: 0.3271 - val_loss: 1.4522 - val_accuracy: 0.1645

Epoch 00001: val_accuracy improved from -inf to 0.16449, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\AlexNet.h5
Epoch 2/200
400/400 [==============================] - 26s 65ms/step - loss: 1.1855 - accuracy: 0.4500 - val_loss: 1.1793 - val_accuracy: 0.3916l - ETA: 6s - loss: 1

Epoch 00002: val_accuracy improved from 0.16449 to 0.39164, saving model to C:\Users\<NAME>\Desktop\M<NAME>\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\AlexNet.h5
Epoch 3/200
400/400 [==============================] - 26s 65ms/step - loss: 1.0238 - accuracy: 0.5541 - val_loss: 1.2453 - val_accuracy: 0.3943

Epoch 00003: val_accuracy improved from 0.39164 to 0.39426, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1[...]<jupyter_text>4.5 Evaluate  model<jupyter_code>_, train_acc = AlexNet.evaluate(Alex_training, verbose=1)
_, val_acc = AlexNet.evaluate(Alex_val, verbose=1)
print('Train: %.3f, Val: %.3f' % (train_acc, val_acc))

# plot training history
plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='val')
plt.legend()
plt.show()
test_loss, test_acc = AlexNet.evaluate(Alex_test)

print('Test loss:', test_loss)
print('Test accuracy:', test_acc)
print("\nTime:",sum(cb.logs)/60,"min")<jupyter_output>130/130 [==============================] - 1s 6ms/step - loss: 0.8966 - accuracy: 0.6462
Test loss: 0.8965646028518677
Test accuracy: 0.6461538672447205

Time: 42.683213278333305 min<jupyter_text>4.5 Make Predictions<jupyter_code>#get predicted probality
prediction = AlexNet.predict(Alex_test,verbose=1)

#Get class of prediction
predicted_class = np.argmax(prediction,axis=1)

#get trueclass
true_classes = Alex_test.classes

# get names of pictures
filenames = Alex_test.filenames

#store info in dataframe
alex_predictions = pd.DataFrame({'Filename': filenames,'AlexNet': predicted_class})
Alex_pred =  pd.DataFrame(prediction, columns = ['Class 0', 'Class 1', 'Class 2', 'Class 3'], index = filenames)

Alex_pred.to_csv(path+'/Models/Alex_pred_prob.csv')

Alex_pred
#join prediction to dataframe
df_predictions = df_predictions.merge(alex_predictions, on = 'Filename', how = 'inner')

del alex_predictions
df_predictions<jupyter_output><empty_output><jupyter_text>4.7  Confusion matrix and classification report<jupyter_code>train_labels, val_labels = Alex_training.classes, Alex_val.classes
pred_train, pred_val = np.argmax(AlexNet.predict(Alex_training), axis = 1), np.argmax(AlexNet.predict(Alex_val), axis = 1)

metrics(train_labels, pred_train , val_labels, pred_val)

class_labels = list(Alex_test.class_indices.keys())   

print(confusion_matrix(true_classes, predicted_class))

report = classification_report(true_classes, predicted_class, target_names=class_labels)
print(report)<jupyter_output>[[ 3  3  3  7]
 [10  5  4 10]
 [ 2  2  4  6]
 [14 11 13 33]]
              precision    recall  f1-score   support

      class0       0.10      0.19      0.13        16
      class1       0.24      0.17      0.20        29
      class2       0.17      0.29      0.21        14
      class3       0.59      0.46      0.52        71

    accuracy                           0.35       130
   macro avg       0.27      0.28      0.27       130
weighted avg       0.41      0.35      0.37       130<jupyter_text>4.8 AlexNet [Model Assessment][mylink][mylink]: Alex_Model_Assessment_RGB.ipynb   5. RESNET50<jupyter_code>checkpoint_filepath = path+'/Models/ResNet50.h5'
mc = ModelCheckpoint(
    filepath=checkpoint_filepath,
    verbose = 1,
    monitor='val_accuracy',
    mode='max',
    save_best_only=True)<jupyter_output><empty_output><jupyter_text>5.1 Preprocess inputresnet50 requires its own preprocessing<jupyter_code>#to play around with these
res_train_datagen = ImageDataGenerator(rescale = 1./255,
                                   preprocessing_function=preprocess_input,
                                   rotation_range = 5,
                                   width_shift_range = 0.06, 
                                   height_shift_range = 0.06, 
                                   vertical_flip = True,
                                   horizontal_flip = True,
                                   brightness_range=[0.2,1.2], 
                                   fill_mode='nearest',
                                   zoom_range = 0.2,
                                   ) 

res_val_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)
res_test_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)

#test different color maps -  class modes and cross validation types
res_train = res_train_datagen.flow_from_directory(path+'/path/train',
                                                 target_size = (32, 32),
                                                 batch_size = 64,
                                                 class_mode="categorical",
                                                 color_mode = 'rgb',
                                                 shuffle = True)

res_val = res_val_datagen.flow_from_directory(path+'/path/val',
                                            target_size = (32, 32),
                                            batch_size = 64,
                                            class_mode="categorical",
                                            color_mode = 'rgb',
                                            shuffle = True)

res_test = res_test_datagen.flow_from_directory(path+'/path/test',
                                            target_size = (32, 32),
                                            batch_size = 1,
                                            class_mode="categorical",
                                            color_mode = 'rgb',
                                            shuffle = True)<jupyter_output>Found 25600 images belonging to 4 classes.
Found 383 images belonging to 4 classes.
Found 130 images belonging to 4 classes.<jupyter_text>5.2 Build model<jupyter_code>res50model = Sequential()

#Layer 1: RES50 without top layer
res50model.add(ResNet50(weights='imagenet', input_shape= (32,32,1),
                 include_top = False, classes=4,))

#Passing it to a Fully Connected layer
res50model.add(Flatten())

# 1st Fully Connected Layer
res50model.add(Dense(2048))
res50model.add(BatchNormalization())
res50model.add(Activation('relu'))
# Add Dropout to prevent overfitting
res50model.add(Dropout(0.4))

#2nd Fully Connected Layer
res50model.add(Dense(1024))
res50model.add(BatchNormalization())
res50model.add(Activation('relu'))

#Add Dropout
res50model.add(Dropout(0.4))

res50model.add(Dense(4, activation = 'softmax'))

#freeze layers in resnet - weights obtained with IMAGENET challenge, we only train final layer
res50model.layers[0].trainable = False

res50model.summary()<jupyter_output>Model: "sequential_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
resnet50 (Functional)        (None, 1, 1, 2048)        23587712  
_________________________________________________________________
flatten_8 (Flatten)          (None, 2048)              0         
_________________________________________________________________
dense_31 (Dense)             (None, 2048)              4196352   
_________________________________________________________________
batch_normalization_62 (Batc (None, 2048)              8192      
_________________________________________________________________
activation_56 (Activation)   (None, 2048)              0         
_________________________________________________________________
dropout_22 (Dropout)         (None, 2048)              0         
______________________________________________________[...]<jupyter_text>5.2 Compile the model<jupyter_code>res50model.compile(loss = keras.losses.categorical_crossentropy, optimizer= 'adam', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>5.3 Fit model<jupyter_code>history = res50model.fit(res_train,
        epochs=200, 
        validation_data=res_val,
        verbose = 1, 
        callbacks = [mc, reduce_lr, es,cb])<jupyter_output>Epoch 1/200
400/400 [==============================] - 32s 72ms/step - loss: 1.5973 - accuracy: 0.3237 - val_loss: 2.1744 - val_accuracy: 0.2193

Epoch 00001: val_accuracy improved from -inf to 0.21932, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\ResNet50.h5
Epoch 2/200
400/400 [==============================] - 27s 66ms/step - loss: 1.3362 - accuracy: 0.3795 - val_loss: 1.4651 - val_accuracy: 0.2402

Epoch 00002: val_accuracy improved from 0.21932 to 0.24021, saving model to C:\Users\<NAME>\Desktop\Mestrado Ricardo\Ano 1\Spring Semester\Deep\Deep_Learning_Project\CNN_SpermCells/Models\ResNet50.h5
Epoch 3/200
400/400 [==============================] - 26s 65ms/step - loss: 1.2796 - accuracy: 0.4019 - val_loss: 1.4026 - val_accuracy: 0.2193

Epoch 00003: val_accuracy did not improve from 0.24021
Epoch 4/200
400/400 [==============================] - 26s 66ms/step - loss: 1.2568 - accuracy[...]<jupyter_text>5.4 Evaluate the model<jupyter_code># evaluate the model
_, train_acc = res50model.evaluate(res_train, verbose=1)
_, val_acc = res50model.evaluate(res_val, verbose=1)
print('Train: %.3f, Val: %.3f' % (train_acc, val_acc))

# plot training history
plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='val')
plt.legend()
plt.show()

test_loss, test_acc = res50model.evaluate(res_test)

print('Test loss:', test_loss)
print('Test accuracy:', test_acc)

print("\nTime:",sum(cb.logs)/60,"min")<jupyter_output>400/400 [==============================] - 26s 65ms/step - loss: 1.1215 - accuracy: 0.51521s - loss: 1.1 - ETA: 0s - loss: 1.120
6/6 [==============================] - 0s 35ms/step - loss: 0.9356 - accuracy: 0.5587
Train: 0.515, Val: 0.559<jupyter_text>5.5 Make Predictions<jupyter_code>#get predicted probality
prediction = res50model.predict(res_test,verbose=1)

#Get class of prediction
predicted_class = np.argmax(prediction,axis=1)

#get trueclass
true_classes = res_test.classes

# get names of pictures
filenames = res_test.filenames

#store info in dataframe
res50_prebuilt_predictions = pd.DataFrame({'Filename': filenames,'ResNet50': predicted_class})
res_pred =  pd.DataFrame(prediction, columns = ['Class 0', 'Class 1', 'Class 2', 'Class 3'], index = filenames)

res_pred.to_csv(path+'/Models/res_pred_prob.csv')

res_pred
Res_pred =  pd.DataFrame(prediction, columns = ['Class 0', 'Class 1', 'Class 2', 'Class 3'], index = filenames)

Res_pred.to_csv(path+'/Models/res50_pred_prob.csv')

Res_pred
#join prediction to dataframe
df_predictions = df_predictions.merge(res50_prebuilt_predictions, on = 'Filename', how = 'inner')

del res50_prebuilt_predictions<jupyter_output><empty_output><jupyter_text>5.7  Confusion matrix and classification report<jupyter_code>train_labels, val_labels = res_train.classes, res_val.classes
pred_train, pred_val = np.argmax(res50model.predict(res_train), axis = 1), np.argmax(res50model.predict(res_val), axis = 1)

metrics(train_labels, pred_train , val_labels, pred_val)
class_labels = list(res_test.class_indices.keys())   

print(confusion_matrix(true_classes, predicted_class))

report = classification_report(true_classes, predicted_class, target_names=class_labels)
print(report)<jupyter_output>[[ 5  3  3  5]
 [10  5  3 11]
 [ 6  2  1  5]
 [16 19  5 31]]
              precision    recall  f1-score   support

      class0       0.14      0.31      0.19        16
      class1       0.17      0.17      0.17        29
      class2       0.08      0.07      0.08        14
      class3       0.60      0.44      0.50        71

    accuracy                           0.32       130
   macro avg       0.25      0.25      0.24       130
weighted avg       0.39      0.32      0.35       130<jupyter_text>6. EFFICIENT NET B7<jupyter_code>checkpoint_filepath = path+'/Models/EffnetB7.h5'<jupyter_output><empty_output><jupyter_text>6.1 Preprocess input<jupyter_code>#to play around with these
EffNetB7train_datagen = ImageDataGenerator(rescale = 1./255,
                                   rotation_range = 5,
                                   width_shift_range = 0.06, 
                                   height_shift_range = 0.06, 
                                   vertical_flip = True,
                                   horizontal_flip = True,
                                   brightness_range=[0.2,1.2], 
                                   fill_mode='nearest',
                                   zoom_range = 0.2,
                                   ) 

EffNetB7val_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)
EffNetB7test_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)

#test different color maps -  class modes and cross validation types
EffNetB7_train = EffNetB7train_datagen.flow_from_directory(path+'/path/train',
                                                 target_size = (32, 32),
                                                 batch_size = 64,
                                                 class_mode="categorical",
                                                 color_mode = 'rgb',
                                                 shuffle = True)

EffNetB7_val = EffNetB7val_datagen.flow_from_directory(path+'/path/val',
                                            target_size = (32, 32),
                                            batch_size = 64,
                                            class_mode="categorical",
                                            color_mode = 'rgb',
                                            shuffle = True)

EffNetB7_test = EffNetB7test_datagen.flow_from_directory(path+'/path/test',
                                            target_size = (32, 32),
                                            batch_size = 1,
                                            class_mode="categorical",
                                            color_mode = 'rgb',
                                            shuffle = True)<jupyter_output>Found 25600 images belonging to 4 classes.
Found 383 images belonging to 4 classes.
Found 130 images belonging to 4 classes.<jupyter_text>6.2 Build model<jupyter_code>dropout_rate = 0.2

EffNetB7model = Sequential()

#Layer 1

EffNetB7model.add(EfficientNetB7(weights='imagenet', input_shape= (32,32,3),
                      include_top = False, classes=4))
EffNetB7model.add(layers.GlobalMaxPooling2D(name="gap")) ### mais eficiente qie o flatten
#EffNetB7.add(Flatten())

# 1st Fully Connected Layer
EffNetB7model.add(Dense(2560))
EffNetB7model.add(BatchNormalization())
EffNetB7model.add(Activation('relu'))
# Add Dropout to prevent overfitting
if dropout_rate > 0:
    EffNetB7model.add(Dropout(dropout_rate, name="dropout_out"))

#2nd Fully Connected Layer
EffNetB7model.add(Dense(1280))
EffNetB7model.add(BatchNormalization())
EffNetB7model.add(Activation('relu'))
if dropout_rate > 0:
    EffNetB7model.add(Dropout(dropout_rate, name="dropout_out1"))

EffNetB7model.add(Dense(4, activation="softmax"))

EffNetB7model.layers[0].trainable = False
EffNetB7model.summary()<jupyter_output>Model: "sequential_10"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
efficientnetb7 (Functional)  (None, 1, 1, 2560)        64097687  
_________________________________________________________________
gap (GlobalMaxPooling2D)     (None, 2560)              0         
_________________________________________________________________
dense_37 (Dense)             (None, 2560)              6556160   
_________________________________________________________________
batch_normalization_66 (Batc (None, 2560)              10240     
_________________________________________________________________
activation_60 (Activation)   (None, 2560)              0         
_________________________________________________________________
dropout_out (Dropout)        (None, 2560)              0         
_____________________________________________________[...]<jupyter_text>6.3 Compile the model<jupyter_code>EffNetB7model.compile(loss = keras.losses.categorical_crossentropy, optimizer= 'adam', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>6.4 Fit model<jupyter_code>EffNet_history = EffNetB7model.fit(EffNetB7_train,
        epochs=200,
        validation_data=EffNetB7_val,
        verbose = 1, 
        validation_steps = len(EffNetB7_val),
        callbacks = [reduce_lr,cb,es])<jupyter_output>Epoch 1/200
400/400 [==============================] - 59s 99ms/step - loss: 1.7011 - accuracy: 0.2532 - val_loss: 1.4497 - val_accuracy: 0.1044
Epoch 2/200
400/400 [==============================] - 35s 87ms/step - loss: 1.4833 - accuracy: 0.2493 - val_loss: 1.4503 - val_accuracy: 0.2193
Epoch 3/200
400/400 [==============================] - 35s 87ms/step - loss: 1.4556 - accuracy: 0.2536 - val_loss: 1.5131 - val_accuracy: 0.1044
Epoch 4/200
400/400 [==============================] - 34s 85ms/step - loss: 1.4433 - accuracy: 0.2529 - val_loss: 1.4055 - val_accuracy: 0.2193
Epoch 5/200
400/400 [==============================] - 35s 87ms/step - loss: 1.4270 - accuracy: 0.2583 - val_loss: 1.5456 - val_accuracy: 0.2193
Epoch 6/200
400/400 [==============================] - 35s 88ms/step - loss: 1.4203 - accuracy: 0.2446 - val_loss: 1.4772 - val_accuracy: 0.1044
Epoch 7/200
400/400 [==============================] - 37s 93ms/step - loss: 1.4100 - accuracy: 0.2485 - val_loss: 1.3884 - val_ac[...]<jupyter_text>6.5 Evaluate the model<jupyter_code># evaluate the model
_, train_acc = EffNetB7model.evaluate(EffNetB7_train, verbose=1)
_, val_acc = EffNetB7model.evaluate(EffNetB7_val, verbose=1)
print('Train: %.3f, Val: %.3f' % (train_acc, val_acc))

# plot training history
plt.plot(EffNet_history.history['loss'], label='train')
plt.plot(EffNet_history.history['val_loss'], label='val')
plt.legend()
plt.show()
test_loss, test_acc = EffNetB7model.evaluate(EffNetB7_test)

print('Test loss:', test_loss)
print('Test accuracy:', test_acc)
print("\nTime:",sum(cb.logs)/60,"min")<jupyter_output>130/130 [==============================] - 8s 57ms/step - loss: 1.3333 - accuracy: 0.5462
Test loss: 1.3332865238189697
Test accuracy: 0.5461538434028625

Time: 88.67366407499988 min<jupyter_text>6.6 Make Predictions<jupyter_code>#get predicted probality
prediction = EffNetB7model.predict(EffNetB7_test,verbose=1)

#Get class of prediction
predicted_class = np.argmax(prediction,axis=1)

#get trueclass
true_classes = EffNetB7_test.classes

# get names of pictures
filenames = EffNetB7_test.filenames

#store info in dataframe
EffNetB7_predictions = pd.DataFrame({'Filename': filenames,'EffNetB7': predicted_class})
Effnet_pred =  pd.DataFrame(prediction, columns = ['Class 0', 'Class 1', 'Class 2', 'Class 3'], index = filenames)

Effnet_pred.to_csv(path+'/Models/eff_pred_prob.csv')

Effnet_pred
#join prediction to dataframe
df_predictions = df_predictions.merge(EffNetB7_predictions, on = 'Filename', how = 'inner')

del EffNetB7_predictions
df_predictions<jupyter_output><empty_output><jupyter_text>6.7 Confusion matrix and classification report<jupyter_code>train_labels, val_labels = EffNetB7_train.classes, EffNetB7_val.classes
pred_train, pred_val = np.argmax(res50model.predict(EffNetB7_train), axis = 1), np.argmax(res50model.predict(EffNetB7_val), axis = 1)

metrics(train_labels, pred_train , val_labels, pred_val)

class_labels = list(EffNetB7_test.class_indices.keys())   

print(confusion_matrix(true_classes, predicted_class))


report = classification_report(true_classes, predicted_class, target_names=class_labels)
print(report)
#SAVE CSV

df_predictions.to_csv(path+'/Models/pred_classes.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12088205813564985, 'fraction_numerical': 0.051431117050896535, 'mean_word_length': 2.9510692598785533, 'pattern_counts': {'":': 0, '<': 139, '<?xml version=': 0, '>': 143, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 175, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '199515', 'license': 'CC0-1.0', 'n_tokens_mistral': 15111, 'n_tokens_neox': 13455, 'n_words': 2987, 'path': '.ipynb_checkpoints/CNN_SpermCells_final-checkpoint.ipynb', 'repo_name': 'RicardoSantos0/Deep_Learning_Project', 'size': '45000'}