<jupyter_start><jupyter_text>**Multi-classification Use Case for ResNet50**For supermarkets and other stores that sell fruits, it could be important to utilize computer vision to automate the inventory management process. Stores might need to periodically detect whether fruits are in good shape and whether some of them are misplaced to a different group.In this project, we explore how a mature model architecture may perform in this use case, with apples, bananas, and oranges, all of which might be fresh or rotten. The model mainly consists of ResNet50 with pre-trained parameters, connected with a flattened and a dropout layers.  **1.Import Libraries**<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import seaborn as sns
import os
from google.colab import drive
drive.mount('/content/drive')

#data: https://drive.google.com/drive/folders/1JrRWbfaNl2EHXzObSPRCXlIThQRhxeI2?usp=sharing<jupyter_output>Mounted at /content/drive<jupyter_text>**2. Data Loading & Augmentation**In our project, we split the training set further into a set for training, and the other set for validation at each epoch.Important hyperparameters here are the batch size, input image size, and the validation split.<jupyter_code>path = "/content/drive/MyDrive/HKSTP DTT /Mini Project/Computer Vision Final Project Files/dataset"
train_path = os.path.join(path, "train")
test_path = os.path.join(path, "test")
from keras.preprocessing.image import ImageDataGenerator  
train_datagen = ImageDataGenerator(rotation_range=75,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   fill_mode='nearest',
                                   validation_split=0.2) # Split 0.2 of the train data to validation data
Classes = ['freshapples', 'freshbanana','freshoranges','rottenapples', 'rottenbanana', 'rottenoranges']
# Separate train, validation and test set 
Batch_Size = 32

train_set = train_datagen.flow_from_directory(train_path,
                                              target_size=(224, 224),
                                              batch_size=Batch_Size,
                                              classes=Classes,
                                              subset='training') # Set as training data

validation_set = train_datagen.flow_from_directory(train_path, # Same directory as training data
                                                   target_size=(224, 224),
                                                   batch_size=Batch_Size,
                                                   classes=Classes,
                                                   subset='validation') # Set as validation data

test_set = ImageDataGenerator().flow_from_directory(test_path, 
                                                    target_size=(224,224), 
                                                    classes=Classes,
                                                    shuffle= False,
                                                    batch_size = Batch_Size)# Set as test data<jupyter_output>Found 8739 images belonging to 6 classes.
Found 2182 images belonging to 6 classes.
Found 2698 images belonging to 6 classes.<jupyter_text>Now we tally the class-wise numbers of images under training, validation, and test sets.<jupyter_code>print ("In train_set ")
for i in range(len (train_set.class_indices)):
    print(Classes[i],":\t",list(train_set.classes).count(i))
print ("") 

print ("In validation_set ")
for i in range(len (validation_set.class_indices)):
    print(Classes[i],":\t",list(validation_set.classes).count(i))
print ("") 

print ("In test_set ")
for i in range(len (test_set.class_indices)):
    print(Classes[i],":\t",list(test_set.classes).count(i))<jupyter_output>In train_set 
freshapples :	 1355
freshbanana :	 1281
freshoranges :	 1173
rottenapples :	 1874
rottenbanana :	 1780
rottenoranges :	 1276

In validation_set 
freshapples :	 338
freshbanana :	 320
freshoranges :	 293
rottenapples :	 468
rottenbanana :	 444
rottenoranges :	 319

In test_set 
freshapples :	 395
freshbanana :	 381
freshoranges :	 388
rottenapples :	 601
rottenbanana :	 530
rottenoranges :	 403<jupyter_text>**3. Model Construction & Training**Our model relies vastly on the pre-constructed and pre-trained ResNet50. We construct the entire model as normal. The connection is loaded data -> ResNet50 -> Flatten -> Dropout(0.5) -> Output. After the construction, we freeze the top 2 layers of ResNet50 before compiling. A noticeable difference of our model is that we train the transfer learning model all at once, unlike the freeze-train-unfreeze-train procedure we studied in class. Moreover, we also limit the steps per training epoch and per validation epoch to speed up runtime.<jupyter_code>import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Flatten, Dense, Dropout
image_size = (224, 224, 3)
freeze_layers = 2
num_epochs = 5 
# Data is ready to be run into the base model, ResNet50, after being loaded & augmented
classifier = tf.keras.applications.ResNet50(include_top=False, weights='imagenet', input_tensor=None,
               input_shape=image_size)

x = classifier.output
x = Flatten()(x)
# Dropout layer
x = Dropout(0.5)(x)
# Dense layer
output_layer = Dense(len(Classes), activation='softmax', name='softmax')(x)
# Freeze some layers on top before compiling, with the number of layers as a h-param
new_classifier = Model(inputs=classifier.input, outputs=output_layer)
for layer in new_classifier.layers[:freeze_layers]:
    layer.trainable = False
for layer in new_classifier.layers[freeze_layers:]:
    layer.trainable = True<jupyter_output><empty_output><jupyter_text>Now we conduct model compilation and training.Surprisingly, even setting a low learning rate of 0.0001, the accuracy manages to climb from about 0.2 to 0.7 during the first epoch, and from 0.7 to 0.88 during the second epoch.<jupyter_code># Complie the model
new_classifier.compile(optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.0001),
                  loss='categorical_crossentropy', metrics=['accuracy'])

new_classifier.summary()<jupyter_output>Model: "model_3"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_4 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 230, 230, 3)  0           input_4[0][0]                    
__________________________________________________________________________________________________
conv1_conv (Conv2D)             (None, 112, 112, 64) 9472        conv1_pad[0][0]                  
__________________________________________________________________________________________________
conv1_bn (BatchNormalization)   (None, 112, 112, 64) 256         conv1_conv[0][0]           [...]<jupyter_text>Visualizing the layers<jupyter_code>pip install git+https://github.com/paulgavrikov/visualkeras
import visualkeras
from tensorflow.keras.layers import BatchNormalization, ZeroPadding2D, Activation, MaxPool2D

# Visualizing our model architecture
visualkeras.layered_view(new_classifier, 
                         legend=True, 
                         to_file='output.png',
                         type_ignore=[MaxPool2D,Activation,ZeroPadding2D,BatchNormalization,Dropout,Flatten])  # font is optional!
# Train the model
history = new_classifier.fit(train_set,
                        steps_per_epoch = 50, # normally 274 (test_set // Batch_Size)
                        validation_data = validation_set,
                        validation_steps = 20, # normally 96 (validation_set // Batch_Size)
                        epochs = num_epochs)
# save the model 
model_path = "model-resnet50-trained.h5"
new_classifier.save(model_path)<jupyter_output>/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  category=CustomMaskWarning)<jupyter_text>**4. Model Evaluation**After the training finishes, we run the validation and test data through the model to obtain loss function values and accuracy scores. This section also contains the single-image prediction algorithm and some visualizations on model performance.<jupyter_code># Validation accuracy
new_classifier.evaluate(validation_set)
# test accuracy
new_classifier.evaluate(test_set)
#New part to visualize the model

acc = history.history['accuracy'] # new change for TF changes
val_acc = history.history['val_accuracy'] # new change for TF changes

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(num_epochs)

plt.figure(figsize=(8, 8))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>**Single-image Prediction**To realize this function, we create a dictionary that maps the file name of the image to its class name, so that after randomly drawing an image file, we can load and run the image through the model, and meanwhile retrieving the true label from the dictionary.<jupyter_code>label_dict = test_set.class_indices  # we can match the label indices and their names
label_dict = {val:key for key, val in label_dict.items()}
label_dict 
from tensorflow.keras.models import load_model
classifier_model = load_model(model_path) # load the model 
root_dir = test_path
folders = ['freshapples','freshbanana','freshoranges','rottenapples','rottenbanana','rottenoranges']
filenames = []
file_dict = {}

# Create the dictionary for mapping filename with label
for folder in folders:
  files = os.listdir(f'{root_dir}/{folder}')
  for fruit_img in files:
    filenames.append(fruit_img)
    file_dict[fruit_img] = folder
from keras.preprocessing import image
def Image_and_Prediction():
  # Randomly draw one image and print it out
  sample = np.random.choice(filenames)
  print(sample)
  folder_name = file_dict[sample]
  img_path = f'{root_dir}/{folder_name}/{sample}'
  img = image.load_img(img_path)
  plt.imshow(img)

  # Run the randomly drawn image through the model and get results 
  test_image = image.load_img(img_path, target_size=image_size)
  test_image = image.img_to_array(test_image)
  test_image = np.expand_dims(test_image,axis=0)
  result = classifier_model.predict(test_image).astype(int)
  result = [1 if x == max(list(result)[0]) else 0 for x in list(result)[0]]
  result_label = label_dict[list(result).index(1)] if 1 in list(result) else 'no prediction'
  print(list(result))
  print('predicted: ',result_label)
  print('true: ',folder_name)
Image_and_Prediction()<jupyter_output>saltandpepper_Screen Shot 2018-06-12 at 11.33.50 PM.png
[1, 1, 1, 1, 1, 1]
predicted:  freshapples
true:  rottenoranges<jupyter_text>**Visualization of Model Performance**In this section, we plot a heatmap for the confusion matrix and print out the classification report so that we can have more insights into the model.When coding for this section, we didn't find any built-in function that produces confusion matrices in keras. Thus, we go back to the sci-kit learn library for that function. In order to sync the sequences of true labels and predicted labels, we iteratively append those labels by loading test images one-by-one from the folders. More efficient codes could be implemented for this function.<jupyter_code>res_idx, res_lbl = [], []

for filename in filenames:
  folder_name = file_dict[filename]
  root_dir = "drive/MyDrive/Mini Project/Computer Vision Final Project Files/dataset/test"
  img_path = f'{root_dir}/{folder_name}/{filename}'
  test_image = image.load_img(img_path, target_size=image_size)
  test_image = image.img_to_array(test_image)
  test_image = np.expand_dims(test_image,axis=0)
  result = classifier_model.predict(test_image).astype(int)
  result = [1 if x == max(list(result)[0]) else 0 for x in list(result)[0]]
  result_idx = list(result).index(1)
  res_idx.append(result_idx)
  result_label = label_dict[list(result).index(1)] if 1 in list(result) else 'no prediction'
  res_lbl.append(result_label)

y_true = list(file_dict.values())
y_pred = res_lbl
# Print the classification report
from sklearn.metrics import confusion_matrix, classification_report
print('----------------CLASSIFICATION REPORT----------------')
print(classification_report(y_true, y_pred))
# Plot the confusion matrix
plt.figure(figsize=(17,9))
sns.heatmap(confusion_matrix(y_true, y_pred, normalize='true'),
            fmt='.1%',
            annot=True,
            cmap="Blues",
            xticklabels=folders,
            yticklabels=folders,
            annot_kws={"size": 14})
plt.title('Normalized Confusion Matrix', {'fontsize':18})
plt.yticks(np.arange(6)+0.5,rotation=0, fontsize="12", va="center")
plt.xticks(np.arange(6)+0.5,rotation=0, fontsize="12", va="center")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09479392624728851, 'fraction_numerical': 0.021836587129428778, 'mean_word_length': 3.633500837520938, 'pattern_counts': {'":': 4, '<': 31, '<?xml version=': 0, '>': 35, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 32, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '173620', 'license': 'MIT', 'n_tokens_mistral': 4141, 'n_tokens_neox': 3830, 'n_words': 1155, 'path': '.ipynb_checkpoints/ResNet50-Final-checkpoint.ipynb', 'repo_name': 'Ritz-19/ResNet50_Fruit_Freshness', 'size': '13830'}