<jupyter_start><jupyter_text>Training Tensorflow MobileNetSSD v2 and Inception v3 models to classify flowers and fruits using EOL Angiosperm Images as training data---*Last Updated 25 September 2020*   Train [MobileNet SSD v2](https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/4) and [Inception v3](https://tfhub.dev/google/imagenet/inception_v3/classification/4) to classify flowers and fruits from EOL images. Images are classified into flower, fruit or null. The training dataset is an image bundle containing up to 30 images per Angiosperm plant family found in EOL that was manually sorted into flower, fruit, or null classes. Classifications will be used to generate image tags to improve searchability of EOLv3 images.EOL max30imgPerFam bundle images were downloaded to Google Drive using [flower_fruit_preprocessing.ipynb](https://colab.research.google.com/github/aubricot/computer_vision_with_eol_images/blob/master/classification_for_image_tagging/flower_fruit/flower_fruit_preprocessing.ipynb). Then, images were manually sorted into folders based on image class (flower, fruit, null, other/excluded images). Excluded images were moved to an external folder for use training future classifiers before running this notebook.Pre-trained MobileNet SSD v2 and Inception v3 models were fine-tuned to classify flower images. **Best results for each model from 42 total trials:**   * **MobileNet SSD v2 was trained for 3 hours to 60 epochs with Batch Size=16, Lr=0.001, Dropout=0.3. Final validation accuracy = 0.81*** Inception v3 was trained for 3.5 hours to 40 epochs with Batch Size=16, Lr=0.001, Dropout=0.2. Final validation accuracy = 0.72. **Notes*** Change filepaths or information using the form fields to the right of code blocks (also noted in code with 'TO DO')* Make sure to set the runtime to GPU Hardware Accelerator with a High Ram Runtime Shape (Runtime -> Change runtime type)**References**   * https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough* https://www.tensorflow.org/tutorials/images/classification* https://medium.com/analytics-vidhya/create-tensorflow-image-classification-model-with-your-own-dataset-in-google-colab-63e9d7853a3e* https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_image_retraining.ipynbscrollTo=umB5tswsfTEQ* https://medium.com/analytics-vidhya/how-to-do-image-classification-on-custom-dataset-using-tensorflow-52309666498e* https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html* https://www.pyimagesearch.com/2018/12/31/keras-conv2d-and-convolutional-layers/  Imports   ---<jupyter_code># Mount google drive to import/export files
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
# For working with data and plotting graphs
import itertools
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# For image classification and training
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D, InputLayer
from tensorflow.keras.preprocessing.image import ImageDataGenerator

print("TF version:", tf.__version__)
print("Hub version:", hub.__version__)
print("GPU is", "available" if tf.test.is_gpu_available() else "NOT AVAILABLE")<jupyter_output><empty_output><jupyter_text>Train Classification Model(s)---  Training Dataset Preparation  If using pre-trained classifierUse dropdown menu on the right to choose which pre-trained model to use and set the image batch size for training<jupyter_code># TO DO: Select pre-trained model to use from Tensorflow Hub Model Zoo
module_selection = ("mobilenet_v2_1.0_224", 224) #@param ["(\"mobilenet_v2_1.0_224\", 224)", "(\"inception_v3\", 299)"] {type:"raw", allow-input: true}
handle_base, pixels = module_selection

if handle_base == "inception_v3":
  MODULE_HANDLE ="https://tfhub.dev/google/imagenet/inception_v3/classification/4".format(handle_base)
elif handle_base == "mobilenet_v2_1.0_224":
  MODULE_HANDLE ="https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/4".format(handle_base) 

# TO DO: adjust batch size to make training faster or slower
BATCH_SIZE = "16" #@param ["16", "32", "64", "128"]

IMAGE_SIZE = (pixels, pixels)
print("Using {} with input size {} and batch size {}".format(handle_base, IMAGE_SIZE, BATCH_SIZE))
# TO DO: Change directory to wherever images are stored
PATH = '/content/drive/My Drive/summer20/classification/flower_fruit/images' #@param {type:"string"}
TRAINING_DATA_DIR = str(PATH)
print(TRAINING_DATA_DIR)

# TO DO: Adjust interpolation method and see how training results change
interp = "nearest" #@param ["nearest", "bilinear"]

# Set data generation and flow parameters
datagen_kwargs = dict(rescale=1./255, validation_split=.20)
dataflow_kwargs = dict(target_size=IMAGE_SIZE, batch_size=int(BATCH_SIZE),
                    interpolation = interp)

# Make test dataset
test_datagen = ImageDataGenerator(**datagen_kwargs)
test_generator = test_datagen.flow_from_directory(
TRAINING_DATA_DIR,
subset="validation",
shuffle=True,
target_size=IMAGE_SIZE
)

# Make train dataset using augmentation parameters below
train_datagen = ImageDataGenerator(
    rotation_range=40, # randomly rotates image 0-40 degrees
    horizontal_flip=True, # random horizontal flip
    width_shift_range=0.2, height_shift_range=0.2, # randomly distorts height and width
    shear_range=0.2, zoom_range=0.2, # randomly clips and zooms in on images
    **datagen_kwargs)
train_generator = train_datagen.flow_from_directory(
    TRAINING_DATA_DIR, subset="training", shuffle=True, **dataflow_kwargs)

# Learn more about data batches
image_batch_train, label_batch_train = next(iter(train_generator))
print("Image batch shape: ", image_batch_train.shape)
print("Label batch shape: ", label_batch_train.shape)
dataset_labels = sorted(train_generator.class_indices.items(), key=lambda pair:pair[1])
dataset_labels = np.array([key.title() for key, value in dataset_labels])
print(dataset_labels)<jupyter_output><empty_output><jupyter_text>If creating model from scratch* Select batch size from dropdown menu on the right  * Type in filepath to image directory using form field on the right<jupyter_code># TO DO: adjust batch size to make training faster or slower
BATCH_SIZE = 16 #@param ["16", "32", "64", "128"] {type:"raw"}

# Set input image size for model
IMAGE_SIZE = (150, 150)

# TO DO: Change directory to wherever images are stored
PATH = '/content/drive/My Drive/summer20/classification/flower_fruit/images' #@param {type:"string"}
TRAINING_DATA_DIR = str(PATH)
print(TRAINING_DATA_DIR)

# TO DO: Adjust interpolation method and see how training results change
interp = "nearest" #@param ["nearest", "bilinear"]

# Set data generation and flow parameters
datagen_kwargs = dict(rescale=1./255, validation_split=.20)
dataflow_kwargs = dict(target_size=IMAGE_SIZE, batch_size=int(BATCH_SIZE),
                    interpolation = interp)

# Make test dataset
test_datagen = ImageDataGenerator(**datagen_kwargs)
test_generator = test_datagen.flow_from_directory(
TRAINING_DATA_DIR,
subset="validation",
shuffle=True,
target_size=IMAGE_SIZE
)

# Make train dataset using augmentation parameters below
train_datagen = ImageDataGenerator(
    rotation_range=40, # randomly rotates image 0-40 degrees
    horizontal_flip=True, # random horizontal flip
    width_shift_range=0.2, height_shift_range=0.2, # randomly distorts height and width
    shear_range=0.2, zoom_range=0.2, # randomly clips and zooms in on images
    **datagen_kwargs)
train_generator = train_datagen.flow_from_directory(
    TRAINING_DATA_DIR, subset="training", shuffle=True, **dataflow_kwargs)

# Learn more about data batches
image_batch_train, label_batch_train = next(iter(train_generator))
print("Image batch shape: ", image_batch_train.shape)
print("Label batch shape: ", label_batch_train.shape)
dataset_labels = sorted(train_generator.class_indices.items(), key=lambda pair:pair[1])
dataset_labels = np.array([key.title() for key, value in dataset_labels])
print(dataset_labels)<jupyter_output><empty_output><jupyter_text>Model Preparation  If fine-tuning pre-trained model<jupyter_code># Build model
print("Building model with", handle_base)
# TO DO: If model is overfitting, add/increase dropout rate
dropout_rate = 0.4 #@param {type:"slider", min:0, max:0.5, step:0.1}

def create_model():
  model = tf.keras.Sequential([
    InputLayer(input_shape=IMAGE_SIZE + (3,)),
    hub.KerasLayer(MODULE_HANDLE, trainable=True),
    Dropout(rate = dropout_rate), 
    Dense(train_generator.num_classes,
                          kernel_regularizer=tf.keras.regularizers.l2(0.0001))
  ])
  
  # Build model
  model.build((None,)+IMAGE_SIZE+(3,))
  
  # Compile model
  model.compile(
    # Parameters for Adam optimizer
    optimizer=tf.keras.optimizers.Adam(
      learning_rate=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False,
      name='Adam'), 
      # Categorical cross entropy because 3 exclusive classes
    loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True, label_smoothing=0.1),
    metrics=['accuracy'])
  return model

# Create new model instance
model = create_model()

# Steps per epoch and testing
steps_per_epoch = train_generator.samples // train_generator.batch_size
test_steps = test_generator.samples // test_generator.batch_size

# Display model architecture
model.summary()<jupyter_output><empty_output><jupyter_text>Create new model from scratch<jupyter_code># Build model
print("Building model from scratch")
# Modified from TF Image Classification Tutorial

# To DO: Adjust model perforamance using below hyperparameters
# See blog post for explanations https://www.pyimagesearch.com/2018/12/31/keras-conv2d-and-convolutional-layers/

# If model is overfitting, add/increase dropout rate
dropout_rate = 0.2 #@param {type:"slider", min:0, max:0.7, step:0.1}
# Layer 1: Start with smaller number of filters and increase number if performance too low
no_filters_lay1 = 64 #@param ["32", "64", "128"] {type:"raw"}
# Layer 2: Use either the same number of layers as Layer 1, or 2x as many
no_filters_lay2 = no_filters_lay1 * 2 #@param ["no_filters_lay1", "no_filters_lay1 * 2"] {type:"raw"}
# Layer 3: Use 2x as many layers as Layer 2
no_filters_lay3 = no_filters_lay2 * 2
# Layer 1: If input image size >128, may need to use initial filter size of 5, 5
filter_size_lay1 = (5, 5) #@param ["(3, 3)", "(5, 5)", "(7, 7)"] {type:"raw"}
# Final Dense Layer: Set number of output nodes for network (same as number of classes)
num_classes = 3 #@param {type:"integer"}
# Compile model: Choose loss function. Categorical supposed to be better for multiple classes, but binary got better results one run
loss_fun = "categorical_crossentropy" #@param ["categorical_crossentropy", "binary_crossentropy"]

def create_model():
  model = Sequential([
    Conv2D(no_filters_lay1, filter_size_lay1, padding='same', activation='relu',
        input_shape=(IMAGE_SIZE + (3,))),
    MaxPooling2D(pool_size=(2, 2)),
    Conv2D(no_filters_lay2, (3, 3), padding='same', activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Conv2D(no_filters_lay3, (3, 3), padding='same', activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Flatten(), # this converts our 3D feature maps to 1D feature vectors
    Dense(64, activation='relu'),
    Dropout(dropout_rate),
    Dense(num_classes, activation='softmax') # softmax good for multiple class models with exclusive classes
])

  # Compile model
  optimizer = tf.keras.optimizers.Adam(learning_rate=0.01, beta_1=0.9, beta_2=0.999, epsilon=1, amsgrad=False,
    name='Adam')
  model.compile(loss=loss_fun,
              optimizer=optimizer,
              metrics=['accuracy'])
  return model

# Create model instance
model = create_model()

# Set steps per epoch and testing
steps_per_epoch = train_generator.samples // train_generator.batch_size
test_steps = test_generator.samples // test_generator.batch_size

# Display model architecture
model.summary()<jupyter_output><empty_output><jupyter_text>Actual Training * For the first time training each model with specified hyper-parameters, go to **First time training**. If hyper-parameters are changed and a model is retrained, also go to **First time training**.   * If hyper-parameters are kept the same, and the model only needs to be trained for additional epochs, go to **Resume training from a saved checkpoint** below.<jupyter_code># TO DO: Adjust number of epochs to find balance between underfit and overfit for training
num_epochs = '80' #@param {type:"string"}

# Save each new training attempt results in new folder
last_attempt = !ls /content/drive/'My Drive'/summer20/classification/flower_fruit/saved_models/ | tail -n 1
if not last_attempt:
  last_attempt = 0
else:
  last_attempt = int(last_attempt.n)
TRAIN_SESS_NUM = str(last_attempt + 1)
CKPT_PATH = '/content/drive/My Drive/summer20/classification/flower_fruit/saved_models/' + TRAIN_SESS_NUM + '/ckpt/cp-{epoch:04d}.ckpt' 

print("Last training attempt number:", last_attempt)
print("Training attempt number: {}, for {} epochs".format(TRAIN_SESS_NUM, num_epochs))

# Create a callback that saves the model's weights during training
ckpt_callback = tf.keras.callbacks.ModelCheckpoint(filepath=CKPT_PATH,
                                                 save_weights_only=True,
                                                 verbose=1)

# Save weights for 0th epoch
model.save_weights(CKPT_PATH.format(epoch=0))

# Train the model with the new callback
hist = model.fit(
    train_generator,
    epochs=int(num_epochs), steps_per_epoch=steps_per_epoch,
    callbacks=[ckpt_callback],
    validation_data=test_generator,
    validation_steps=test_steps).history

# Save trained model 
saved_model_path = '/content/drive/My Drive/summer20/classification/flower_fruit/saved_models/' + TRAIN_SESS_NUM
tf.saved_model.save(model, saved_model_path)<jupyter_output><empty_output><jupyter_text>Plot loss and accuracy for training session<jupyter_code># Plot loss
plt.figure()
plt.title("Attempt {}:Training and Validation Loss".format(TRAIN_SESS_NUM))
plt.xlabel("Training Steps")
plt.ylim([0,2])
plt.plot(hist["loss"], label='Train')
plt.plot(hist["val_loss"], label='Test')
plt.legend(loc='lower right')

# Plot accuracy
plt.figure()
plt.title("Attempt {}:Training and Validation Accuracy".format(TRAIN_SESS_NUM))
plt.xlabel("Training Steps")
plt.ylim([0,1])
plt.plot(hist["accuracy"], label='Train')
plt.plot(hist["val_accuracy"], label='Test')
plt.legend(loc='upper right')
path = '/content/drive/My Drive/summer20/classification/flower_fruit/train_graphs/' + TRAIN_SESS_NUM + '.png'
plt.savefig(path)

# Print final loss and accuracy values
final_loss, final_accuracy = model.evaluate(test_generator, steps = test_steps)
print('Final loss: {:.2f}'.format(final_loss))
print('Final accuracy: {:.2f}%'.format(final_accuracy * 100))<jupyter_output><empty_output><jupyter_text>Review training results---   Display classification results on images<jupyter_code># Define functions

# TO DO: Do you want to display classification results for the most recently trained model?
answer = "No" #@param ["Yes", "No"]
# TO DO: If No, manually input desired training attempt number to the right
if answer == "Yes":
  # Display results from most recent training attempt
  last_attempt = !ls /content/drive/'My Drive'/summer20/classification/flower_fruit/saved_models/ | tail -n 1
  TRAIN_SESS_NUM = str(last_attempt.n)
else:
  TRAIN_SESS_NUM = "11" #@param ["09", "07", "08", "03", "05", "22_retrain", "23_retrain", "11"]

# Load trained model from path
saved_model_path = '/content/drive/My Drive/summer20/classification/flower_fruit/saved_models/' + TRAIN_SESS_NUM
flower_model = tf.keras.models.load_model(saved_model_path)

# Function for plotting classification results with color-coded label if true or false prediction
label_names = ['Flower', 'Fruit', 'Null']
# Run inference
from PIL import Image
import time

# TO DO: Choose which image class to inspect results for in true_imclass to right
# TO DO: Choose start and end image numbers to inspect (inspect up to 50 images at a time)
base = '/content/drive/My Drive/summer20/classification/'
classifier = "flower_fruit/" #@param ["flower_fruit/"]
true_imclass = "03_null" #@param ["02_fruit", "01_flower", "03_null"]
PATH_TO_TEST_IMAGES_DIR = base + classifier + "images/" + true_imclass
names = os.listdir(PATH_TO_TEST_IMAGES_DIR)
TEST_IMAGE_PATHS = [os.path.join(PATH_TO_TEST_IMAGES_DIR, name) for name in names]

# Loops through first 5 image urls from the text file
start = 0 #@param {type:"number"}
end =  50 #@param {type:"number"}
for im_num, im_path in enumerate(TEST_IMAGE_PATHS[start:end], start=1):
    # Load in image
    img = Image.open(im_path)
    image = img.resize((224,224))
    image = np.reshape(image,[1,224,224,3])
    # Record inference time
    start_time = time.time()
    # Detection and draw boxes on image
    predictions = flower_model.predict(image, batch_size=1)
    label_num = np.argmax(predictions)
    conf = predictions[0][label_num]
    otherconfa = predictions[0][:label_num]
    otherconfb = predictions[0][label_num+1:]
    imclass = label_names[label_num]
    other_class = label_names[:label_num]+label_names[label_num+1:]
    end_time = time.time()
    # Display progress message after each image
    print('Inference complete for {} of {} images'.format(im_num, (end-start)))

    # Plot and show detection boxes on images
    _, ax = plt.subplots(figsize=(10, 10))
    ax.imshow(img)
    plt.title('{}) Prediction: {}, Confidence: {}, Inference time: {}, \
    \n Other Predictions: {}, Other Conf: {}, {}'.format(im_num, imclass, \
    format(conf, '.2f'), format(end_time-start_time, '.2f'), other_class, \
    otherconfa, otherconfb))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10918611776502701, 'fraction_numerical': 0.023118489220656232, 'mean_word_length': 5.386339381003202, 'pattern_counts': {'":': 3, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 13, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '301056', 'license': 'MIT', 'n_tokens_mistral': 5940, 'n_tokens_neox': 5533, 'n_words': 1577, 'path': 'classification_for_image_tagging/flower_fruit/flower_fruit_train.ipynb', 'repo_name': 'aubricot/computer_vision_with_eol_images', 'size': '17951'}