<jupyter_start><jupyter_text>Image Classification1. Examine and understand data2. Build an input pipeline3. Build the model4. Train the model5. Test the model6. Improve the model and repeat the process  Import TensorFlow and other libraries<jupyter_code>import PIL
import os, warnings
import pathlib
import matplotlib.pyplot as plt
from matplotlib import gridspec

import numpy as np
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
from tensorflow.keras.applications import ResNet50
print(tf.__version__)
if tf.test.gpu_device_name():
    print('GPU found')
else:
    print("No GPU found")

# Reproducability
def set_seed(seed=31415):
    np.random.seed(seed)
    tf.random.set_seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    os.environ['TF_DETERMINISTIC_OPS'] = '1'
set_seed(31415)
print(tf.config.list_physical_devices(
    device_type=None
))<jupyter_output>[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]<jupyter_text>Load and explore the dataset<jupyter_code>train_data_dir = "tf_ssldoctable/train"
train_data_dir = pathlib.Path(train_data_dir)
if train_data_dir.exists():
    image_count = len(list(train_data_dir.glob('*/*.png')))
    print(image_count)
else:
    print("Image directory not found !")
bargraph = list(train_data_dir.glob('bargraph/*'))
PIL.Image.open(str(bargraph[0]))
doctable = list(train_data_dir.glob('doctable/*'))
PIL.Image.open(str(doctable[0]))
test_data_dir = "tf_ssldoctable/test"
test_data_dir = pathlib.Path(test_data_dir)
if test_data_dir.exists():
    image_count = len(list(test_data_dir.glob('*/*.png')))
    print(image_count)
else:
    print("Image directory not found !")
bargraph = list(test_data_dir.glob('bargraph/*'))
PIL.Image.open(str(bargraph[0]))
doctable = list(test_data_dir.glob('doctable/*'))
PIL.Image.open(str(doctable[0]))<jupyter_output><empty_output><jupyter_text>Load data using a Keras utilityTrain and Validation Split: 80 % and 20%<jupyter_code>batch_size = 8
img_height = 224
img_width = 224
train_dataset = tf.keras.preprocessing.image_dataset_from_directory(
              train_data_dir,
              validation_split=0.2,
              subset="training",
              seed=123,
              image_size=(img_height, img_width),
              batch_size=batch_size)

val_dataset = tf.keras.preprocessing.image_dataset_from_directory(
              train_data_dir,
              validation_split=0.2,
              subset="validation",
              seed=123,
              image_size=(img_height, img_width),
              batch_size=batch_size)

test_dataset = tf.keras.preprocessing.image_dataset_from_directory(
                test_data_dir,
                shuffle=True,
                image_size=(img_height, img_width),
                batch_size=batch_size)<jupyter_output>Found 10000 files belonging to 2 classes.
Using 8000 files for training.
Found 10000 files belonging to 2 classes.
Using 2000 files for validation.
Found 200 files belonging to 2 classes.<jupyter_text>Visualize data* Visualize first batch of size 8* If **class_names** not specified, label-0 is bargraph and label-1 is doctable* Print shape of batch size - **(Batch_size, Height, Width, RGB)**<jupyter_code>plt.figure(figsize=(10, 10))
for images, labels in train_dataset.take(1):
    for i in range(8):
        ax = plt.subplot(2, 4, i + 1)
        plt.imshow(images[i].numpy().astype("uint8"))
        plt.title(int(labels[i]))
        plt.axis("off")
class_names = train_dataset.class_names
print(class_names)
plt.figure(figsize=(10, 10))
for images, labels in train_dataset.take(1):
    for i in range(8):
        ax = plt.subplot(2, 4, i + 1)
        plt.imshow(images[i].numpy().astype("uint8"))
        plt.title(class_names[labels[i]])
        plt.axis("off")
for image_batch, labels_batch in train_dataset:
    print(image_batch.shape)
    print(labels_batch.shape)
    break<jupyter_output>(8, 224, 224, 3)
(8,)<jupyter_text>Data augmentation and standardize data**Data augmentation is inactive at test time**, so the input samples will only be augmented **during fit()**, not when calling evaluate() or predict().* The RGB channel values are in the [0, 255] range* Standardize values to be in the [0, 1] range by using **tf.keras.layers.Rescaling*** Don't have a large image dataset - a good practice to artificially introduce sample diversity by applying random, yet realistic, transformations to the training images, such as rotation and horizontal flipping* Helps expose the model to different aspects of the training data and reduce overfitting* Rescale pixel values - use **tf.keras.applications.MobileNetV2** as base model. This model expects pixel values in [-1, 1]<jupyter_code>data_augmentation = tf.keras.Sequential([
  tf.keras.layers.Resizing(img_height, img_width),
  tf.keras.layers.Rescaling(1./255),
  #tf.keras.layers.Rescaling(1./127.5, offset=-1),
  tf.keras.layers.RandomFlip('horizontal'),
  tf.keras.layers.RandomRotation(0.2),
  
])
for image, _ in train_dataset.take(1):
    plt.figure(figsize=(10, 10))
    first_image = image[1]
    for i in range(8):
        ax = plt.subplot(2, 4, i + 1)
        augmented_image = data_augmentation(tf.expand_dims(first_image, 0))
        #plt.imshow(augmented_image[0] / 255)
        plt.imshow(augmented_image[0])
        plt.axis('off')<jupyter_output><empty_output><jupyter_text>Configure dataset for performance* Two important methods to use when loading data* Use buffered prefetching to yield data from disk without having I/O become blocking* **Dataset.cache** - keeps the images in memory after they're loaded off disk during the first epoch. This will ensure the dataset does not become a bottleneck while training your model. If your dataset is too large to fit into memory, you can also use this method to create a performant on-disk cache.* **Dataset.prefetch** - overlaps data preprocessing and model execution while training.<jupyter_code>AUTOTUNE = tf.data.AUTOTUNE

# train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)
train_dataset = train_dataset.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_dataset = val_dataset.cache().prefetch(buffer_size=AUTOTUNE)
test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>\*\*\*\*\* Feature Extractor \*\*\*\*\** **Feature Extractor : Not correct usecase** - pre-trained model on ImageNet dataset and our dataset is [bargparh, doctable] dataset* **Correct usecase** - Fine-tune pre-trained model on [bargparh, doctable] dataset Feature extraction* Freeze the convolutional base created from the previous step and to use as a feature extractor* Add a classifier on top of it and train the top-level classifier Create base model from the pre-trained convnets* Base model - ResNet50 model* Pre-trained on the ImageNet dataset, a large dataset consisting of 1.4M images and 1000 classes* Base of knowledge will help to classify bargraph and doctable from specific dataset. To Do:* Instantiate a ResNet50 model pre-loaded with weights trained on ImageNet* Ideal for feature extraction - **include_top=False** ensures that the last layer of ResNet50 model will not be loaded* If **weights=None**, then the weights would be initialized randomly (in this case, not be performing transfer learning)  Freeze the convolutional base* Important- freeze the convolutional base before you compile and train the model* By setting the trainable attribute to False (**resnet_50.trainable = False**), ensure that the original (ImageNet) weights of the model will remain constant Important note about BatchNormalization layers* **tf.keras.layers.BatchNormalization** - precautions should be taken in the context of fine-tuning* If **layer.trainable = False**, the **BatchNormalization** layer will run in inference mode, and will not update its mean and variance statistics*  If unfreeze a model that contains **BatchNormalization** layers in order to do fine-tuning, keep the **BatchNormalization** layers in inference mode by passing **training = False** when calling the base model* Otherwise, the updates applied to the non-trainable weights will destroy what the model has learned* Reason to use Sigmoid activation function on bargraph and doctable dataset-Reference: https://keras.io/examples/vision/image_classification_from_scratch/if num_classes == 2:        activation = "sigmoid"    units = 1else:        activation = "softmax"    units = num_classes  Important Note:Reference: https://stackoverflow.com/questions/47312219/what-is-the-definition-of-a-non-trainable-parameterIn keras, non-trainable parameters (as shown in model.summary()) means the number of weights that are not updated during training with backpropagation.There are mainly two types of non-trainable weights:* The ones that you have chosen to keep constant when training. This means that keras won't update these weights during training at all.* The ones that work like statistics in BatchNormalization layers. They're updated with mean and variance, but they're not "trained with backpropagation".Weights are the values inside the network that perform the operations and can be adjusted to result in what we want. The backpropagation algorithm changes the weights towards a lower error at the end.**By default, all weights in a keras model are trainable.**When you create layers, internally it creates its own weights and they're trainable. (The backpropagation algorithm will update these weights)When you make them untrainable, the algorithm will not update these weights anymore. This is useful, for instance, when you want a convolutional layer with a specific filter, like a Sobel filter, for instance. You don't want the training to change this operation, so these weights/filters should be kept constant.There is a lot of other reasons why you might want to make weights untrainable.Changing parameters:For deciding whether weights are trainable or not, you take layers from the model and set trainable:**model.get_layer(layerName).trainable = False or True**This must be done before compilation. Compile the model* Compile the model before training it* Since there are two classes, use the **tf.keras.losses.BinaryCrossentropy** loss with **from_logits=True** since the model provides a linear output<jupyter_code># Initialize the Pretrained Model
fc_resnet_50 = ResNet50(weights='imagenet',
                      input_shape=(img_height, img_width, 3))
print(fc_resnet_50.input_shape)
print("-----------------------")
print(fc_resnet_50.summary())
# Initialize the Pretrained Model
resnet_50 = ResNet50(weights='imagenet',
                      input_shape=(img_height, img_width, 3),
                      include_top=False,)

# Set this parameter to make sure it's not being trained
resnet_50.trainable = False

# Set the input layer
inputs = tf.keras.Input(shape=(img_height, img_width, 3))
x = data_augmentation(inputs)

# Set the feature extractor layer
x = resnet_50(x, training=False)

# Set the pooling layer and Dropout layer
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.2)(x)

# Set the final layer with sigmoid activation function
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)

# Create the new model object
model = tf.keras.Model(inputs=inputs, outputs=outputs, name="uos_imgclsfr_model")

# Compile it
base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=base_learning_rate),
             loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
             metrics=['accuracy'])

print(resnet_50.summary())
print(model.summary())
image_batch, label_batch = next(iter(train_dataset))
feature_batch = resnet_50(image_batch)
print(feature_batch.shape)
loss0, accuracy0 = model.evaluate(val_dataset)
print("initial loss: {:.2f}".format(loss0))
print("initial accuracy: {:.2f}".format(accuracy0))
epochs = 10

checkpoint_path = "model_weight/fe_cp_{epoch:04d}.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1,
                                                 save_freq=125*batch_size)
model.save_weights(checkpoint_path.format(epoch=0))

history = model.fit(train_dataset,
                    epochs=epochs,
                    callbacks=cp_callback,
                    validation_data=val_dataset)<jupyter_output>Epoch 1/10<jupyter_text>Learning curves* Check at the learning curves of the training and validation accuracy/loss when using the ResNet50 base model as a fixed feature extractor* Overfitting as val_loss is higher than train_loss<jupyter_code>acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

print("acc: ",acc)
print("\n val_acc: ", val_acc)
print("\n loss: ", loss)
print("\n val_loss: ", val_loss)
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.ylabel('Accuracy')
plt.ylim([min(plt.ylim()),1])
plt.title('Training and Validation Accuracy')

plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.ylabel('Cross Entropy')
plt.ylim([0,1.0])
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation and prediction* Verify the performance of the model on new data using test set* Use fine-tuned model to predict if image is a bargraph or doctable<jupyter_code>loss, accuracy = model.evaluate(test_dataset)
print('Test Loss :', loss)
print('Test accuracy :', accuracy)
# Retrieve a batch of images from the test set
image_batch, label_batch = test_dataset.as_numpy_iterator().next()
predictions = model.predict_on_batch(image_batch).flatten()

# Apply a sigmoid since our model returns logits
predictions = tf.nn.sigmoid(predictions)
predictions = tf.where(predictions < 0.5, 0, 1)

print('Predictions:\n', predictions.numpy())
print('Labels:\n', label_batch)
plt.figure(figsize=(10, 10))
for i in range(8):
    ax = plt.subplot(2, 4, i + 1)
    plt.imshow(image_batch[i].astype("uint8"))
    plt.title(class_names[predictions[i]])
    plt.axis("off")<jupyter_output><empty_output><jupyter_text>\*\*\*\*\* Fine-tuning \*\*\*\*\* Fine-tuning* In feature extraction experiment - only training a few layers on top of a ResNet50 base model* Weights of the pre-trained network were not updated during training* Increase performance - train (or "fine-tune") the weights of the top layers of the pre-trained model alongside the training of the classifier you added* The training process will force the weights to be tuned from generic feature maps to features associated specifically with the dataset* Should try to fine-tune a small number of top layers rather than the whole ResNet50 model* **In most CNN, the higher layers are more specific to the dataset on which the model was trained*** **First few layers learn very simple and generic features that generalize to almost all types of images*** **Goal of fine-tuning** - adapt these specialized features to work with new dataset, rather than overwrite generic learning Un-freeze the top layers of the model* Unfreeze the **resnet_50** and set the bottom layers to be un-trainable* Recompile the model (necessary for these changes to take effect), and resume training<jupyter_code># Initialize the Pretrained Model
ft_resnet_50 = ResNet50(weights='imagenet',
                      input_shape=(img_height, img_width, 3),
                      include_top=False,)

ft_resnet_50.trainable = True

# Let's take a look to see how many layers are in the base model
print("Number of layers in the base model: ", len(ft_resnet_50.layers))

# Set the input layer
ft_inputs = tf.keras.Input(shape=(img_height, img_width, 3))
x = data_augmentation(ft_inputs)

# Set the feature extractor layer
x = ft_resnet_50(x)

# Set the pooling layer and Dropout layer
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.2)(x)

# Set the final layer with sigmoid activation function
ft_outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)

# Create the new model object
ft_model = tf.keras.Model(inputs=ft_inputs, outputs=ft_outputs, name="tf_ft_uos_model")

# Fine-tune from this layer onwards
fine_tune_at = 100

# Freeze all the layers before the `fine_tune_at` layer
for layer in ft_resnet_50.layers[:fine_tune_at]:
    layer.trainable = False

base_learning_rate = 0.0001
ft_model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer = tf.keras.optimizers.RMSprop(learning_rate=base_learning_rate/10),
              metrics=['accuracy'])
print(ft_model.summary())

import os

checkpoint_path = "model_weight/ft_cp_{epoch:04d}.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1,
                                                 save_freq=125*batch_size)
ft_model.save_weights(checkpoint_path.format(epoch=0))

epochs =  10

history_fine = ft_model.fit(train_dataset,
                         epochs=epochs,
                         initial_epoch=0, # history.epoch[-1] from Feature Extractor
                         validation_data=val_dataset,
                         callbacks=[cp_callback])<jupyter_output>Number of layers in the base model:  175
Model: "tf_ft_uos_model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_4 (InputLayer)        [(None, 224, 224, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 224, 224, 3)       0         
                                                                 
 resnet50 (Functional)       (None, 7, 7, 2048)        23587712  
                                                                 
 global_average_pooling2d_1   (None, 2048)             0         
 (GlobalAveragePooling2D)                                        
                                                                 
 dropout_1 (Dropout)         (None, 2048)              0         
                                                                 
 dense_1 ([...]<jupyter_text>Fine-tune Learning curves* Check at the learning curves of the training and validation accuracy/loss when fine-tuning the last few layers of the ResNet50 base model and training the classifier on top of it* New training set is relatively small* After fine tuning the model nearly reaches 98% accuracy on the validation set<jupyter_code>print(ft_model.summary())
image_batch, label_batch = next(iter(train_dataset))
feature_batch = ft_model(image_batch)
print(feature_batch.shape)
acc = history_fine.history['accuracy']
val_acc = history_fine.history['val_accuracy']

loss = history_fine.history['loss']
val_loss = history_fine.history['val_loss']

print("acc: ",acc)
print("\n val_acc: ", val_acc)
print("\n loss: ", loss)
print("\n val_loss: ", val_loss)
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.ylim([0.8, 1])
plt.plot([epochs-1,epochs-1],
          plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.ylim([0, 1.0])
plt.plot([epochs-1,epochs-1],
         plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation and prediction* Verify the performance of the model on new data using test set* Use fine-tuned model to predict if image is a bargraph or doctable<jupyter_code>loss, accuracy = ft_model.evaluate(test_dataset)
print('Test Loss :', loss)
print('Test accuracy :', accuracy)
# Retrieve a batch of images from the test set
image_batch, label_batch = test_dataset.as_numpy_iterator().next()
predictions = ft_model.predict_on_batch(image_batch).flatten()

# Apply a sigmoid since our model returns logits
predictions = tf.nn.sigmoid(predictions)
predictions = tf.where(predictions < 0.5, 0, 1)

print('Predictions:\n', predictions.numpy())
print('Labels:\n', label_batch)
plt.figure(figsize=(10, 10))
for i in range(8):
    ax = plt.subplot(2, 4, i + 1)
    plt.imshow(image_batch[i].astype("uint8"))
    plt.title(class_names[predictions[i]])
    plt.axis("off")<jupyter_output><empty_output><jupyter_text>Summary* **Using a pre-trained model for feature extraction:** When working with a small dataset, it is a common practice to take advantage of features learned by a model trained on a larger dataset in the same domain. This is done by instantiating the pre-trained model and adding a fully-connected classifier on top. The pre-trained model is "frozen" and only the weights of the classifier get updated during training. In this case, the convolutional base extracted all the features associated with each image and you just trained a classifier that determines the image class given that set of extracted features.* **Fine-tuning a pre-trained model:** To further improve performance, one might want to repurpose the top-level layers of the pre-trained models to the new dataset via fine-tuning. In this case, you tuned your weights such that your model learned high-level features specific to the dataset. This technique is usually recommended when the training dataset is large and very similar to the original dataset that the pre-trained model was trained on.Reference: * https://www.tensorflow.org/tutorials/images/classification* https://www.tensorflow.org/tutorials/images/transfer_learning* https://www.earthdatascience.org/courses/intro-to-earth-data-science/file-formats/use-text-files/format-text-with-markdown-jupyter-notebook/* https://github.com/miladfa7/Image-Classification-Transfer-Learning/blob/master/ResNet_image_classification.ipynb* https://towardsdatascience.com/transfer-learning-for-image-classification-using-tensorflow-71c359b56673<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10003497726477789, 'fraction_numerical': 0.016876530255334032, 'mean_word_length': 4.095344174649142, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 48, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '390482', 'license': 'MIT', 'n_tokens_mistral': 6977, 'n_tokens_neox': 6539, 'n_words': 2089, 'path': 'TfKeras/src/GoogleColab_TF_Img_Classifier.ipynb', 'repo_name': 'agcdke/UOS', 'size': '22872'}