<jupyter_start><jupyter_text>Transfer Learning with TensorFlow Part 2: Fine-tuningIn the previous notebook, we covered transfer learning feature extraction, now it's time to learn about a new kind of transfer learning: fine-tuning.See full course materials on GitHub: https://github.com/mrdbourke/tensorflow-deep-learning/<jupyter_code># Check if we're using a GPU
!nvidia-smi<jupyter_output>Thu Feb 18 04:00:02 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39       Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   41C    P8    10W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-------[...]<jupyter_text>Creating helper functionsIn previous notebooks, we've created a bunch of helper functions, now we could rewrite them all, however, this is tedious.So, it's a good idea to put functions you'll want to use again in a script you can download and import into your notebooks (or elsewhere).We've done this for some of the functions we've used previously here: https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py<jupyter_code>!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py
# Import helper functions we're going to use in this notebook
from helper_functions import create_tensorboard_callback, plot_loss_curves, unzip_data, walk_through_dir<jupyter_output><empty_output><jupyter_text>> 🔑 **Note:** If you're running this notebook in Google Colab, when it times out Colab will delete `helper_functions.py`, so you'll have to redownload it if you want access to your helper functions.  Let's get some dataThis time we're going to see how we can use the pretrained models within `tf.keras.applications` and apply them to our own problem (recognizing images of food).link: https://www.tensorflow.org/api_docs/python/tf/keras/applications<jupyter_code># Get 10% of training data of 10 classes of Food101
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_10_percent.zip

unzip_data("10_food_classes_10_percent.zip")
# Check out how many images and subdirectories are in our dataset
walk_through_dir("10_food_classes_10_percent")
# Create training and test directory paths
train_dir = "10_food_classes_10_percent/train"
test_dir = "10_food_classes_10_percent/test"
import tensorflow as tf
IMG_SIZE = (224, 224)
BATCH_SIZE = 32
train_data_10_percent = tf.keras.preprocessing.image_dataset_from_directory(directory=train_dir,
                                                                            image_size=IMG_SIZE,
                                                                            label_mode="categorical",
                                                                            batch_size=BATCH_SIZE)
test_data = tf.keras.preprocessing.image_dataset_from_directory(directory=test_dir,
                                                                image_size=IMG_SIZE,
                                                                label_mode="categorical",
                                                                batch_size=BATCH_SIZE)
train_data_10_percent
# Check out the class names of our dataset
train_data_10_percent.class_names
# See an example of a batch of data
for images, labels in train_data_10_percent.take(1):
  print(images, labels)<jupyter_output>tf.Tensor(
[[[[2.02459183e+02 1.87459183e+02 1.56459183e+02]
   [2.05214279e+02 1.90811218e+02 1.56617340e+02]
   [2.05071426e+02 1.92071426e+02 1.57994888e+02]
   ...
   [6.77755661e+01 4.94847183e+01 5.42859316e+00]
   [7.14489288e+01 4.88520012e+01 3.71423340e+00]
   [7.50155411e+01 5.10155373e+01 3.84202361e+00]]

  [[2.02877548e+02 1.87877548e+02 1.55683670e+02]
   [2.01137756e+02 1.87132645e+02 1.50418365e+02]
   [2.07841843e+02 1.94056122e+02 1.57841843e+02]
   ...
   [6.97857361e+01 5.12142639e+01 8.92857170e+00]
   [7.18520126e+01 4.88571167e+01 6.56627369e+00]
   [7.78829346e+01 5.38829346e+01 9.07674789e+00]]

  [[2.00704086e+02 1.84704086e+02 1.50984695e+02]
   [1.98729584e+02 1.84515305e+02 1.45229584e+02]
   [2.04000000e+02 1.90494904e+02 1.48739792e+02]
   ...
   [6.96889038e+01 5.09031639e+01 1.09031649e+01]
   [7.45713654e+01 5.25305405e+01 1.14437885e+01]
   [7.37144623e+01 4.94236717e+01 7.36746597e+00]]

  ...

  [[2.11994965e+02 1.93637756e+02 1.60637756e+02]
   [2[...]<jupyter_text>Model 0: Building a transfer learning feature extraction model using the Keras Functional APIThe sequential API is straight-forward, it runs our layers in sequential order.But the functional API gives us more flexibility with our models - https://www.tensorflow.org/guide/keras/functional<jupyter_code># 1. Create base model with tf.keras.applications
base_model = tf.keras.applications.EfficientNetB0(include_top=False)

# 2. Freeze the base model (so the underlying pre-trained patterns aren't updated during training)
base_model.trainable = False 

# 3. Create inputs into our model
inputs = tf.keras.layers.Input(shape=(224, 224, 3), name="input_layer")

# 4. If using a model like ResNet50V2 you will need to normalize inputs (you don't have to for EfficientNet(s))
# x = tf.keras.layers.experimental.preprocessing.Rescaling(1./255)(inputs)

# 5. Pass the inputs to the base_model
x = base_model(inputs)
print(f"Shape after passing inputs through base model: {x.shape}")

# 6. Average pool the outputs of the base model (aggregate all the most important infromation, reduce number of computations)
x = tf.keras.layers.GlobalAveragePooling2D(name="global_average_pooling_layer")(x)
print(f"Shape after GlobalAveragePooling2D: {x.shape}")

# 7. Create the output activation layer
outputs = tf.keras.layers.Dense(10, activation="softmax", name="output_layer")(x)

# 8. Combine the inputs with the outputs into a model
model_0 = tf.keras.Model(inputs, outputs)

# 9. Compile the model
model_0.compile(loss="categorical_crossentropy",
                optimizer=tf.keras.optimizers.Adam(),
                metrics=["accuracy"])

# 10. Fit the model and save its history
history_10_percent = model_0.fit(train_data_10_percent,
                                 epochs=5,
                                 steps_per_epoch=len(train_data_10_percent),
                                 validation_data=test_data,
                                 validation_steps=int(0.25 * len(test_data)),
                                 callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                        experiment_name="10_percent_feature_extraction")])
# Evalaute on the full test dataset
model_0.evaluate(test_data)
# Check the layers in our base model
for layer_number, layer in enumerate(base_model.layers):
  print(layer_number, layer.name)
# How about we get a summary of the base model?
base_model.summary()
# How about a summary of our whole model?
model_0.summary()
# Check out our model's training curves
plot_loss_curves(history_10_percent)<jupyter_output><empty_output><jupyter_text>Getting a feature vector from a trained modelLet's demonstrate the Global Average Pooling 2D layer...We have a tensor after our model goes through `base_model` of shape (None, 7, 7, 1280).But then when it passes through GlobalAveragePooling2D, it turns into (None, 1280).Let's use a similar shaped tensor of (1, 4, 4, 3) and then pass it to GlobalAveragePooling2D.<jupyter_code># Define the input shape
input_shape = (1, 4, 4, 3)

# Create a random tensor
tf.random.set_seed(42)
input_tensor = tf.random.normal(input_shape)
print(f"Random input tensor:\n {input_tensor}\n")

# Pass the random tensor through a global average pooling 2D layer
global_average_pooled_tensor = tf.keras.layers.GlobalAveragePooling2D()(input_tensor)
print(f"2D global average pooled random tensor:\n {global_average_pooled_tensor}\n")

# Check the shape of the different tensors
print(f"Shape of input tensor: {input_tensor.shape}")
print(f"Shape of Global Average Pooled 2D tensor: {global_average_pooled_tensor.shape}")
# Let's replicate the GlobalAveragePool2D layer
tf.reduce_mean(input_tensor, axis=[1, 2])<jupyter_output><empty_output><jupyter_text>> 🛠 **Practice:** Try to do the same with the above two cells but this time use `GlobalMaxPool2D`... and see what happens.> 🔑 **Note:** One of the reasons feature extraction transfer learning is named how it is is because what often happens is pretrained model outputs a **feature vector** (a long tensor of numbers which represents the learned representation of the model on a particular sample, in our case, this is the output of the `tf.keras.layers.GlobalAveragePooling2D()` layer) which can then be used to extract patterns out of for our own specifc problem.  Running a series of transfer learning experimentsWe've seen the incredible results transfer learning can get with only 10% of the training data, but how does it go with 1% of the training data... how about we set up a bunch of experiments to find out:1. `model_1` - use feature extraction transfer learning with 1% of the training data with data augmentation 2. `model_2` - use feature extraction transfer learning with 10% of the training with data augmentaton 3. `model_3` - use fine-tuning transfer learning on 10% of the training data with data augmentation4. `model_4` - use fine-tuning transfer learning on 100% of the training data with data augmentation > 🔑 **Note:** throughout all experiments the same test dataset will be used to evaluate our model... this ensures consistency across evaluation metrics.  Getting and preprocessing data for model_1<jupyter_code># Download and unzip data - preprocessed from Food101
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_1_percent.zip 
unzip_data("10_food_classes_1_percent.zip")
# Create training and test dirs
train_dir_1_percent = "10_food_classes_1_percent/train"
test_dir = "10_food_classes_1_percent/test"
# How many images are we working with?
walk_through_dir("10_food_classes_1_percent")
# Setup data loaders 
IMG_SIZE = (224, 224)
train_data_1_percent = tf.keras.preprocessing.image_dataset_from_directory(train_dir_1_percent,
                                                                           label_mode="categorical",
                                                                           image_size=IMG_SIZE,
                                                                           batch_size=BATCH_SIZE) # default = 32

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode="categorical",
                                                                image_size=IMG_SIZE,
                                                                batch_size=BATCH_SIZE)<jupyter_output>Found 70 files belonging to 10 classes.
Found 2500 files belonging to 10 classes.<jupyter_text>Adding data augmentation right into the modelTo add data augmentation right into our models, we can use the layers inside:* `tf.keras.layers.experimental.preprocessing()`We can see the benefits of doing this within the TensorFlow Data augmentation documentation: https://www.tensorflow.org/tutorials/images/data_augmentationuse_keras_preprocessing_layers Off the top our of heads, after reading the docs, the benefits of using data augmentation inside the model are:* Preprocessing of imges (augmenting them) happens on the GPU (much faster) rather than the CPU.* Iamge data augmentation only happens during training, so we can still export our whole model and use it elsewhere.<jupyter_code>import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing

# Create data augmentation stage with horizontal flipping, rotations, zooms, etc
data_augmentation = keras.Sequential([
  preprocessing.RandomFlip("horizontal"),
  preprocessing.RandomRotation(0.2),
  preprocessing.RandomZoom(0.2),
  preprocessing.RandomHeight(0.2),
  preprocessing.RandomWidth(0.2),
  # preprocessing.Rescale(1./255) # Keep for models like ResNet50V2 but EfficientNet's having resclaing built-in
], name="data_augmentation")<jupyter_output><empty_output><jupyter_text>Visualize our data augmentation layer (and see what happens to our data)<jupyter_code># View a random image and compare it to its augmented version
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import random
target_class = random.choice(train_data_1_percent.class_names)
target_dir = "10_food_classes_1_percent/train/" + target_class
random_image = random.choice(os.listdir(target_dir))
random_image_path = target_dir + "/" + random_image

# Read and plot in the random image 
img = mpimg.imread(random_image_path)
plt.imshow(img)
plt.title(f"Original random image from class: {target_class}")
plt.axis(False);

# Now let's plot our augmented random image
augmented_img = data_augmentation(tf.expand_dims(img, axis=0))
plt.figure()
plt.imshow(tf.squeeze(augmented_img)/255.)
plt.title(f"Augmented random image from class: {target_class}")
plt.axis(False);
# print(augmented_img)<jupyter_output><empty_output><jupyter_text>Model 1: Feature extraction transfer learning on 1% of the data with data augmentation<jupyter_code># Setup input shape and base model, freezing the base model layers
input_shape = (224, 224, 3)
base_model = tf.keras.applications.EfficientNetB0(include_top=False)
base_model.trainable = False

# Create input layer
inputs = layers.Input(shape=input_shape, name="input_layer")

# Add in data augmentation Sequential model as a layer
x = data_augmentation(inputs)

# Give base_model the inputs (after augmentation) and don't train it
x = base_model(x, training=False)

# Pool output features of the base model
x = layers.GlobalAveragePooling2D(name="global_average_pooling_layer")(x)

# Put a dense layer on as the output
outputs = layers.Dense(10, activation="softmax", name="output_layer")(x)

# Make a model using the inputs and outputs
model_1 = keras.Model(inputs, outputs)

# Compile the model
model_1.compile(loss="categorical_crossentropy",
                optimizer=tf.keras.optimizers.Adam(),
                metrics=["accuracy"])

# Fit the model
history_1_percent = model_1.fit(train_data_1_percent,
                                epochs=5,
                                steps_per_epoch=len(train_data_1_percent),
                                validation_data=test_data,
                                validation_steps=int(0.25 * len(test_data)),
                                # Track model training logs
                                callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                       experiment_name="1_percent_data_aug")])
# Check out a model summary
model_1.summary()
# Evalute on the full test dataset
results_1_percent_data_aug = model_1.evaluate(test_data)
results_1_percent_data_aug
# How do the model with 1% of the and data augmentation loss curves look?
plot_loss_curves(history_1_percent)<jupyter_output><empty_output><jupyter_text>Model 2: feature extraction transfer learning model with 10% of data and data augmentation<jupyter_code># Get 10% of data... (uncomment if you don't have it)
# !wget https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_10_percent.zip
# unzip_data(10_food_classes_10_percent)

train_dir_10_percent = "10_food_classes_10_percent/train"
test_dir = "10_food_classes_10_percent/test"
# How many images are in our directories?
walk_through_dir("10_food_classes_10_percent")
# Set data inputs
import tensorflow as tf
IMG_SIZE = (224, 224)
train_data_10_percent = tf.keras.preprocessing.image_dataset_from_directory(train_dir_10_percent,
                                                                            label_mode="categorical",
                                                                            image_size=IMG_SIZE)
test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode="categorical",
                                                                image_size=IMG_SIZE)
# Create model 2 with data augmentation built in
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing
from tensorflow.keras.models import Sequential

# Build data augmentation layer
data_augmentation = Sequential([
  preprocessing.RandomFlip("horizontal"),
  preprocessing.RandomHeight(0.2),
  preprocessing.RandomWidth(0.2),
  preprocessing.RandomZoom(0.2),
  preprocessing.RandomRotation(0.2),
  # preprocessing.Rescaling(1./255) # if you're using a model such as ResNet50V2, you'll need to rescale your data, efficientnet has rescaling built-in
], name="data_augmentation")

# Setup the input shape to our model
input_shape = (224, 224, 3)

# Create a frozen base model (also called the backbone)
base_model = tf.keras.applications.EfficientNetB0(include_top=False)
base_model.trainable = False

# Create the inputs and outputs (including the layers in between)
inputs = layers.Input(shape=input_shape, name="input_layer")
x = data_augmentation(inputs) # augment our training images (augmentation doesn't occur on test data)
x = base_model(x, training=False) # pass augmented images to base model but keep it in inference mode, this also insures batchnorm layers don't get updated - https://keras.io/guides/transfer_learning/#build-a-model
x = layers.GlobalAveragePooling2D(name="global_average_pooling_2D")(x)
outputs = layers.Dense(10, activation="softmax", name="output_layer")(x)
model_2 = tf.keras.Model(inputs, outputs)

# Compile
model_2.compile(loss="categorical_crossentropy",
                optimizer=tf.keras.optimizers.Adam(),
                metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>Creating a ModelCheckpoint callbackThe ModelCheckpoint callback intermediately saves our model (the full model or just the weights) during training. This is useful so we can come and start where we left off.<jupyter_code># Set checkpoint path
checkpoint_path = "ten_percent_model_checkpoints_weights/checkpoint.ckpt"

# Create a ModelCheckpoint callback that saves the model's weights only
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                         save_weights_only=True,
                                                         save_best_only=False,
                                                         save_freq="epoch", # save every epoch
                                                         verbose=1)<jupyter_output><empty_output><jupyter_text>Fit model 2 passing in the ModelCheckpoint callback<jupyter_code># Fit the model saving checkpoints every epoch
initial_epochs = 5
history_10_percent_data_aug = model_2.fit(train_data_10_percent,
                                          epochs=initial_epochs,
                                          validation_data=test_data,
                                          validation_steps=int(0.25 * len(test_data)),
                                          callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                                 experiment_name="10_percent_data_aug"),
                                                     checkpoint_callback])
# What were model_0 results?
model_0.evaluate(test_data)
# Check model_2 results on all test_data
results_10_percent_data_aug = model_2.evaluate(test_data)
results_10_percent_data_aug
# Plot model loss curves
plot_loss_curves(history_10_percent_data_aug)<jupyter_output><empty_output><jupyter_text>Loading in checkpointed weightsLoading in checkpointed weights returns a model to a specific checkpoint.<jupyter_code># Load in saved model weights and evaluate model
model_2.load_weights(checkpoint_path)
# Evalaute model_2 with loaded weights
loaded_weights_model_results = model_2.evaluate(test_data)
# If the results from our previously evaluated model_2 match the loaded weights, everything has worked!
results_10_percent_data_aug == loaded_weights_model_results
results_10_percent_data_aug
loaded_weights_model_results
# Check to see if loaded model results are very close to our previous non-loaded model results
import numpy as np
np.isclose(np.array(results_10_percent_data_aug), np.array(loaded_weights_model_results))
# Check the difference between the two results
print(np.array(results_10_percent_data_aug) - np.array(loaded_weights_model_results))<jupyter_output>[-5.96046448e-08  0.00000000e+00]<jupyter_text>Model 3: Fine-tuning an existing model on 10% of the data> 🔑 **Note:** Fine-tuning usually works best *after* training a feature extraction model for a few epochs with large amounts of custom data.<jupyter_code># Layers in loaded model
model_2.layers
# Are these layers trainable?
for layer in model_2.layers:
  print(layer, layer.trainable)
# What layers are in our base_model (EfficientNetB0) and are they trainable?
for i, layer in enumerate(model_2.layers[2].layers):
  print(i, layer.name, layer.trainable)
# How many trainable varialbes are in our base model?
print(len(model_2.layers[2].trainable_variables))
# To begin fine-tuning, let's start by setting the last 10 layers of our base_model.trainable = True
base_model.trainable = True

# Freeze all layers except for the last 10
for layer in base_model.layers[:-10]:
  layer.trainable = False

# Recompile (we have to recompile our models every time we make a change)
model_2.compile(loss="categorical_crossentropy",
                optimizer=tf.keras.optimizers.Adam(lr=0.0001), # when fine-tuning you typically want to lower the learning rate by 10x*
                metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>> 🔑 **Note:** When using fine-tuning it's best practice to lower your learning rate by some amount. How much? This is a hyperparameter you can tune. But a good rule of thumb is at least 10x (though different sources will claim other values). A good resource for information on this is the ULMFiT paper: https://arxiv.org/abs/1801.06146<jupyter_code># Check which layers are tunable (trainable)
for layer_number, layer in enumerate(model_2.layers[2].layers):
  print(layer_number, layer.name, layer.trainable)
# Now we've unfrozen some of the layers closer to the top, how many trainable variables are there?
print(len(model_2.trainable_variables))
# Fine tune for another 5 epochs
fine_tune_epochs = initial_epochs + 5

# Refit the model (same as model_2 except with more trainable layers)
history_fine_10_percent_data_aug = model_2.fit(train_data_10_percent,
                                               epochs=fine_tune_epochs,
                                               validation_data=test_data,
                                               validation_steps=int(0.25 * len(test_data)),
                                               initial_epoch=history_10_percent_data_aug.epoch[-1], # start training from previous last epoch
                                               callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                                      experiment_name="10_percent_fine_tune_last_10")])
# Evaluate the fine-tuned model (model_3 which is actualy model_2 fine-tuned for another 5 epochs)
results_fine_tune_10_percent = model_2.evaluate(test_data)
# Check out the loss curves of our fine-tuned model
plot_loss_curves(history_fine_10_percent_data_aug)<jupyter_output><empty_output><jupyter_text>The `plot_loss_curves` function works great with models which have only been fit once, however, we want something to compare one series of running `fit()` with another (e.g. before and after fine-tuning).<jupyter_code># Let's create a function to compare training histories
def compare_historys(original_history, new_history, initial_epochs=5):
  """
  Compares two TensorFlow History objects.
  """
  # Get original history measurements
  acc = original_history.history["accuracy"]
  loss = original_history.history["loss"]

  val_acc = original_history.history["val_accuracy"]
  val_loss = original_history.history["val_loss"]

  # Combine original history metrics with new_history metrics
  total_acc = acc + new_history.history["accuracy"]
  total_loss = loss + new_history.history["loss"]

  total_val_acc = val_acc + new_history.history["val_accuracy"]
  total_val_loss = val_loss + new_history.history["val_loss"]

  # Make plot for accuracy
  plt.figure(figsize=(8, 8))
  plt.subplot(2, 1, 1)
  plt.plot(total_acc, label="Training Accuracy")
  plt.plot(total_val_acc, label="Val Accuracy")
  plt.plot([initial_epochs-1, initial_epochs-1], plt.ylim(), label="Start Fine Tuning")
  plt.legend(loc="lower right")
  plt.title("Training and Validation Accuracy")

  # Make plot for loss
  plt.figure(figsize=(8, 8))
  plt.subplot(2, 1, 2)
  plt.plot(total_loss, label="Training Loss")
  plt.plot(total_val_loss, label="Val Loss")
  plt.plot([initial_epochs-1, initial_epochs-1], plt.ylim(), label="Start Fine Tuning")
  plt.legend(loc="upper right")
  plt.title("Training and Validation Loss")
compare_historys(history_10_percent_data_aug,
                 history_fine_10_percent_data_aug,
                 initial_epochs=5)<jupyter_output><empty_output><jupyter_text>Model 4: Fine-tuning and existing model on all of the data<jupyter_code># Download and unzip 10 classes of Food101 data with all images
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_all_data.zip
unzip_data("10_food_classes_all_data.zip")
# Setup training and test dir
train_dir_all_data = "10_food_classes_all_data/train"
test_dir = "10_food_classes_all_data/test"
# How many images are we working with now?
walk_through_dir("10_food_classes_all_data")
# Setup data inputs
import tensorflow as tf
IMG_SIZE = (224, 224)
train_data_10_classes_full = tf.keras.preprocessing.image_dataset_from_directory(train_dir_all_data,
                                                                                 label_mode="categorical",
                                                                                 image_size=IMG_SIZE)
test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                                label_mode="categorical",
                                                                image_size=IMG_SIZE)<jupyter_output>Found 7500 files belonging to 10 classes.
Found 2500 files belonging to 10 classes.<jupyter_text>The test dataset we've loaded in is the same as what we've been using for previous experiments (all experiments have used the same test dataset).Let's verify this...<jupyter_code># Evaluate model 2 (this is the fine-tuned on 10 percent of data version)
model_2.evaluate(test_data)
results_fine_tune_10_percent<jupyter_output><empty_output><jupyter_text>To train a fine-tuning model (model_4) we need to revert model_2 back to its feature extraction weights.<jupyter_code># Load weights from checkpoint, that way we can fine-tune from 
# the same stage the 10 percent data model was fine-tuned from
model_2.load_weights(checkpoint_path)
# Let's evaluate model_2 now
model_2.evaluate(test_data)
# Check to see if our model_2 has been reverted back to feature extraction results
results_10_percent_data_aug<jupyter_output><empty_output><jupyter_text>Alright, the previous steps might seem quite confusing but all we've done is:1. Trained a feature extraction transfer learning model for 5 epochs on 10% of the data with data augmentation (model_2) and we saved the model's weights using `ModelCheckpoint` callback.2. Fine-tuned the same model on the same 10% of the data for a further 5 epochs with the top 10 layers of the base model unfrozen (model_3).3. Saved the results and training logs each time.4. Reloaded the model from step 1 to do the same steps as step 2 except this time we're going to use all of the data (model_4).<jupyter_code># Check which layers are tunable in the whole model
for layer_number, layer in enumerate(model_2.layers):
  print(layer_number, layer.name, layer.trainable)
# Let's drill into our base_model (efficientnetb0) and see what layers are trainable
for layer_number, layer in enumerate(model_2.layers[2].layers):
  print(layer_number, layer.name, layer.trainable)
# Compile
model_2.compile(loss="categorical_crossentropy",
                optimizer=tf.keras.optimizers.Adam(lr=0.0001),
                metrics=["accuracy"])
# Continue to train and fine-tune the model to our data (100% of training data)
fine_tune_epochs = initial_epochs + 5

history_fine_10_classes_full = model_2.fit(train_data_10_classes_full,
                                           epochs=fine_tune_epochs,
                                           validation_data=test_data,
                                           validation_steps=int(0.25 * len(test_data)),
                                           initial_epoch=history_10_percent_data_aug.epoch[-1],
                                           callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                                  experiment_name="full_10_classes_fine_tune_last_10")])<jupyter_output>Saving TensorBoard log files to: transfer_learning/full_10_classes_fine_tune_last_10/20210218-052607
Epoch 5/10
235/235 [==============================] - 52s 203ms/step - loss: 0.7904 - accuracy: 0.7506 - val_loss: 0.4620 - val_accuracy: 0.8273
Epoch 6/10
235/235 [==============================] - 49s 204ms/step - loss: 0.5858 - accuracy: 0.8077 - val_loss: 0.3926 - val_accuracy: 0.8750
Epoch 7/10
235/235 [==============================] - 47s 200ms/step - loss: 0.5134 - accuracy: 0.8331 - val_loss: 0.3676 - val_accuracy: 0.8750
Epoch 8/10
235/235 [==============================] - 45s 191ms/step - loss: 0.4742 - accuracy: 0.8458 - val_loss: 0.3384 - val_accuracy: 0.8865
Epoch 9/10
235/235 [==============================] - 43s 181ms/step - loss: 0.4342 - accuracy: 0.8629 - val_loss: 0.3106 - val_accuracy: 0.8898
Epoch 10/10
235/235 [==============================] - 43s 180ms/step - loss: 0.4138 - accuracy: 0.8708 - val_loss: 0.3509 - val_accuracy: 0.8816<jupyter_text>> 🔑 **Note:** Fine-tuning generally takes longer per epoch (more layers being updated) and in the case of the model we just ran we used 10x more data than before (more patterns to find) so it makes sense training took longer.<jupyter_code># Let's evaluate on all of the test data
results_fine_tune_full_data = model_2.evaluate(test_data)
results_fine_tune_full_data
# How did fine-tuning go with more data?
compare_historys(original_history=history_10_percent_data_aug,
                 new_history=history_fine_10_classes_full,
                 initial_epochs=5)<jupyter_output><empty_output><jupyter_text>Viewing our experiment data on TensorBoard> 🔑 **Note:** Anything you upload to TensorBoard.dev is going to be public. So if you have private data, do not upload.<jupyter_code># View tensorboard logs of transfer learning modelling experiments (should ~4 models)
# Upload TensorBoard dev records
!tensorboard dev upload --logdir ./transfer_learning \
  --name "Transfer Learning Experiments with 10 Food101 Classes" \
  --description "A series of different transfer learning experiments with varying amounts of data and fine-tuning." \
  --one_shot # exits the uploader once its finished uploading<jupyter_output><empty_output><jupyter_text>My TensorBoard experiments are available at: https://tensorboard.dev/experiment/vcySzjmkRkKBLVSdAQMO8g/ If you need to view or delete a previous TensorBoard experiment, you can run the commands below.<jupyter_code># # View all of your uploaded TensorBoard.dev experiments (public)
# !tensorboard dev list
# # To delete an experiment
# !tensorboard dev delete --experiment_id vcySzjmkRkKBLVSdAQMO8g # Change this for the experiment ID you want to delete<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09199246063111814, 'fraction_numerical': 0.0446586003526479, 'mean_word_length': 3.219471523858389, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 95, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 123, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '25875', 'license': 'MIT', 'n_tokens_mistral': 10381, 'n_tokens_neox': 9250, 'n_words': 2760, 'path': 'video_notebooks/05_transfer_learning_in_tensorflow_part_2_fine_tuning_video.ipynb', 'repo_name': 'Ankur099IIT/Tensorflow-and-DeepLearning', 'size': '32894'}