title transfer learning finetuning fchollethttps twitter comfchollet date created 20200415 last modified 20230625 description complete guide to transfer learning finetuning in keras accelerator gpu setup introduction transfer learning consists of taking features learned on one problem and leveraging them on a new similar problem for instance features from a model that has learned to identify racoons may be useful to kickstart a model meant to identify tanukis transfer learning is usually done for tasks where your dataset has too little data to train a fullscale model from scratch the most common incarnation of transfer learning in the context of deep learning is the following workflow 1 take layers from a previously trained model 2 freeze them so as to avoid destroying any of the information they contain during future training rounds 3 add some new trainable layers on top of the frozen layers they will learn to turn the old features into predictions on a new dataset 4 train the new layers on your dataset a last optional step is finetuning which consists of unfreezing the entire model you obtained above or part of it and retraining it on the new data with a very low learning rate this can potentially achieve meaningful improvements by incrementally adapting the pretrained features to the new data first we will go over the keras trainable api in detail which underlies most transfer learning finetuning workflows then we ll demonstrate the typical workflow by taking a model pretrained on the imagenet dataset and retraining it on the kaggle cats vs dogs classification dataset this is adapted from deep learning with pythonhttps www manning combooksdeeplearningwithpython and the 2016 blog post building powerful image classification models using very little datahttps blog keras iobuildingpowerfulimageclassificationmodelsusingverylittledata html freezing layers understanding the trainable attribute layers models have three weight attributes weights is the list of all weights variables of the layer trainableweights is the list of those that are meant to be updated via gradient descent to minimize the loss during training nontrainableweights is the list of those that aren t meant to be trained typically they are updated by the model during the forward pass example the dense layer has 2 trainable weights kernel bias in general all weights are trainable weights the only builtin layer that has nontrainable weights is the batchnormalization layer it uses nontrainable weights to keep track of the mean and variance of its inputs during training to learn how to use nontrainable weights in your own custom layers see the guide to writing new layers from scratchhttps keras ioguidesmakingnewlayersandmodelsviasubclassing example the batchnormalization layer has 2 trainable weights and 2 nontrainable weights layers models also feature a boolean attribute trainable its value can be changed setting layer trainable to false moves all the layer s weights from trainable to nontrainable this is called freezing the layer the state of a frozen layer won t be updated during training either when training with fit or when training with any custom loop that relies on trainableweights to apply gradient updates example setting trainable to false when a trainable weight becomes nontrainable its value is no longer updated during training make a model with 2 layers freeze the first layer keep a copy of the weights of layer1 for later reference train the model check that the weights of layer1 have not changed during training do not confuse the layer trainable attribute with the argument training in layer call which controls whether the layer should run its forward pass in inference mode or training mode for more information see the keras faq https keras iogettingstartedfaqwhatsthedifferencebetweenthetrainingargumentincallandthetrainableattribute recursive setting of the trainable attribute if you set trainable false on a model or on any layer that has sublayers all children layers become nontrainable as well example the typical transferlearning workflow this leads us to how a typical transfer learning workflow can be implemented in keras 1 instantiate a base model and load pretrained weights into it 2 freeze all layers in the base model by setting trainable false 3 create a new model on top of the output of one or several layers from the base model 4 train your new model on your new dataset note that an alternative more lightweight workflow could also be 1 instantiate a base model and load pretrained weights into it 2 run your new dataset through it and record the output of one or several layers from the base model this is called feature extraction 3 use that output as input data for a new smaller model a key advantage of that second workflow is that you only run the base model once on your data rather than once per epoch of training so it s a lot faster cheaper an issue with that second workflow though is that it doesn t allow you to dynamically modify the input data of your new model during training which is required when doing data augmentation for instance transfer learning is typically used for tasks when your new dataset has too little data to train a fullscale model from scratch and in such scenarios data augmentation is very important so in what follows we will focus on the first workflow here s what the first workflow looks like in keras first instantiate a base model with pretrained weights python basemodel keras applications xception weights imagenet load weights pretrained on imagenet inputshape150 150 3 includetopfalse do not include the imagenet classifier at the top then freeze the base model python basemodel trainable false create a new model on top python inputs keras inputshape150 150 3 we make sure that the basemodel is running in inference mode here by passing trainingfalse this is important for finetuning as you will learn in a few paragraphs x basemodelinputs trainingfalse convert features of shape basemodel outputshape1 to vectors x keras layers globalaveragepooling2dx a dense classifier with a single unit binary classification outputs keras layers dense1x model keras modelinputs outputs train the model on new data python model compileoptimizerkeras optimizers adam losskeras losses binarycrossentropyfromlogitstrue metricskeras metrics binaryaccuracy model fitnewdataset epochs20 callbacks validationdata finetuning once your model has converged on the new data you can try to unfreeze all or part of the base model and retrain the whole model endtoend with a very low learning rate this is an optional last step that can potentially give you incremental improvements it could also potentially lead to quick overfitting keep that in mind it is critical to only do this step after the model with frozen layers has been trained to convergence if you mix randomlyinitialized trainable layers with trainable layers that hold pretrained features the randomlyinitialized layers will cause very large gradient updates during training which will destroy your pretrained features it s also critical to use a very low learning rate at this stage because you are training a much larger model than in the first round of training on a dataset that is typically very small as a result you are at risk of overfitting very quickly if you apply large weight updates here you only want to readapt the pretrained weights in an incremental way this is how to implement finetuning of the whole base model python unfreeze the base model basemodel trainable true it s important to recompile your model after you make any changes to the trainable attribute of any inner layer so that your changes are take into account model compileoptimizerkeras optimizers adam1e5 very low learning rate losskeras losses binarycrossentropyfromlogitstrue metricskeras metrics binaryaccuracy train endtoend be careful to stop before you overfit model fitnewdataset epochs10 callbacks validationdata important note about compile and trainable calling compile on a model is meant to freeze the behavior of that model this implies that the trainable attribute values at the time the model is compiled should be preserved throughout the lifetime of that model until compile is called again hence if you change any trainable value make sure to call compile again on your model for your changes to be taken into account important notes about batchnormalization layer many image models contain batchnormalization layers that layer is a special case on every imaginable count here are a few things to keep in mind batchnormalization contains 2 nontrainable weights that get updated during training these are the variables tracking the mean and variance of the inputs when you set bnlayer trainable false the batchnormalization layer will run in inference mode and will not update its mean variance statistics this is not the case for other layers in general as weight trainability inferencetraining modes are two orthogonal concepts https keras iogettingstartedfaqwhatsthedifferencebetweenthetrainingargumentincallandthetrainableattribute but the two are tied in the case of the batchnormalization layer when you unfreeze a model that contains batchnormalization layers in order to do finetuning you should keep the batchnormalization layers in inference mode by passing trainingfalse when calling the base model otherwise the updates applied to the nontrainable weights will suddenly destroy what the model has learned you ll see this pattern in action in the endtoend example at the end of this guide an endtoend example finetuning an image classification model on a cats vs dogs dataset to solidify these concepts let s walk you through a concrete endtoend transfer learning finetuning example we will load the xception model pretrained on imagenet and use it on the kaggle cats vs dogs classification dataset getting the data first let s fetch the cats vs dogs dataset using tfds if you have your own dataset you ll probably want to use the utility keras utils imagedatasetfromdirectory to generate similar labeled dataset objects from a set of images on disk filed into classspecific folders transfer learning is most useful when working with very small datasets to keep our dataset small we will use 40 of the original training data 25 000 images for training 10 for validation and 10 for testing reserve 10 for validation and 10 for test these are the first 9 images in the training dataset as you can see they re all different sizes we can also see that label 1 is dog and label 0 is cat standardizing the data our raw images have a variety of sizes in addition each pixel consists of 3 integer values between 0 and 255 rgb level values this isn t a great fit for feeding a neural network we need to do 2 things standardize to a fixed image size we pick 150x150 normalize pixel values between 1 and 1 we ll do this using a normalization layer as part of the model itself in general it s a good practice to develop models that take raw data as input as opposed to models that take alreadypreprocessed data the reason being that if your model expects preprocessed data any time you export your model to use it elsewhere in a web browser in a mobile app you ll need to reimplement the exact same preprocessing pipeline this gets very tricky very quickly so we should do the least possible amount of preprocessing before hitting the model here we ll do image resizing in the data pipeline because a deep neural network can only process contiguous batches of data and we ll do the input value scaling as part of the model when we create it let s resize images to 150x150 using random data augmentation when you don t have a large image dataset it s a good practice to artificially introduce sample diversity by applying random yet realistic transformations to the training images such as random horizontal flipping or small random rotations this helps expose the model to different aspects of the training data while slowing down overfitting let s batch the data and use prefetching to optimize loading speed let s visualize what the first image of the first batch looks like after various random transformations build a model now let s built a model that follows the blueprint we ve explained earlier note that we add a rescaling layer to scale input values initially in the 0 255 range to the 1 1 range we add a dropout layer before the classification layer for regularization we make sure to pass trainingfalse when calling the base model so that it runs in inference mode so that batchnorm statistics don t get updated even after we unfreeze the base model for finetuning freeze the basemodel create new model on top pretrained xception weights requires that input be scaled from 0 255 to a range of 1 1 the rescaling layer outputs inputs scale offset the base model contains batchnorm layers we want to keep them in inference mode when we unfreeze the base model for finetuning so we make sure that the basemodel is running in inference mode here train the top layer do a round of finetuning of the entire model finally let s unfreeze the base model and train the entire model endtoend with a low learning rate importantly although the base model becomes trainable it is still running in inference mode since we passed trainingfalse when calling it when we built the model this means that the batch normalization layers inside won t update their batch statistics if they did they would wreck havoc on the representations learned by the model so far unfreeze the basemodel note that it keeps running in inference mode since we passed trainingfalse when calling it this means that the batchnorm layers will not update their batch statistics this prevents the batchnorm layers from undoing all the training we ve done so far after 10 epochs finetuning gains us a nice improvement here let s evaluate the model on the test dataset setup introduction transfer learning consists of taking features learned on one problem and leveraging them on a new similar problem for instance features from a model that has learned to identify racoons may be useful to kick start a model meant to identify tanukis transfer learning is usually done for tasks where your dataset has too little data to train a full scale model from scratch the most common incarnation of transfer learning in the context of deep learning is the following workflow 1 take layers from a previously trained model 2 freeze them so as to avoid destroying any of the information they contain during future training rounds 3 add some new trainable layers on top of the frozen layers they will learn to turn the old features into predictions on a new dataset 4 train the new layers on your dataset a last optional step is fine tuning which consists of unfreezing the entire model you obtained above or part of it and re training it on the new data with a very low learning rate this can potentially achieve meaningful improvements by incrementally adapting the pretrained features to the new data first we will go over the keras trainable api in detail which underlies most transfer learning fine tuning workflows then we ll demonstrate the typical workflow by taking a model pretrained on the imagenet dataset and retraining it on the kaggle cats vs dogs classification dataset this is adapted from deep learning with python https www manning com books deep learning with python and the 2016 blog post building powerful image classification models using very little data https blog keras io building powerful image classification models using very little data html freezing layers understanding the trainable attribute layers models have three weight attributes weights is the list of all weights variables of the layer trainable_weights is the list of those that are meant to be updated via gradient descent to minimize the loss during training non_trainable_weights is the list of those that aren t meant to be trained typically they are updated by the model during the forward pass example the dense layer has 2 trainable weights kernel bias create the weights in general all weights are trainable weights the only built in layer that has non trainable weights is the batchnormalization layer it uses non trainable weights to keep track of the mean and variance of its inputs during training to learn how to use non trainable weights in your own custom layers see the guide to writing new layers from scratch https keras io guides making_new_layers_and_models_via_subclassing example the batchnormalization layer has 2 trainable weights and 2 non trainable weights create the weights layers models also feature a boolean attribute trainable its value can be changed setting layer trainable to false moves all the layer s weights from trainable to non trainable this is called freezing the layer the state of a frozen layer won t be updated during training either when training with fit or when training with any custom loop that relies on trainable_weights to apply gradient updates example setting trainable to false create the weights freeze the layer when a trainable weight becomes non trainable its value is no longer updated during training make a model with 2 layers freeze the first layer keep a copy of the weights of layer1 for later reference train the model check that the weights of layer1 have not changed during training do not confuse the layer trainable attribute with the argument training in layer __call__ which controls whether the layer should run its forward pass in inference mode or training mode for more information see the keras faq https keras io getting_started faq whats the difference between the training argument in call and the trainable attribute recursive setting of the trainable attribute if you set trainable false on a model or on any layer that has sublayers all children layers become non trainable as well example freeze the outer model all layers in model are now frozen trainable is propagated recursively the typical transfer learning workflow this leads us to how a typical transfer learning workflow can be implemented in keras 1 instantiate a base model and load pre trained weights into it 2 freeze all layers in the base model by setting trainable false 3 create a new model on top of the output of one or several layers from the base model 4 train your new model on your new dataset note that an alternative more lightweight workflow could also be 1 instantiate a base model and load pre trained weights into it 2 run your new dataset through it and record the output of one or several layers from the base model this is called feature extraction 3 use that output as input data for a new smaller model a key advantage of that second workflow is that you only run the base model once on your data rather than once per epoch of training so it s a lot faster cheaper an issue with that second workflow though is that it doesn t allow you to dynamically modify the input data of your new model during training which is required when doing data augmentation for instance transfer learning is typically used for tasks when your new dataset has too little data to train a full scale model from scratch and in such scenarios data augmentation is very important so in what follows we will focus on the first workflow here s what the first workflow looks like in keras first instantiate a base model with pre trained weights python base_model keras applications xception weights imagenet load weights pre trained on imagenet input_shape 150 150 3 include_top false do not include the imagenet classifier at the top then freeze the base model python base_model trainable false create a new model on top python inputs keras input shape 150 150 3 we make sure that the base_model is running in inference mode here by passing training false this is important for fine tuning as you will learn in a few paragraphs x base_model inputs training false convert features of shape base_model output_shape 1 to vectors x keras layers globalaveragepooling2d x a dense classifier with a single unit binary classification outputs keras layers dense 1 x model keras model inputs outputs train the model on new data python model compile optimizer keras optimizers adam loss keras losses binarycrossentropy from_logits true metrics keras metrics binaryaccuracy model fit new_dataset epochs 20 callbacks validation_data fine tuning once your model has converged on the new data you can try to unfreeze all or part of the base model and retrain the whole model end to end with a very low learning rate this is an optional last step that can potentially give you incremental improvements it could also potentially lead to quick overfitting keep that in mind it is critical to only do this step after the model with frozen layers has been trained to convergence if you mix randomly initialized trainable layers with trainable layers that hold pre trained features the randomly initialized layers will cause very large gradient updates during training which will destroy your pre trained features it s also critical to use a very low learning rate at this stage because you are training a much larger model than in the first round of training on a dataset that is typically very small as a result you are at risk of overfitting very quickly if you apply large weight updates here you only want to readapt the pretrained weights in an incremental way this is how to implement fine tuning of the whole base model python unfreeze the base model base_model trainable true it s important to recompile your model after you make any changes to the trainable attribute of any inner layer so that your changes are take into account model compile optimizer keras optimizers adam 1e 5 very low learning rate loss keras losses binarycrossentropy from_logits true metrics keras metrics binaryaccuracy train end to end be careful to stop before you overfit model fit new_dataset epochs 10 callbacks validation_data important note about compile and trainable calling compile on a model is meant to freeze the behavior of that model this implies that the trainable attribute values at the time the model is compiled should be preserved throughout the lifetime of that model until compile is called again hence if you change any trainable value make sure to call compile again on your model for your changes to be taken into account important notes about batchnormalization layer many image models contain batchnormalization layers that layer is a special case on every imaginable count here are a few things to keep in mind batchnormalization contains 2 non trainable weights that get updated during training these are the variables tracking the mean and variance of the inputs when you set bn_layer trainable false the batchnormalization layer will run in inference mode and will not update its mean variance statistics this is not the case for other layers in general as weight trainability inference training modes are two orthogonal concepts https keras io getting_started faq whats the difference between the training argument in call and the trainable attribute but the two are tied in the case of the batchnormalization layer when you unfreeze a model that contains batchnormalization layers in order to do fine tuning you should keep the batchnormalization layers in inference mode by passing training false when calling the base model otherwise the updates applied to the non trainable weights will suddenly destroy what the model has learned you ll see this pattern in action in the end to end example at the end of this guide an end to end example fine tuning an image classification model on a cats vs dogs dataset to solidify these concepts let s walk you through a concrete end to end transfer learning fine tuning example we will load the xception model pre trained on imagenet and use it on the kaggle cats vs dogs classification dataset getting the data first let s fetch the cats vs dogs dataset using tfds if you have your own dataset you ll probably want to use the utility keras utils image_dataset_from_directory to generate similar labeled dataset objects from a set of images on disk filed into class specific folders transfer learning is most useful when working with very small datasets to keep our dataset small we will use 40 of the original training data 25 000 images for training 10 for validation and 10 for testing reserve 10 for validation and 10 for test include labels these are the first 9 images in the training dataset as you can see they re all different sizes we can also see that label 1 is dog and label 0 is cat standardizing the data our raw images have a variety of sizes in addition each pixel consists of 3 integer values between 0 and 255 rgb level values this isn t a great fit for feeding a neural network we need to do 2 things standardize to a fixed image size we pick 150x150 normalize pixel values between 1 and 1 we ll do this using a normalization layer as part of the model itself in general it s a good practice to develop models that take raw data as input as opposed to models that take already preprocessed data the reason being that if your model expects preprocessed data any time you export your model to use it elsewhere in a web browser in a mobile app you ll need to reimplement the exact same preprocessing pipeline this gets very tricky very quickly so we should do the least possible amount of preprocessing before hitting the model here we ll do image resizing in the data pipeline because a deep neural network can only process contiguous batches of data and we ll do the input value scaling as part of the model when we create it let s resize images to 150x150 using random data augmentation when you don t have a large image dataset it s a good practice to artificially introduce sample diversity by applying random yet realistic transformations to the training images such as random horizontal flipping or small random rotations this helps expose the model to different aspects of the training data while slowing down overfitting let s batch the data and use prefetching to optimize loading speed let s visualize what the first image of the first batch looks like after various random transformations build a model now let s built a model that follows the blueprint we ve explained earlier note that we add a rescaling layer to scale input values initially in the 0 255 range to the 1 1 range we add a dropout layer before the classification layer for regularization we make sure to pass training false when calling the base model so that it runs in inference mode so that batchnorm statistics don t get updated even after we unfreeze the base model for fine tuning load weights pre trained on imagenet do not include the imagenet classifier at the top freeze the base_model create new model on top pre trained xception weights requires that input be scaled from 0 255 to a range of 1 1 the rescaling layer outputs inputs scale offset the base model contains batchnorm layers we want to keep them in inference mode when we unfreeze the base model for fine tuning so we make sure that the base_model is running in inference mode here regularize with dropout train the top layer do a round of fine tuning of the entire model finally let s unfreeze the base model and train the entire model end to end with a low learning rate importantly although the base model becomes trainable it is still running in inference mode since we passed training false when calling it when we built the model this means that the batch normalization layers inside won t update their batch statistics if they did they would wreck havoc on the representations learned by the model so far unfreeze the base_model note that it keeps running in inference mode since we passed training false when calling it this means that the batchnorm layers will not update their batch statistics this prevents the batchnorm layers from undoing all the training we ve done so far low learning rate after 10 epochs fine tuning gains us a nice improvement here let s evaluate the model on the test dataset
import numpy as np
import keras
from keras import layers
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt





layer = keras.layers.Dense(3)
layer.build((None, 4))  

print("weights:", len(layer.weights))
print("trainable_weights:", len(layer.trainable_weights))
print("non_trainable_weights:", len(layer.non_trainable_weights))



layer = keras.layers.BatchNormalization()
layer.build((None, 4))  

print("weights:", len(layer.weights))
print("trainable_weights:", len(layer.trainable_weights))
print("non_trainable_weights:", len(layer.non_trainable_weights))



layer = keras.layers.Dense(3)
layer.build((None, 4))  
layer.trainable = False  

print("weights:", len(layer.weights))
print("trainable_weights:", len(layer.trainable_weights))
print("non_trainable_weights:", len(layer.non_trainable_weights))




layer1 = keras.layers.Dense(3, activation="relu")
layer2 = keras.layers.Dense(3, activation="sigmoid")
model = keras.Sequential([keras.Input(shape=(3,)), layer1, layer2])


layer1.trainable = False


initial_layer1_weights_values = layer1.get_weights()


model.compile(optimizer="adam", loss="mse")
model.fit(np.random.random((2, 3)), np.random.random((2, 3)))


final_layer1_weights_values = layer1.get_weights()
np.testing.assert_allclose(
    initial_layer1_weights_values[0], final_layer1_weights_values[0]
)
np.testing.assert_allclose(
    initial_layer1_weights_values[1], final_layer1_weights_values[1]
)





inner_model = keras.Sequential(
    [
        keras.Input(shape=(3,)),
        keras.layers.Dense(3, activation="relu"),
        keras.layers.Dense(3, activation="relu"),
    ]
)

model = keras.Sequential(
    [
        keras.Input(shape=(3,)),
        inner_model,
        keras.layers.Dense(3, activation="sigmoid"),
    ]
)

model.trainable = False  

assert inner_model.trainable == False  
assert (
    inner_model.layers[0].trainable == False
)  









tfds.disable_progress_bar()

train_ds, validation_ds, test_ds = tfds.load(
    "cats_vs_dogs",
    
    split=["train[:40%]", "train[40%:50%]", "train[50%:60%]"],
    as_supervised=True,  
)

print(f"Number of training samples: {train_ds.cardinality()}")
print(f"Number of validation samples: {validation_ds.cardinality()}")
print(f"Number of test samples: {test_ds.cardinality()}")



plt.figure(figsize=(10, 10))
for i, (image, label) in enumerate(train_ds.take(9)):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(image)
    plt.title(int(label))
    plt.axis("off")





resize_fn = keras.layers.Resizing(150, 150)

train_ds = train_ds.map(lambda x, y: (resize_fn(x), y))
validation_ds = validation_ds.map(lambda x, y: (resize_fn(x), y))
test_ds = test_ds.map(lambda x, y: (resize_fn(x), y))



augmentation_layers = [
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),
]


def data_augmentation(x):
    for layer in augmentation_layers:
        x = layer(x)
    return x


train_ds = train_ds.map(lambda x, y: (data_augmentation(x), y))



from tensorflow import data as tf_data

batch_size = 64

train_ds = train_ds.batch(batch_size).prefetch(tf_data.AUTOTUNE).cache()
validation_ds = (
    validation_ds.batch(batch_size).prefetch(tf_data.AUTOTUNE).cache()
)
test_ds = test_ds.batch(batch_size).prefetch(tf_data.AUTOTUNE).cache()



for images, labels in train_ds.take(1):
    plt.figure(figsize=(10, 10))
    first_image = images[0]
    for i in range(9):
        ax = plt.subplot(3, 3, i + 1)
        augmented_image = data_augmentation(np.expand_dims(first_image, 0))
        plt.imshow(np.array(augmented_image[0]).astype("int32"))
        plt.title(int(labels[0]))
        plt.axis("off")



base_model = keras.applications.Xception(
    weights="imagenet",  
    input_shape=(150, 150, 3),
    include_top=False,
)  


base_model.trainable = False


inputs = keras.Input(shape=(150, 150, 3))




scale_layer = keras.layers.Rescaling(scale=1 / 127.5, offset=-1)
x = scale_layer(inputs)




x = base_model(x, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dropout(0.2)(x)  
outputs = keras.layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

model.summary(show_trainable=True)



model.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[keras.metrics.BinaryAccuracy()],
)

epochs = 2
print("Fitting the top layer of the model")
model.fit(train_ds, epochs=epochs, validation_data=validation_ds)








base_model.trainable = True
model.summary(show_trainable=True)

model.compile(
    optimizer=keras.optimizers.Adam(1e-5),  
    loss=keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[keras.metrics.BinaryAccuracy()],
)

epochs = 1
print("Fitting the end-to-end model")
model.fit(train_ds, epochs=epochs, validation_data=validation_ds)



print("Test dataset evaluation")
model.evaluate(test_ds)