title image classification via finetuning with efficientnet yixing fuhttps github comyixingfu date created 20200630 last modified 20230710 description use efficientnet with weights pretrained on imagenet for stanford dogs classification accelerator gpu introduction what is efficientnet efficientnet first introduced in tan and le 2019https arxiv orgabs1905 11946 is among the most efficient models i e requiring least flops for inference that reaches stateoftheart accuracy on both imagenet and common image classification transfer learning tasks the smallest base model is similar to mnasnethttps arxiv orgabs1807 11626 which reached nearsota with a significantly smaller model by introducing a heuristic way to scale the model efficientnet provides a family of models b0 to b7 that represents a good combination of efficiency and accuracy on a variety of scales such a scaling heuristics compoundscaling details see tan and le 2019https arxiv orgabs1905 11946 allows the efficiencyoriented base model b0 to surpass models at every scale while avoiding extensive gridsearch of hyperparameters a summary of the latest updates on the model is available at herehttps github comtensorflowtputreemastermodelsofficialefficientnet where various augmentation schemes and semisupervised learning approaches are applied to further improve the imagenet performance of the models these extensions of the model can be used by updating weights without changing model architecture b0 to b7 variants of efficientnet this section provides some details on compound scaling and can be skipped if you re only interested in using the models based on the original paperhttps arxiv orgabs1905 11946 people may have the impression that efficientnet is a continuous family of models arbitrarily choosing scaling factor in as eq 3 of the paper however choice of resolution depth and width are also restricted by many factors resolution resolutions not divisible by 8 16 etc cause zeropadding near boundaries of some layers which wastes computational resources this especially applies to smaller variants of the model hence the input resolution for b0 and b1 are chosen as 224 and 240 depth and width the building blocks of efficientnet demands channel size to be multiples of 8 resource limit memory limitation may bottleneck resolution when depth and width can still increase in such a situation increasing depth andor width but keep resolution can still improve performance as a result the depth width and resolution of each variant of the efficientnet models are handpicked and proven to produce good results though they may be significantly off from the compound scaling formula therefore the keras implementation detailed below only provide these 8 models b0 to b7 instead of allowing arbitray choice of width depth resolution parameters keras implementation of efficientnet an implementation of efficientnet b0 to b7 has been shipped with keras since v2 3 to use efficientnetb0 for classifying 1000 classes of images from imagenet run python from tensorflow keras applications import efficientnetb0 model efficientnetb0weights imagenet this model takes input images of shape 224 224 3 and the input data should be in the range 0 255 normalization is included as part of the model because training efficientnet on imagenet takes a tremendous amount of resources and several techniques that are not a part of the model architecture itself hence the keras implementation by default loads pretrained weights obtained via training with autoaugmenthttps arxiv orgabs1805 09501 for b0 to b7 base models the input shapes are different here is a list of input shape expected for each model base model resolution efficientnetb0 224 efficientnetb1 240 efficientnetb2 260 efficientnetb3 300 efficientnetb4 380 efficientnetb5 456 efficientnetb6 528 efficientnetb7 600 when the model is intended for transfer learning the keras implementation provides a option to remove the top layers model efficientnetb0includetopfalse weights imagenet this option excludes the final dense layer that turns 1280 features on the penultimate layer into prediction of the 1000 imagenet classes replacing the top layer with custom layers allows using efficientnet as a feature extractor in a transfer learning workflow another argument in the model constructor worth noticing is dropconnectrate which controls the dropout rate responsible for stochastic depthhttps arxiv orgabs1603 09382 this parameter serves as a toggle for extra regularization in finetuning but does not affect loaded weights for example when stronger regularization is desired try python model efficientnetb0weights imagenet dropconnectrate0 4 the default value is 0 2 example efficientnetb0 for stanford dogs efficientnet is capable of a wide range of image classification tasks this makes it a good model for transfer learning as an endtoend example we will show using pretrained efficientnetb0 on stanford dogshttp vision stanford eduaditya86imagenetdogsmain html dataset setup and data loading imgsize is determined by efficientnet model choice loading data here we load data from tensorflowdatasetshttps www tensorflow orgdatasets hereafter tfds stanford dogs dataset is provided in tfds as stanforddogshttps www tensorflow orgdatasetscatalogstanforddogs it features 20 580 images that belong to 120 classes of dog breeds 12 000 for training and 8 580 for testing by simply changing datasetname below you may also try this notebook for other datasets in tfds such as cifar10https www tensorflow orgdatasetscatalogcifar10 cifar100https www tensorflow orgdatasetscatalogcifar100 food101https www tensorflow orgdatasetscatalogfood101 etc when the images are much smaller than the size of efficientnet input we can simply upsample the input images it has been shown in tan and le 2019https arxiv orgabs1905 11946 that transfer learning result is better for increased resolution even if input images remain small when the dataset include images with various size we need to resize them into a shared size the stanford dogs dataset includes only images at least 200x200 pixels in size here we resize the images to the input size needed for efficientnet visualizing the data the following code shows the first 9 images with their labels data augmentation we can use the preprocessing layers apis for image augmentation this sequential model object can be used both as a part of the model we later build and as a function to preprocess data before feeding into the model using them as function makes it easy to visualize the augmented images here we plot 9 examples of augmentation result of a given figure prepare inputs once we verify the input data and augmentation are working correctly we prepare dataset for training the input data are resized to uniform imgsize the labels are put into onehot a k a categorical encoding the dataset is batched note prefetch and autotune may in some situation improve performance but depends on environment and the specific dataset used see this guidehttps www tensorflow orgguidedataperformance for more information on data pipeline performance onehot categorical encoding training a model from scratch we build an efficientnetb0 with 120 output classes that is initialized from scratch note the accuracy will increase very slowly and may overfit training the model is relatively fast this might make it sounds easy to simply train efficientnet on any dataset wanted from scratch however training efficientnet on smaller datasets especially those with lower resolution like cifar100 faces the significant challenge of overfitting hence training from scratch requires very careful choice of hyperparameters and is difficult to find suitable regularization it would also be much more demanding in resources plotting the training and validation accuracy makes it clear that validation accuracy stagnates at a low value transfer learning from pretrained weights here we initialize the model with pretrained imagenet weights and we finetune it on our own dataset freeze the pretrained weights rebuild top compile the first step to transfer learning is to freeze all layers and train only the top layers for this step a relatively large learning rate 1e2 can be used note that validation accuracy and loss will usually be better than training accuracy and loss this is because the regularization is strong which only suppresses trainingtime metrics note that the convergence may take up to 50 epochs depending on choice of learning rate if image augmentation layers were not applied the validation accuracy may only reach 60 the second step is to unfreeze a number of layers and fit the model using smaller learning rate in this example we show unfreezing all layers but depending on specific dataset it may be desireble to only unfreeze a fraction of all layers when the feature extraction with pretrained model works good enough this step would give a very limited gain on validation accuracy in our case we only see a small improvement as imagenet pretraining already exposed the model to a good amount of dogs on the other hand when we use pretrained weights on a dataset that is more different from imagenet this finetuning step can be crucial as the feature extractor also needs to be adjusted by a considerable amount such a situation can be demonstrated if choosing cifar100 dataset instead where finetuning boosts validation accuracy by about 10 to pass 80 on efficientnetb0 in such a case the convergence may take more than 50 epochs a side note on freezingunfreezing models setting trainable of a model will simultaneously set all layers belonging to the model to the same trainable attribute each layer is trainable only if both the layer itself and the model containing it are trainable hence when we need to partially freezeunfreeze a model we need to make sure the trainable attribute of the model is set to true we unfreeze the top 20 layers while leaving batchnorm layers frozen tips for fine tuning efficientnet on unfreezing layers the batchnormalization layers need to be kept frozen more detailshttps keras ioguidestransferlearning if they are also turned to trainable the first epoch after unfreezing will significantly reduce accuracy in some cases it may be beneficial to open up only a portion of layers instead of unfreezing all this will make fine tuning much faster when going to larger models like b7 each block needs to be all turned on or off this is because the architecture includes a shortcut from the first layer to the last layer for each block not respecting blocks also significantly harms the final performance some other tips for utilizing efficientnet larger variants of efficientnet do not guarantee improved performance especially for tasks with less data or fewer classes in such a case the larger variant of efficientnet chosen the harder it is to tune hyperparameters ema exponential moving average is very helpful in training efficientnet from scratch but not so much for transfer learning do not use the rmsprop setup as in the original paper for transfer learning the momentum and learning rate are too high for transfer learning it will easily corrupt the pretrained weight and blow up the loss a quick check is to see if loss as categorical cross entropy is getting significantly larger than lognumclasses after the same epoch if so the initial learning ratemomentum is too high smaller batch size benefit validation accuracy possibly due to effectively providing regularization introduction what is efficientnet efficientnet first introduced in tan and le 2019 https arxiv org abs 1905 11946 is among the most efficient models i e requiring least flops for inference that reaches state of the art accuracy on both imagenet and common image classification transfer learning tasks the smallest base model is similar to mnasnet https arxiv org abs 1807 11626 which reached near sota with a significantly smaller model by introducing a heuristic way to scale the model efficientnet provides a family of models b0 to b7 that represents a good combination of efficiency and accuracy on a variety of scales such a scaling heuristics compound scaling details see tan and le 2019 https arxiv org abs 1905 11946 allows the efficiency oriented base model b0 to surpass models at every scale while avoiding extensive grid search of hyperparameters a summary of the latest updates on the model is available at here https github com tensorflow tpu tree master models official efficientnet where various augmentation schemes and semi supervised learning approaches are applied to further improve the imagenet performance of the models these extensions of the model can be used by updating weights without changing model architecture b0 to b7 variants of efficientnet this section provides some details on compound scaling and can be skipped if you re only interested in using the models based on the original paper https arxiv org abs 1905 11946 people may have the impression that efficientnet is a continuous family of models arbitrarily choosing scaling factor in as eq 3 of the paper however choice of resolution depth and width are also restricted by many factors resolution resolutions not divisible by 8 16 etc cause zero padding near boundaries of some layers which wastes computational resources this especially applies to smaller variants of the model hence the input resolution for b0 and b1 are chosen as 224 and 240 depth and width the building blocks of efficientnet demands channel size to be multiples of 8 resource limit memory limitation may bottleneck resolution when depth and width can still increase in such a situation increasing depth and or width but keep resolution can still improve performance as a result the depth width and resolution of each variant of the efficientnet models are hand picked and proven to produce good results though they may be significantly off from the compound scaling formula therefore the keras implementation detailed below only provide these 8 models b0 to b7 instead of allowing arbitray choice of width depth resolution parameters keras implementation of efficientnet an implementation of efficientnet b0 to b7 has been shipped with keras since v2 3 to use efficientnetb0 for classifying 1000 classes of images from imagenet run python from tensorflow keras applications import efficientnetb0 model efficientnetb0 weights imagenet this model takes input images of shape 224 224 3 and the input data should be in the range 0 255 normalization is included as part of the model because training efficientnet on imagenet takes a tremendous amount of resources and several techniques that are not a part of the model architecture itself hence the keras implementation by default loads pre trained weights obtained via training with autoaugment https arxiv org abs 1805 09501 for b0 to b7 base models the input shapes are different here is a list of input shape expected for each model base model resolution efficientnetb0 224 efficientnetb1 240 efficientnetb2 260 efficientnetb3 300 efficientnetb4 380 efficientnetb5 456 efficientnetb6 528 efficientnetb7 600 when the model is intended for transfer learning the keras implementation provides a option to remove the top layers model efficientnetb0 include_top false weights imagenet this option excludes the final dense layer that turns 1280 features on the penultimate layer into prediction of the 1000 imagenet classes replacing the top layer with custom layers allows using efficientnet as a feature extractor in a transfer learning workflow another argument in the model constructor worth noticing is drop_connect_rate which controls the dropout rate responsible for stochastic depth https arxiv org abs 1603 09382 this parameter serves as a toggle for extra regularization in finetuning but does not affect loaded weights for example when stronger regularization is desired try python model efficientnetb0 weights imagenet drop_connect_rate 0 4 the default value is 0 2 example efficientnetb0 for stanford dogs efficientnet is capable of a wide range of image classification tasks this makes it a good model for transfer learning as an end to end example we will show using pre trained efficientnetb0 on stanford dogs http vision stanford edu aditya86 imagenetdogs main html dataset setup and data loading for tf data img_size is determined by efficientnet model choice loading data here we load data from tensorflow_datasets https www tensorflow org datasets hereafter tfds stanford dogs dataset is provided in tfds as stanford_dogs https www tensorflow org datasets catalog stanford_dogs it features 20 580 images that belong to 120 classes of dog breeds 12 000 for training and 8 580 for testing by simply changing dataset_name below you may also try this notebook for other datasets in tfds such as cifar10 https www tensorflow org datasets catalog cifar10 cifar100 https www tensorflow org datasets catalog cifar100 food101 https www tensorflow org datasets catalog food101 etc when the images are much smaller than the size of efficientnet input we can simply upsample the input images it has been shown in tan and le 2019 https arxiv org abs 1905 11946 that transfer learning result is better for increased resolution even if input images remain small when the dataset include images with various size we need to resize them into a shared size the stanford dogs dataset includes only images at least 200x200 pixels in size here we resize the images to the input size needed for efficientnet visualizing the data the following code shows the first 9 images with their labels data augmentation we can use the preprocessing layers apis for image augmentation this sequential model object can be used both as a part of the model we later build and as a function to preprocess data before feeding into the model using them as function makes it easy to visualize the augmented images here we plot 9 examples of augmentation result of a given figure prepare inputs once we verify the input data and augmentation are working correctly we prepare dataset for training the input data are resized to uniform img_size the labels are put into one hot a k a categorical encoding the dataset is batched note prefetch and autotune may in some situation improve performance but depends on environment and the specific dataset used see this guide https www tensorflow org guide data_performance for more information on data pipeline performance one hot categorical encoding training a model from scratch we build an efficientnetb0 with 120 output classes that is initialized from scratch note the accuracy will increase very slowly and may overfit param type slider min 10 max 100 training the model is relatively fast this might make it sounds easy to simply train efficientnet on any dataset wanted from scratch however training efficientnet on smaller datasets especially those with lower resolution like cifar 100 faces the significant challenge of overfitting hence training from scratch requires very careful choice of hyperparameters and is difficult to find suitable regularization it would also be much more demanding in resources plotting the training and validation accuracy makes it clear that validation accuracy stagnates at a low value transfer learning from pre trained weights here we initialize the model with pre trained imagenet weights and we fine tune it on our own dataset freeze the pretrained weights rebuild top compile the first step to transfer learning is to freeze all layers and train only the top layers for this step a relatively large learning rate 1e 2 can be used note that validation accuracy and loss will usually be better than training accuracy and loss this is because the regularization is strong which only suppresses training time metrics note that the convergence may take up to 50 epochs depending on choice of learning rate if image augmentation layers were not applied the validation accuracy may only reach 60 param type slider min 8 max 80 the second step is to unfreeze a number of layers and fit the model using smaller learning rate in this example we show unfreezing all layers but depending on specific dataset it may be desireble to only unfreeze a fraction of all layers when the feature extraction with pretrained model works good enough this step would give a very limited gain on validation accuracy in our case we only see a small improvement as imagenet pretraining already exposed the model to a good amount of dogs on the other hand when we use pretrained weights on a dataset that is more different from imagenet this fine tuning step can be crucial as the feature extractor also needs to be adjusted by a considerable amount such a situation can be demonstrated if choosing cifar 100 dataset instead where fine tuning boosts validation accuracy by about 10 to pass 80 on efficientnetb0 in such a case the convergence may take more than 50 epochs a side note on freezing unfreezing models setting trainable of a model will simultaneously set all layers belonging to the model to the same trainable attribute each layer is trainable only if both the layer itself and the model containing it are trainable hence when we need to partially freeze unfreeze a model we need to make sure the trainable attribute of the model is set to true we unfreeze the top 20 layers while leaving batchnorm layers frozen param type slider min 8 max 50 tips for fine tuning efficientnet on unfreezing layers the batchnormalization layers need to be kept frozen more details https keras io guides transfer_learning if they are also turned to trainable the first epoch after unfreezing will significantly reduce accuracy in some cases it may be beneficial to open up only a portion of layers instead of unfreezing all this will make fine tuning much faster when going to larger models like b7 each block needs to be all turned on or off this is because the architecture includes a shortcut from the first layer to the last layer for each block not respecting blocks also significantly harms the final performance some other tips for utilizing efficientnet larger variants of efficientnet do not guarantee improved performance especially for tasks with less data or fewer classes in such a case the larger variant of efficientnet chosen the harder it is to tune hyperparameters ema exponential moving average is very helpful in training efficientnet from scratch but not so much for transfer learning do not use the rmsprop setup as in the original paper for transfer learning the momentum and learning rate are too high for transfer learning it will easily corrupt the pretrained weight and blow up the loss a quick check is to see if loss as categorical cross entropy is getting significantly larger than log num_classes after the same epoch if so the initial learning rate momentum is too high smaller batch size benefit validation accuracy possibly due to effectively providing regularization
import numpy as np
import tensorflow_datasets as tfds
import tensorflow as tf  
import matplotlib.pyplot as plt
import keras
from keras import layers
from keras.applications import EfficientNetB0


IMG_SIZE = 224
BATCH_SIZE = 64




dataset_name = "stanford_dogs"
(ds_train, ds_test), ds_info = tfds.load(
    dataset_name, split=["train", "test"], with_info=True, as_supervised=True
)
NUM_CLASSES = ds_info.features["label"].num_classes




size = (IMG_SIZE, IMG_SIZE)
ds_train = ds_train.map(lambda image, label: (tf.image.resize(image, size), label))
ds_test = ds_test.map(lambda image, label: (tf.image.resize(image, size), label))




def format_label(label):
    string_label = label_info.int2str(label)
    return string_label.split("-")[1]


label_info = ds_info.features["label"]
for i, (image, label) in enumerate(ds_train.take(9)):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(image.numpy().astype("uint8"))
    plt.title("{}".format(format_label(label)))
    plt.axis("off")




img_augmentation_layers = [
    layers.RandomRotation(factor=0.15),
    layers.RandomTranslation(height_factor=0.1, width_factor=0.1),
    layers.RandomFlip(),
    layers.RandomContrast(factor=0.1),
]

def img_augmentation(images):
    for layer in img_augmentation_layers:
        images = layer(images)
    return images




for image, label in ds_train.take(1):
    for i in range(9):
        ax = plt.subplot(3, 3, i + 1)
        aug_img = img_augmentation(np.expand_dims(image.numpy(), axis=0))
        aug_img = np.array(aug_img)
        plt.imshow(aug_img[0].astype("uint8"))
        plt.title("{}".format(format_label(label)))
        plt.axis("off")






def input_preprocess_train(image, label):
    image = img_augmentation(image)
    label = tf.one_hot(label, NUM_CLASSES)
    return image, label


def input_preprocess_test(image, label):
    label = tf.one_hot(label, NUM_CLASSES)
    return image, label


ds_train = ds_train.map(input_preprocess_train, num_parallel_calls=tf.data.AUTOTUNE)
ds_train = ds_train.batch(batch_size=BATCH_SIZE, drop_remainder=True)
ds_train = ds_train.prefetch(tf.data.AUTOTUNE)

ds_test = ds_test.map(input_preprocess_test, num_parallel_calls=tf.data.AUTOTUNE)
ds_test = ds_test.batch(batch_size=BATCH_SIZE, drop_remainder=True)




model = EfficientNetB0(include_top=True, weights=None, classes=NUM_CLASSES, input_shape=(IMG_SIZE, IMG_SIZE, 3))
model.compile(
    optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"]
)

model.summary()

epochs = 40  
hist = model.fit(ds_train, epochs=epochs, validation_data=ds_test)




import matplotlib.pyplot as plt


def plot_hist(hist):
    plt.plot(hist.history["accuracy"])
    plt.plot(hist.history["val_accuracy"])
    plt.title("model accuracy")
    plt.ylabel("accuracy")
    plt.xlabel("epoch")
    plt.legend(["train", "validation"], loc="upper left")
    plt.show()


plot_hist(hist)




def build_model(num_classes):
    inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
    model = EfficientNetB0(include_top=False, input_tensor=inputs, weights="imagenet")

    
    model.trainable = False

    
    x = layers.GlobalAveragePooling2D(name="avg_pool")(model.output)
    x = layers.BatchNormalization()(x)

    top_dropout_rate = 0.2
    x = layers.Dropout(top_dropout_rate, name="top_dropout")(x)
    outputs = layers.Dense(num_classes, activation="softmax", name="pred")(x)

    
    model = keras.Model(inputs, outputs, name="EfficientNet")
    optimizer = keras.optimizers.Adam(learning_rate=1e-2)
    model.compile(
        optimizer=optimizer, loss="categorical_crossentropy", metrics=["accuracy"]
    )
    return model




model = build_model(num_classes=NUM_CLASSES)

epochs = 25  
hist = model.fit(ds_train, epochs=epochs, validation_data=ds_test)
plot_hist(hist)




def unfreeze_model(model):
    
    for layer in model.layers[-20:]:
        if not isinstance(layer, layers.BatchNormalization):
            layer.trainable = True

    optimizer = keras.optimizers.Adam(learning_rate=1e-4)
    model.compile(
        optimizer=optimizer, loss="categorical_crossentropy", metrics=["accuracy"]
    )


unfreeze_model(model)

epochs = 10  
hist = model.fit(ds_train, epochs=epochs, validation_data=ds_test)
plot_hist(hist)