title learning to resize in computer vision sayak paulhttps twitter comrisingsayak date created 20210430 last modified 20230726 description how to optimally learn representations of images for a given resolution accelerator gpu it is a common belief that if we constrain vision models to perceive things as humans do their performance can be improved for example in this workhttps arxiv orgabs1811 12231 geirhos et al showed that the vision models pretrained on the imagenet1k dataset are biased towards texture whereas human beings mostly use the shape descriptor to develop a common perception but does this belief always apply especially when it comes to improving the performance of vision models it turns out it may not always be the case when training vision models it is common to resize images to a lower dimension 224 x 224 299 x 299 etc to allow minibatch learning and also to keep up the compute limitations we generally make use of image resizing methods like bilinear interpolation for this step and the resized images do not lose much of their perceptual character to the human eyes in learning to resize images for computer vision taskshttps arxiv orgabs2103 09950v1 talebi et al show that if we try to optimize the perceptual quality of the images for the vision models rather than the human eyes their performance can further be improved they investigate the following question for a given image resolution and a model how to best resize the given images as shown in the paper this idea helps to consistently improve the performance of the common vision models pretrained on imagenet1k like densenet121 resnet50 mobilenetv2 and efficientnets in this example we will implement the learnable image resizing module as proposed in the paper and demonstrate that on the cats and dogs datasethttps www microsoft comenusdownloaddetails aspx id54765 using the densenet121https arxiv orgabs1608 06993 architecture setup define hyperparameters in order to facilitate minibatch learning we need to have a fixed shape for the images inside a given batch this is why an initial resizing is required we first resize all the images to 300 x 300 shape and then learn their optimal representation for the 150 x 150 resolution in this example we will use the bilinear interpolation but the learnable image resizer module is not dependent on any specific interpolation method we can also use others such as bicubic load and prepare the dataset for this example we will only use 40 of the total training dataset reserve 10 for validation define the learnable resizer utilities the figure below courtesy learning to resize images for computer vision taskshttps arxiv orgabs2103 09950v1 presents the structure of the learnable resizing module https i ibb cogjytss0image png first perform naive resizing first convolution block without batch normalization second convolution block with batch normalization intermediate resizing as a bottleneck residual passes projection skip connection final resized image visualize the outputs of the learnable resizing module here we visualize how the resized images would look like after being passed through the random weights of the resizer model building utility the structure of the learnable image resizer module allows for flexible integrations with different vision models compile and train our model with learnable resizer visualize the outputs of the trained visualizer the plot shows that the visuals of the images have improved with training the following table shows the benefits of using the resizing module in comparison to using the bilinear interpolation model number of parameters million top1 accuracy with the learnable resizer 7 051717 67 67 without the learnable resizer 7 039554 60 19 for more details you can check out this repositoryhttps github comsayakpaullearnableimageresizing note the abovereported models were trained for 10 epochs on 90 of the training set of cats and dogs unlike this example also note that the increase in the number of parameters due to the resizing module is very negligible to ensure that the improvement in the performance is not due to stochasticity the models were trained using the same initial random weights now a question worth asking here is isn t the improved accuracy simply a consequence of adding more layers the resizer is a mini network after all to the model compared to the baseline to show that it is not the case the s conduct the following experiment take a pretrained model trained some size say 224 x 224 now first use it to infer predictions on images resized to a lower resolution record the performance for the second experiment plug in the resizer module at the top of the pretrained model and warmstart the training record the performance now the s argue that using the second option is better because it helps the model learn how to adjust the representations better with respect to the given resolution since the results purely are empirical a few more experiments such as analyzing the crosschannel interaction would have been even better it is worth noting that elements like squeeze and excitation se blockshttps arxiv orgabs1709 01507 global context gc blockshttps arxiv orgpdf1904 11492 also add a few parameters to an existing network but they are known to help a network process information in systematic ways to improve the overall performance notes to impose shape bias inside the vision models geirhos et al trained them with a combination of natural and stylized images it might be interesting to investigate if this learnable resizing module could achieve something similar as the outputs seem to discard the texture information the resizer module can handle arbitrary resolutions and aspect ratios which is very important for tasks like object detection and segmentation there is another closely related topic on adaptive image resizing that attempts to resize imagesfeature maps adaptively during training efficientv2https arxiv orgpdf2104 00298 uses this idea it is a common belief that if we constrain vision models to perceive things as humans do their performance can be improved for example in this work https arxiv org abs 1811 12231 geirhos et al showed that the vision models pre trained on the imagenet 1k dataset are biased towards texture whereas human beings mostly use the shape descriptor to develop a common perception but does this belief always apply especially when it comes to improving the performance of vision models it turns out it may not always be the case when training vision models it is common to resize images to a lower dimension 224 x 224 299 x 299 etc to allow mini batch learning and also to keep up the compute limitations we generally make use of image resizing methods like bilinear interpolation for this step and the resized images do not lose much of their perceptual character to the human eyes in learning to resize images for computer vision tasks https arxiv org abs 2103 09950v1 talebi et al show that if we try to optimize the perceptual quality of the images for the vision models rather than the human eyes their performance can further be improved they investigate the following question for a given image resolution and a model how to best resize the given images as shown in the paper this idea helps to consistently improve the performance of the common vision models pre trained on imagenet 1k like densenet 121 resnet 50 mobilenetv2 and efficientnets in this example we will implement the learnable image resizing module as proposed in the paper and demonstrate that on the cats and dogs dataset https www microsoft com en us download details aspx id 54765 using the densenet 121 https arxiv org abs 1608 06993 architecture setup define hyperparameters in order to facilitate mini batch learning we need to have a fixed shape for the images inside a given batch this is why an initial resizing is required we first resize all the images to 300 x 300 shape and then learn their optimal representation for the 150 x 150 resolution in this example we will use the bilinear interpolation but the learnable image resizer module is not dependent on any specific interpolation method we can also use others such as bicubic load and prepare the dataset for this example we will only use 40 of the total training dataset reserve 10 for validation define the learnable resizer utilities the figure below courtesy learning to resize images for computer vision tasks https arxiv org abs 2103 09950v1 presents the structure of the learnable resizing module https i ibb co gjytss0 image png first perform naive resizing first convolution block without batch normalization second convolution block with batch normalization intermediate resizing as a bottleneck residual passes projection skip connection final resized image visualize the outputs of the learnable resizing module here we visualize how the resized images would look like after being passed through the random weights of the resizer helper to convert image from any backend to numpy model building utility the structure of the learnable image resizer module allows for flexible integrations with different vision models compile and train our model with learnable resizer visualize the outputs of the trained visualizer the plot shows that the visuals of the images have improved with training the following table shows the benefits of using the resizing module in comparison to using the bilinear interpolation model number of parameters million top 1 accuracy with the learnable resizer 7 051717 67 67 without the learnable resizer 7 039554 60 19 for more details you can check out this repository https github com sayakpaul learnable image resizing note the above reported models were trained for 10 epochs on 90 of the training set of cats and dogs unlike this example also note that the increase in the number of parameters due to the resizing module is very negligible to ensure that the improvement in the performance is not due to stochasticity the models were trained using the same initial random weights now a question worth asking here is _isn t the improved accuracy simply a consequence of adding more layers the resizer is a mini network after all to the model compared to the baseline _ to show that it is not the case the s conduct the following experiment take a pre trained model trained some size say 224 x 224 now first use it to infer predictions on images resized to a lower resolution record the performance for the second experiment plug in the resizer module at the top of the pre trained model and warm start the training record the performance now the s argue that using the second option is better because it helps the model learn how to adjust the representations better with respect to the given resolution since the results purely are empirical a few more experiments such as analyzing the cross channel interaction would have been even better it is worth noting that elements like squeeze and excitation se blocks https arxiv org abs 1709 01507 global context gc blocks https arxiv org pdf 1904 11492 also add a few parameters to an existing network but they are known to help a network process information in systematic ways to improve the overall performance notes to impose shape bias inside the vision models geirhos et al trained them with a combination of natural and stylized images it might be interesting to investigate if this learnable resizing module could achieve something similar as the outputs seem to discard the texture information the resizer module can handle arbitrary resolutions and aspect ratios which is very important for tasks like object detection and segmentation there is another closely related topic on adaptive image resizing that attempts to resize images feature maps adaptively during training efficientv2 https arxiv org pdf 2104 00298 uses this idea
from keras import layers
import keras
from keras import ops

from tensorflow import data as tf_data
from tensorflow import image as tf_image
from tensorflow import one_hot as tf_one_hot

import tensorflow_datasets as tfds

tfds.disable_progress_bar()

import matplotlib.pyplot as plt
import numpy as np





INP_SIZE = (300, 300)
TARGET_SIZE = (150, 150)
INTERPOLATION = "bilinear"

AUTO = tf_data.AUTOTUNE
BATCH_SIZE = 50
EPOCHS = 5





train_ds, validation_ds = tfds.load(
    "cats_vs_dogs",
    
    split=["train[:40%]", "train[40%:50%]"],
    as_supervised=True,
)


def preprocess_dataset(image, label):
    image = tf_image.resize(image, (INP_SIZE[0], INP_SIZE[1]))
    label = tf_one_hot(label, depth=2)
    return (image, label)


train_ds = (
    train_ds.shuffle(BATCH_SIZE * 100)
    .map(preprocess_dataset, num_parallel_calls=AUTO)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
)
validation_ds = (
    validation_ds.map(preprocess_dataset, num_parallel_calls=AUTO)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
)




def conv_block(
    x, filters, kernel_size, strides, activation=layers.LeakyReLU(0.2)
):
    x = layers.Conv2D(
        filters, kernel_size, strides, padding="same", use_bias=False
    )(x)
    x = layers.BatchNormalization()(x)
    if activation:
        x = activation(x)
    return x


def res_block(x):
    inputs = x
    x = conv_block(x, 16, 3, 1)
    x = conv_block(x, 16, 3, 1, activation=None)
    return layers.Add()([inputs, x])


def get_learnable_resizer(
    filters=16, num_res_blocks=1, interpolation=INTERPOLATION
):
    inputs = layers.Input(shape=[None, None, 3])

    
    naive_resize = layers.Resizing(*TARGET_SIZE, interpolation=interpolation)(
        inputs
    )

    
    x = layers.Conv2D(
        filters=filters, kernel_size=7, strides=1, padding="same"
    )(inputs)
    x = layers.LeakyReLU(0.2)(x)

    
    x = layers.Conv2D(
        filters=filters, kernel_size=1, strides=1, padding="same"
    )(x)
    x = layers.LeakyReLU(0.2)(x)
    x = layers.BatchNormalization()(x)

    
    bottleneck = layers.Resizing(*TARGET_SIZE, interpolation=interpolation)(x)

    
    for _ in range(num_res_blocks):
        x = res_block(bottleneck)

    
    x = layers.Conv2D(
        filters=filters,
        kernel_size=3,
        strides=1,
        padding="same",
        use_bias=False,
    )(x)
    x = layers.BatchNormalization()(x)

    
    x = layers.Add()([bottleneck, x])

    
    x = layers.Conv2D(filters=3, kernel_size=7, strides=1, padding="same")(x)
    final_resize = layers.Add()([naive_resize, x])

    return keras.Model(inputs, final_resize, name="learnable_resizer")


learnable_resizer = get_learnable_resizer()



sample_images, _ = next(iter(train_ds))

get_np = lambda image: ops.convert_to_numpy(
    ops.squeeze(image)
)  

plt.figure(figsize=(16, 10))
for i, image in enumerate(sample_images[:6]):
    image = image / 255

    ax = plt.subplot(3, 4, 2 * i + 1)
    plt.title("Input Image")
    plt.imshow(image.numpy().squeeze())
    plt.axis("off")

    ax = plt.subplot(3, 4, 2 * i + 2)
    resized_image = learnable_resizer(image[None, ...])
    plt.title("Resized Image")
    plt.imshow(get_np(resized_image))
    plt.axis("off")




def get_model():
    backbone = keras.applications.DenseNet121(
        weights=None,
        include_top=True,
        classes=2,
        input_shape=((TARGET_SIZE[0], TARGET_SIZE[1], 3)),
    )
    backbone.trainable = True

    inputs = layers.Input((INP_SIZE[0], INP_SIZE[1], 3))
    x = layers.Rescaling(scale=1.0 / 255)(inputs)
    x = learnable_resizer(x)
    outputs = backbone(x)

    return keras.Model(inputs, outputs)






model = get_model()
model.compile(
    loss=keras.losses.CategoricalCrossentropy(label_smoothing=0.1),
    optimizer="sgd",
    metrics=["accuracy"],
)
model.fit(train_ds, validation_data=validation_ds, epochs=EPOCHS)



plt.figure(figsize=(16, 10))
for i, image in enumerate(sample_images[:6]):
    image = image / 255

    ax = plt.subplot(3, 4, 2 * i + 1)
    plt.title("Input Image")
    plt.imshow(image.numpy().squeeze())
    plt.axis("off")

    ax = plt.subplot(3, 4, 2 * i + 2)
    resized_image = learnable_resizer(image[None, ...])
    plt.title("Resized Image")
    plt.imshow(get_np(resized_image) / 10)
    plt.axis("off")