<jupyter_start><jupyter_text>CutMix, MixUp, and RandAugment image augmentation with KerasCV**Author:** [lukewood](https://lukewood.xyz)**Date created:** 2022/04/08**Last modified:** 2022/04/08**Description:** Use KerasCV to augment images with CutMix, MixUp, RandAugment, and more.  OverviewKerasCV makes it easy to assemble state-of-the-art, industry-grade data augmentationpipelines for image classification and object detection tasks. KerasCV offers a widesuite of preprocessing layers implementing common data augmentation techniques.Perhaps three of the most useful layers are `CutMix`, `MixUp`, and `RandAugment`. Theselayers are used in nearly all state-of-the-art image classification pipelines.This guide will show you how to compose these layers into your own dataaugmentation pipeline for image classification tasks. This guide will also walk youthrough the process of customizing a KerasCV data augmentation pipeline.  Imports & setupThis tutorial requires you to have KerasCV installed:```shellpip install keras-cv```We begin by importing all required packages:<jupyter_code>import keras_cv
import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow import keras
from tensorflow.keras import applications
from tensorflow.keras import losses
from tensorflow.keras import optimizers<jupyter_output><empty_output><jupyter_text>Data loadingThis guide uses the[102 Category Flower Dataset](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/)for demonstration purposes.To get started, we first load the dataset:<jupyter_code>BATCH_SIZE = 32
AUTOTUNE = tf.data.AUTOTUNE
tfds.disable_progress_bar()
data, dataset_info = tfds.load("oxford_flowers102", with_info=True, as_supervised=True)
train_steps_per_epoch = dataset_info.splits["train"].num_examples // BATCH_SIZE
val_steps_per_epoch = dataset_info.splits["test"].num_examples // BATCH_SIZE<jupyter_output><empty_output><jupyter_text>Next, we resize the images to a constant size, `(224, 224)`, and one-hot encode thelabels. Please note that `keras_cv.layers.CutMix` and `keras_cv.layers.MixUp` expecttargets to be one-hot encoded. This is because they modify the values of the targetsin a way that is not possible with a sparse label representation.<jupyter_code>IMAGE_SIZE = (224, 224)
num_classes = dataset_info.features["label"].num_classes


def prepare(image, label):
    image = tf.image.resize(image, IMAGE_SIZE)
    image = tf.cast(image, tf.float32)
    label = tf.one_hot(label, num_classes)
    return {"images": image, "labels": label}


def prepare_dataset(dataset, split):
    if split == "train":
        return (
            dataset.shuffle(10 * BATCH_SIZE)
            .map(prepare, num_parallel_calls=AUTOTUNE)
            .batch(BATCH_SIZE)
        )
    if split == "test":
        return dataset.map(prepare, num_parallel_calls=AUTOTUNE).batch(BATCH_SIZE)


def load_dataset(split="train"):
    dataset = data[split]
    return prepare_dataset(dataset, split)


train_dataset = load_dataset()<jupyter_output><empty_output><jupyter_text>Let's inspect some samples from our dataset:<jupyter_code>def visualize_dataset(dataset, title):
    plt.figure(figsize=(6, 6)).suptitle(title, fontsize=18)
    for i, samples in enumerate(iter(dataset.take(9))):
        images = samples["images"]
        plt.subplot(3, 3, i + 1)
        plt.imshow(images[0].numpy().astype("uint8"))
        plt.axis("off")
    plt.show()


visualize_dataset(train_dataset, title="Before Augmentation")<jupyter_output><empty_output><jupyter_text>Great! Now we can move onto the augmentation step.  RandAugment [RandAugment](https://arxiv.org/abs/1909.13719)has been shown to provide improved imageclassification results across numerous datasets.It performs a standard set of augmentations on an image.To use RandAugment in KerasCV, you need to provide a few values:- `value_range` describes the range of values covered in your images- `magnitude` is a value between 0 and 1, describing the strength of the perturbationsapplied- `augmentations_per_image` is an integer telling the layer how many augmentations to apply to eachindividual image- (Optional) `magnitude_stddev` allows `magnitude` to be randomly sampledfrom a distribution with a standard deviation of `magnitude_stddev`- (Optional) `rate` indicates the probability to apply the augmentationapplied at each layer.You can read more about theseparameters in the[`RandAugment` API documentation](/api/keras_cv/layers/rand_augment).Let's use KerasCV's RandAugment implementation.<jupyter_code>rand_augment = keras_cv.layers.RandAugment(
    value_range=(0, 255),
    augmentations_per_image=3,
    magnitude=0.3,
    magnitude_stddev=0.2,
    rate=0.5,
)


def apply_rand_augment(inputs):
    inputs["images"] = rand_augment(inputs["images"])
    return inputs


train_dataset = load_dataset().map(apply_rand_augment, num_parallel_calls=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>Finally, let's inspect some of the results:<jupyter_code>visualize_dataset(train_dataset, title="After RandAugment")<jupyter_output><empty_output><jupyter_text>Try tweaking the magnitude settings to see a wider variety of results.  CutMix and MixUp: generate high-quality inter-class examples`CutMix` and `MixUp` allow us to produce inter-class examples. `CutMix` randomly cuts outportions of one image and places them over another, and `MixUp` interpolates the pixelvalues between two images. Both of these prevent the model from overfitting thetraining distribution and improve the likelihood that the model can generalize to out ofdistribution examples. Additionally, `CutMix` prevents your model from over-relying onany particular feature to perform its classifications. You can read more about thesetechniques in their respective papers:- [CutMix: Train Strong Classifiers](https://arxiv.org/abs/1905.04899)- [MixUp: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412)In this example, we will use `CutMix` and `MixUp` independently in a manually createdpreprocessing pipeline. In most state of the art pipelines images are randomlyaugmented by either `CutMix`, `MixUp`, or neither. The function below implements thisin an equal 1/3 split.Note that our `cut_mix_and_mix_up` function is annotated with a `tf.function` to ensureoptimal performance.<jupyter_code>cut_mix = keras_cv.layers.CutMix()
mix_up = keras_cv.layers.MixUp()


def cut_mix_and_mix_up(samples):
    samples = cut_mix(samples, training=True)
    samples = mix_up(samples, training=True)
    return samples


train_dataset = load_dataset().map(cut_mix_and_mix_up, num_parallel_calls=AUTOTUNE)

visualize_dataset(train_dataset, title="After CutMix and MixUp")<jupyter_output><empty_output><jupyter_text>Great! Looks like we have successfully added `CutMix` and `MixUp` to our preprocessingpipeline.  Customizing your augmentation pipelinePerhaps you want to exclude an augmentation from `RandAugment`, or perhaps you want toinclude the `GridMask()` as an option alongside the default `RandAugment` augmentations.KerasCV allows you to construct production grade custom data augmentation pipelines usingthe `keras_cv.layers.RandomAugmentationPipeline` layer. This class operates similarly to`RandAugment`; selecting a random layer to apply to each image `augmentations_per_image`times. `RandAugment` can be thought of as a specific case of`RandomAugmentationPipeline`. In fact, our `RandAugment` implementation inherits from`RandomAugmentationPipeline` internally.In this example, we will create a custom `RandomAugmentationPipeline` by removing`RandomRotation` layers from the standard `RandAugment` policy, and substitutex a`GridMask` layer in its place. As a first step, let's use the helper method `RandAugment.get_standard_policy()` tocreate a base pipeline.<jupyter_code>layers = keras_cv.layers.RandAugment.get_standard_policy(
    value_range=(0, 255), magnitude=0.75, magnitude_stddev=0.3
)<jupyter_output><empty_output><jupyter_text>First, let's filter out `RandomRotation` layers<jupyter_code>layers = [
    layer for layer in layers if not isinstance(layer, keras_cv.layers.RandomRotation)
]<jupyter_output><empty_output><jupyter_text>Next, let's add `GridMask` to our layers:<jupyter_code>layers = layers + [keras_cv.layers.GridMask()]<jupyter_output><empty_output><jupyter_text>Finally, we can put together our pipeline<jupyter_code>pipeline = keras_cv.layers.RandomAugmentationPipeline(
    layers=layers, augmentations_per_image=3
)<jupyter_output><empty_output><jupyter_text>Let's check out the results!<jupyter_code>def apply_pipeline(inputs):
    inputs["images"] = pipeline(inputs["images"])
    return inputs


train_dataset = load_dataset().map(apply_pipeline, num_parallel_calls=AUTOTUNE)
visualize_dataset(train_dataset, title="After custom pipeline")<jupyter_output><empty_output><jupyter_text>Awesome! As you can see, no images were randomly rotated. You can customize thepipeline however you like:<jupyter_code>pipeline = keras_cv.layers.RandomAugmentationPipeline(
    layers=[keras_cv.layers.GridMask(), keras_cv.layers.Grayscale(output_channels=3)],
    augmentations_per_image=1,
)<jupyter_output><empty_output><jupyter_text>This pipeline will either apply `GrayScale` or GridMask:<jupyter_code>def apply_pipeline(inputs):
    inputs["images"] = pipeline(inputs["images"])
    return inputs


train_dataset = load_dataset().map(apply_pipeline, num_parallel_calls=AUTOTUNE)
visualize_dataset(train_dataset, title="After custom pipeline")<jupyter_output><empty_output><jupyter_text>Looks great! You can use `RandomAugmentationPipeline` however you want.  Training a CNNAs a final exercise, let's take some of these layers for a spin. In this section, wewill use `CutMix`, `MixUp`, and `RandAugment` to train a state of the art `ResNet50`image classifier on the Oxford flowers dataset.<jupyter_code>def preprocess_for_model(inputs):
    images, labels = inputs["images"], inputs["labels"]
    images = tf.cast(images, tf.float32)
    return images, labels


train_dataset = (
    load_dataset()
    .map(apply_rand_augment, num_parallel_calls=AUTOTUNE)
    .map(cut_mix_and_mix_up, num_parallel_calls=AUTOTUNE)
)

visualize_dataset(train_dataset, "CutMix, MixUp and RandAugment")

train_dataset = train_dataset.map(preprocess_for_model, num_parallel_calls=AUTOTUNE)

test_dataset = load_dataset(split="test")
test_dataset = test_dataset.map(preprocess_for_model, num_parallel_calls=AUTOTUNE)

train_dataset = train_dataset.prefetch(AUTOTUNE)
test_dataset = test_dataset.prefetch(AUTOTUNE)

train_dataset = train_dataset
test_dataset = test_dataset<jupyter_output><empty_output><jupyter_text>Next we should create a the model itself. Notice that we use `label_smoothing=0.1` inthe loss function. When using `MixUp`, label smoothing is _highly_ recommended.<jupyter_code>input_shape = IMAGE_SIZE + (3,)


def get_model():
    inputs = keras.layers.Input(input_shape)
    x = applications.ResNet50V2(
        input_shape=input_shape, classes=num_classes, weights=None
    )(inputs)
    model = keras.Model(inputs, x)
    model.compile(
        loss=losses.CategoricalCrossentropy(label_smoothing=0.1),
        optimizer=optimizers.SGD(momentum=0.9),
        metrics=["accuracy"],
    )
    return model<jupyter_output><empty_output><jupyter_text>Finally we train the model:<jupyter_code>strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = get_model()
    model.fit(
        train_dataset,
        epochs=1,
        validation_data=test_dataset,
    )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08873129810602785, 'fraction_numerical': 0.010550895096428262, 'mean_word_length': 6.160371517027864, 'pattern_counts': {'":': 4, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '195363', 'license': 'Apache-2.0', 'n_tokens_mistral': 3601, 'n_tokens_neox': 3426, 'n_words': 983, 'path': 'guides/ipynb/keras_cv/cut_mix_mix_up_and_rand_augment.ipynb', 'repo_name': 'wariua/keras-io-ko', 'size': '11563'}