title neural style transfer fchollethttps twitter comfchollet date created 20160111 last modified 20200502 description transferring the style of a reference image to target image using gradient descent accelerator gpu introduction style transfer consists in generating an image with the same content as a base image but with the style of a different picture typically artistic this is achieved through the optimization of a loss function that has 3 components style loss content loss and total variation loss the total variation loss imposes local spatial continuity between the pixels of the combination image giving it visual coherence the style loss is where the deep learning keeps in that one is defined using a deep convolutional neural network precisely it consists in a sum of l2 distances between the gram matrices of the representations of the base image and the style reference image extracted from different layers of a convnet trained on imagenet the general idea is to capture colortexture information at different spatial scales fairly large scales defined by the depth of the layer considered the content loss is a l2 distance between the features of the base image extracted from a deep layer and the features of the combination image keeping the generated image close enough to the original one reference a neural algorithm of artistic style http arxiv orgabs1508 06576 setup weights of the different loss components dimensions of the generated picture let s take a look at our base content image and our style reference image image preprocessing deprocessing utilities util function to open resize and format pictures into appropriate tensors util function to convert a tensor into a valid image remove zerocenter by mean pixel bgr rgb compute the style transfer loss first we need to define 4 utility functions grammatrix used to compute the style loss the styleloss function which keeps the generated image close to the local textures of the style reference image the contentloss function which keeps the highlevel representation of the generated image close to that of the base image the totalvariationloss function a regularization loss which keeps the generated image locallycoherent the gram matrix of an image tensor featurewise outer product the style loss is designed to maintain the style of the reference image in the generated image it is based on the gram matrices which capture style of feature maps from the style reference image and from the generated image an auxiliary loss function designed to maintain the content of the base image in the generated image the 3rd loss function total variation loss designed to keep the generated image locally coherent next let s create a feature extraction model that retrieves the intermediate activations of vgg19 as a dict by name build a vgg19 model loaded with pretrained imagenet weights get the symbolic outputs of each key layer we gave them unique names set up a model that returns the activation values for every layer in vgg19 as a dict finally here s the code that computes the style transfer loss list of layers to use for the style loss the layer to use for the content loss initialize the loss add content loss add style loss add total variation loss add a tf function decorator to loss gradient computation to compile it and thus make it fast the training loop repeatedly run vanilla gradient descent steps to minimize the loss and save the resulting image every 100 iterations we decay the learning rate by 0 96 every 100 steps after 4000 iterations you get the following result example available on huggingface trained model demo generic badgehttps img shields iobadgef09fa49720modelneural20style20transferblack svghttps huggingface cokerasiovgg19 generic badgehttps img shields iobadgef09fa49720spacesneural20style20transferblack svghttps huggingface cospaceskerasioneuralstyletransfer introduction style transfer consists in generating an image with the same content as a base image but with the style of a different picture typically artistic this is achieved through the optimization of a loss function that has 3 components style loss content loss and total variation loss the total variation loss imposes local spatial continuity between the pixels of the combination image giving it visual coherence the style loss is where the deep learning keeps in that one is defined using a deep convolutional neural network precisely it consists in a sum of l2 distances between the gram matrices of the representations of the base image and the style reference image extracted from different layers of a convnet trained on imagenet the general idea is to capture color texture information at different spatial scales fairly large scales defined by the depth of the layer considered the content loss is a l2 distance between the features of the base image extracted from a deep layer and the features of the combination image keeping the generated image close enough to the original one reference a neural algorithm of artistic style http arxiv org abs 1508 06576 setup weights of the different loss components dimensions of the generated picture let s take a look at our base content image and our style reference image image preprocessing deprocessing utilities util function to open resize and format pictures into appropriate tensors util function to convert a tensor into a valid image remove zero center by mean pixel bgr rgb compute the style transfer loss first we need to define 4 utility functions gram_matrix used to compute the style loss the style_loss function which keeps the generated image close to the local textures of the style reference image the content_loss function which keeps the high level representation of the generated image close to that of the base image the total_variation_loss function a regularization loss which keeps the generated image locally coherent the gram matrix of an image tensor feature wise outer product the style loss is designed to maintain the style of the reference image in the generated image it is based on the gram matrices which capture style of feature maps from the style reference image and from the generated image an auxiliary loss function designed to maintain the content of the base image in the generated image the 3rd loss function total variation loss designed to keep the generated image locally coherent next let s create a feature extraction model that retrieves the intermediate activations of vgg19 as a dict by name build a vgg19 model loaded with pre trained imagenet weights get the symbolic outputs of each key layer we gave them unique names set up a model that returns the activation values for every layer in vgg19 as a dict finally here s the code that computes the style transfer loss list of layers to use for the style loss the layer to use for the content loss initialize the loss add content loss add style loss add total variation loss add a tf function decorator to loss gradient computation to compile it and thus make it fast the training loop repeatedly run vanilla gradient descent steps to minimize the loss and save the resulting image every 100 iterations we decay the learning rate by 0 96 every 100 steps after 4000 iterations you get the following result example available on huggingface trained model demo generic badge https img shields io badge f0 9f a4 97 20model neural 20style 20transfer black svg https huggingface co keras io vgg19 generic badge https img shields io badge f0 9f a4 97 20spaces neural 20style 20transfer black svg https huggingface co spaces keras io neural style transfer
import numpy as np
import tensorflow as tf
import keras
from keras.applications import vgg19

base_image_path = keras.utils.get_file(
    "paris.jpg", "https://i.imgur.com/F28w3Ac.jpg"
)
style_reference_image_path = keras.utils.get_file(
    "starry_night.jpg", "https://i.imgur.com/9ooB60I.jpg"
)
result_prefix = "paris_generated"


total_variation_weight = 1e-6
style_weight = 1e-6
content_weight = 2.5e-8


width, height = keras.utils.load_img(base_image_path).size
img_nrows = 400
img_ncols = int(width * img_nrows / height)



from IPython.display import Image, display

display(Image(base_image_path))
display(Image(style_reference_image_path))




def preprocess_image(image_path):
    
    img = keras.utils.load_img(image_path, target_size=(img_nrows, img_ncols))
    img = keras.utils.img_to_array(img)
    img = np.expand_dims(img, axis=0)
    img = vgg19.preprocess_input(img)
    return tf.convert_to_tensor(img)


def deprocess_image(x):
    
    x = x.reshape((img_nrows, img_ncols, 3))
    
    x[:, :, 0] += 103.939
    x[:, :, 1] += 116.779
    x[:, :, 2] += 123.68
    
    x = x[:, :, ::-1]
    x = np.clip(x, 0, 255).astype("uint8")
    return x







def gram_matrix(x):
    x = tf.transpose(x, (2, 0, 1))
    features = tf.reshape(x, (tf.shape(x)[0], -1))
    gram = tf.matmul(features, tf.transpose(features))
    return gram









def style_loss(style, combination):
    S = gram_matrix(style)
    C = gram_matrix(combination)
    channels = 3
    size = img_nrows * img_ncols
    return tf.reduce_sum(tf.square(S - C)) / (
        4.0 * (channels**2) * (size**2)
    )







def content_loss(base, combination):
    return tf.reduce_sum(tf.square(combination - base))






def total_variation_loss(x):
    a = tf.square(
        x[:, : img_nrows - 1, : img_ncols - 1, :] - x[:, 1:, : img_ncols - 1, :]
    )
    b = tf.square(
        x[:, : img_nrows - 1, : img_ncols - 1, :] - x[:, : img_nrows - 1, 1:, :]
    )
    return tf.reduce_sum(tf.pow(a + b, 1.25))





model = vgg19.VGG19(weights="imagenet", include_top=False)


outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])



feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)




style_layer_names = [
    "block1_conv1",
    "block2_conv1",
    "block3_conv1",
    "block4_conv1",
    "block5_conv1",
]

content_layer_name = "block5_conv2"


def compute_loss(combination_image, base_image, style_reference_image):
    input_tensor = tf.concat(
        [base_image, style_reference_image, combination_image], axis=0
    )
    features = feature_extractor(input_tensor)

    
    loss = tf.zeros(shape=())

    
    layer_features = features[content_layer_name]
    base_image_features = layer_features[0, :, :, :]
    combination_features = layer_features[2, :, :, :]
    loss = loss + content_weight * content_loss(
        base_image_features, combination_features
    )
    
    for layer_name in style_layer_names:
        layer_features = features[layer_name]
        style_reference_features = layer_features[1, :, :, :]
        combination_features = layer_features[2, :, :, :]
        sl = style_loss(style_reference_features, combination_features)
        loss += (style_weight / len(style_layer_names)) * sl

    
    loss += total_variation_weight * total_variation_loss(combination_image)
    return loss





@tf.function
def compute_loss_and_grads(
    combination_image, base_image, style_reference_image
):
    with tf.GradientTape() as tape:
        loss = compute_loss(
            combination_image, base_image, style_reference_image
        )
    grads = tape.gradient(loss, combination_image)
    return loss, grads




optimizer = keras.optimizers.SGD(
    keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96
    )
)

base_image = preprocess_image(base_image_path)
style_reference_image = preprocess_image(style_reference_image_path)
combination_image = tf.Variable(preprocess_image(base_image_path))

iterations = 4000
for i in range(1, iterations + 1):
    loss, grads = compute_loss_and_grads(
        combination_image, base_image, style_reference_image
    )
    optimizer.apply_gradients([(grads, combination_image)])
    if i % 100 == 0:
        print("Iteration %d: loss=%.2f" % (i, loss))
        img = deprocess_image(combination_image.numpy())
        fname = result_prefix + "_at_iteration_%d.png" % i
        keras.utils.save_img(fname, img)



display(Image(result_prefix + "_at_iteration_4000.png"))