<jupyter_start><jupyter_text>**Neural Style Transfer for Artistic Style Pictures** Author: <NAME>Created: 23/7/2020Last update: 8/8/2020This notebook was created after my course in the [Deep Learning Specialization](https://www.coursera.org/learn/convolutional-neural-networks) from [Deeplearning.ai](https://www.deeplearning.ai/). Some of the notes here are from the course assignment notebooks and some of the code are from this [notebook](https://www.kaggle.com/sayakdasgupta/neural-style-transfer-using-vgg19) by [sayak]( https://www.kaggle.com/sayakdasgupta).**Neural Style Transfer (NST)** is a machine learning optimization technique that uses 2 images - a content image and a style image, and blends them together so the output looks like the content image, but 'painted' in the style of the style image. This algorithm was first introduced by Gatys et al. in the paper called [A Neural Algorithm of Artistic Style](https://arxiv.org/pdf/1508.06576.pdf). The core innovation of NST is the use of deep learning to minimize the generated image's content and style so that it matches that of the content and style images. By doing so, it allows us to compose our own pictures in the style of any other paintings/pictures ranging from google images to famous paintings.This notebook will go through the implementation of NST and the process is as follows:1. [Preparing Data](DataPrep)2. [Image Preprocessing](Preprocess)3. [Model Setup](Model)4. [Cost Function](Cost)    4.1. [Content Cost Function](ContentCost)    4.2. [Style Cost Function](StyleCost)    5. [Train Model](Train)   6. [Conclusion](Conclusion)  1. Preparing DataWe start by loading the dependencies, set the filepaths of our content and style iamges, and plot out some of them.<jupyter_code>import numpy as np 
import pandas as pd 

import matplotlib.pyplot as plt
from matplotlib.image import imread
from PIL import Image
import IPython.display

import tensorflow as tf
import keras.preprocessing.image as process_im
from keras.models import Model
from tensorflow.python.keras import models 
from tensorflow.python.keras import losses
from tensorflow.python.keras import layers
from tensorflow.python.keras import backend as K

import functools

# Set paths to content and style pictures

content_paths = {'rainbow': '/kaggle/input/pictures/DSC04907_IG.jpg',
                'rainbow2': '/kaggle/input/pictures/DSC05029_IG.jpg',
                'tower': '/kaggle/input/pictures/DSC06205-Edit_IG.jpg',
                'mp': '/kaggle/input/pictures/LRM_EXPORT_20190925_190945.jpg'}
style_paths = {'vg': '/kaggle/input/pictures/vangogh.jpeg',
              'umbrella': '/kaggle/input/pictures/rainprincess.jpg',
              'face': '/kaggle/input/pictures/face.png',
              'scream': '/kaggle/input/pictures/the-scream.jpg'}
# Plot sample images

def plot_samples(path1, path2):  
    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10,5))
    image = imread(path1)
    ax[0].imshow(image)
    ax[0].axis('off')
    ax[0].set_title("Content Image", fontsize=20)
    image = imread(path2)
    ax[1].imshow(image)
    ax[1].axis('off')
    ax[1].set_title("Style Image", fontsize=20)
    
%matplotlib inline 
plot_samples(content_paths['rainbow'], style_paths['face'])<jupyter_output><empty_output><jupyter_text>2. Image PreprocessingHere we write some functions to prepare the image for inputting into the convolutional neural network.<jupyter_code># load image as array
def load_file(image_path):
    image =  Image.open(image_path)
    max_dim = 512
    factor = max_dim/max(image.size)
    image = image.resize((round(image.size[0]*factor),round(image.size[1]*factor)),
                         Image.ANTIALIAS)
    im_array = process_im.img_to_array(image)
    #adding extra axis to the array as to generate a batch of single image 
    im_array = np.expand_dims(im_array,axis=0) 
    return im_array

# preprocess image to vgg19 input requirements
def img_preprocess(img_path):
    image = load_file(img_path)
    img = tf.keras.applications.vgg19.preprocess_input(image)
    return img

# inverse preprocess step
def deprocess_img(processed_img):
  x = processed_img.copy()
  if len(x.shape) == 4:
    x = np.squeeze(x, 0)
  # Input dimension must be [1, height, width, channel] or [height, width, channel]
  assert len(x.shape) == 3 
  
  
  # perform the inverse of the preprocessing step
  x[:, :, 0] += 103.939
  x[:, :, 1] += 116.779
  x[:, :, 2] += 123.68
  x = x[:, :, ::-1] # converting BGR to RGB channel

  x = np.clip(x, 0, 255).astype('uint8')
  return x<jupyter_output><empty_output><jupyter_text>It is always good to test out our functions to make sure they work.<jupyter_code># Test image loading functions

img = img_preprocess(content_paths['rainbow'])

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10,5))
ax[0].imshow(img[0])
ax[0].axis('off')
ax[0].set_title("Processed Image", fontsize=20)

img_deprocess = deprocess_img(img)
ax[1].imshow(img_deprocess)
ax[1].axis('off')
ax[1].set_title("Deprocessed Image", fontsize=20)

plt.show()<jupyter_output><empty_output><jupyter_text>3. Model SetupNeural style transfer works by passing the content and style image through the neural network and extracting out the features/activations in certain layers. Here we select the layers that we want to extract the content and features of the images. For the content, we usually want to select a layer from the middle of the CNN as the earlier layers usually contain lower-level features and vice versa for the later layers. However, there is no 'right' way and instead, we can treat this as a hyperparameter that can be tuned to improve the final generated image we achieve. For the style layers, we usually select multiple layers which contain low-level to high-level features.For the CNN, we will be using VGG19 as it has been found to perform the best.<jupyter_code># set the layers to get content and style 
content_layers = ['block5_conv2']
style_layers = ['block1_conv1',
                'block2_conv1',
                'block3_conv1', 
                'block4_conv1', 
                'block5_conv1']
number_content=len(content_layers)
number_style =len(style_layers)
def get_model():
    # get vgg model
    vgg=tf.keras.applications.vgg19.VGG19(include_top=False,weights='imagenet')
    vgg.trainable=False
    # get the content and style layers
    content_output=[vgg.get_layer(layer).output for layer in content_layers]
    style_output=[vgg.get_layer(layer).output for layer in style_layers]
    model_output= style_output+content_output
    return models.Model(vgg.input,model_output) # set model input and output

model = get_model()
model.summary()
model.output<jupyter_output><empty_output><jupyter_text>4. Cost FunctionThe cost function is the most important idea in neural style transfer. Here, I am using cost and loss interchangeably. The overall idea is simple. We want the generated image to have the same content as the content image while having the same style as the style image. Therefore, we are looking to reduce the cost/loss of the generated image, which we define as the sum of its content cost and style cost:$$\Large J_{generated}(G) = \alpha J_{content}(C,G) + \beta J_{style}(S,G)\tag{1}$$* J is the cost/loss* α and β are relative weightings applied to the content and style cost respectively* G is "Generated", C is "Content", S is "Style"The idea is to use gradient descent to minimize `Jgenerated`, and use it to update the generated Image by the following steps:1. Initialize `G` (generated image pixels) randomly, for e.g., 100x100x3 2. Use gradient descent to minimize `Jgenerated` and update `G`  4.1. Content Cost Function The idea of the content cost is that you want the content of the generated image to be similar to the content image. Therefore, we want the activations of the generated image and content image at the selected layer to be similar. To compute this disimilarity, we can use the **euclidean distance** of the 2 activations and minimize this disimilarity/cost:$$\Large J_{content}(C,G) = \sum _{ \text{all entries}} (a^{(C)} - a^{(G)})^2\tag{2} $$* a is the activation of a layer in the neural network, C is content and G is generated***Note:*** the original paper applies a normalization constant of 1/2 to the cost/euclidean distance. However this can be accounted for in the weighting hyperparameter applied to the cost, so it can be omitted here.This picture gives an idea of the activations of a convolutional layer:<jupyter_code>def get_content_loss(generated_content_feature,content_features):
    # equation 2
    loss = tf.reduce_mean(tf.square(generated_content_feature-content_features))
    return loss<jupyter_output><empty_output><jupyter_text>4.2. Style Cost FunctionThe style of an image is harder to understand. However, it is a simple idea that involves a gram matrix or style matrix, which is simply a correlation matrix of a layer's channels/features. To be correct, it is actually cross-correlation that is used, which is a unnormalized correlation. Gram/style matrix* The style matrix is also called a gram matrix. * In linear algebra, the Gram matrix $G_{gram}$ of a set of vectors $(v_{1},\dots ,v_{n})$ is the matrix of dot products, whose entries are ${\displaystyle G_{(gram)i,j} = v_{i}^T v_{j} = np.dot(v_{i}, v_{j})  }$. * In other words, $G_{(gram),i,j}$ compares how similar $v_i$ is to $v_j$: If they are highly similar, you would expect them to have a large dot product, and thus for $G_{(gram)i,j}$ to be large.  $G_{(gram)i,j}$: correlationThe result is a matrix of dimension $(n_C,n_C)$ where $n_C$ is the number of filters (channels). The value $G_{(gram)i,j}$ measures how similar the activations of filter $i$ are to the activations of filter $j$. *Note:* $G_{(gram),i,i}$: prevalence of patterns or textures* The diagonal elements $G_{(gram)ii}$ measure how "active" a filter $i$ is. * For example, suppose filter $i$ is detecting vertical textures in the image. Then $G_{(gram)ii}$ measures how common  vertical textures are in the image as a whole.* If $G_{(gram)ii}$ is large, this means that the image has a lot of vertical texture. By capturing the prevalence of different types of features $G_{(gram)ii}$, as well as how much different features occur together $G_{(gram)ij}$, the Style matrix $G_{gram}$ measures the style of an image.  ExampleHere is an example of activations from a layer of a convolutional neural network. Each 9x9 corresponds to one neuron/channel. The channel with the red border is detecting vertical features while the channel with the green border is looking for orange colored patches in an image. If these two channels are highly correlated, it means that whenever a part of the image has these kind of vertical texture, then it will also probably have an orange hue. Therefore, the correlation tells us which of these features tend to occur or not occur together in different parts of an image. Style CostThe idea of the style cost is that we want the style of the generated image to be similar to the style image. Therefore, we want the style/gram matrix of the generated image and style image at the selected layer to be similar. To compute this disimilarity, we can use the euclidean distance of the 2 gram/style matrix **(Frobenius norm)** and minimize this disimilarity/cost: $$\Large J_{style}^{[l]}(S,G) = \sum _{i=1}^{n_C}\sum_{j=1}^{n_C}(G^{(S)}_{(gram)i,j} - G^{(G)}_{(gram)i,j})^2\tag{3} $$* $G_{gram}^{(S)}$ Gram matrix of the "style" image.* $G_{gram}^{(G)}$ Gram matrix of the "generated" image.* Remember, this cost is computed using the hidden layer activations for a particular hidden layer in the network $a^{[l]}$<jupyter_code>def gram_matrix(tensor):
    # refer to picture above for better visualization
    channels=int(tensor.shape[-1]) # get number of channels
    vector=tf.reshape(tensor,[-1,channels]) # unroll into 2d matrix (h*w,channels)
    n=tf.shape(vector)[0]
    gram_matrix=tf.matmul(vector,vector,transpose_a=True) # compute gram matrix
    return gram_matrix/tf.cast(n,tf.float32)

def get_style_loss(generated_style_features,style_gram_matrix):
    # get gram matrix of generated activations
    generated_gram_matrix = gram_matrix(generated_style_features) 
    # frobenius norm of gram matrices -- equation 3
    loss = tf.reduce_mean(tf.square(style_gram_matrix-generated_gram_matrix)) 
    return loss
# extract out features/activations of content and style images 
# from model output
def get_features(model,content_path,style_path):
    # preprocess images
    content_img=img_preprocess(content_path)
    style_image=img_preprocess(style_path)
    
    # output of model for each image
    content_output=model(content_img)
    style_output=model(style_image)
    
    # extract out the features/activations, content is last layer
    content_feature = [layer[0] for layer in content_output[number_style:]]
    # style has few layers
    style_feature = [layer[0] for layer in style_output[:number_style]]
    return content_feature,style_feature<jupyter_output><empty_output><jupyter_text>Since we are using many layers for the style of the image, we have to combine them. This is done through applying weightings `λ` to the layers. `λ` can be tuned to give more weightings to specific layers of the network. For example, if we want high level styles, we will give more weightings to the higher layers of the network. $$\Large J_{style}(S,G) = \sum_{l} \lambda^{[l]} J^{[l]}_{style}(S,G)\tag{4}$$<jupyter_code>def compute_loss(model, loss_weights,image, style_gram_matrix, content_features):
    #style weight and content weight are user given parameters
    #that define what percentage of content and/or style will be preserved 
    # in the generated image
    style_weight,content_weight = loss_weights 
    
    output=model(image)
    content_loss=0
    style_loss=0
    
    # extract content,style activations of generated image
    generated_style_features = output[:number_style]
    generated_content_feature = output[number_style:]
    
    # compute style loss
    weight_per_layer = 1.0/float(number_style) # our λ weighting is equal here
    for a,b in zip(style_gram_matrix,generated_style_features):
        style_loss+=weight_per_layer*get_style_loss(b[0],a) # equation 4
        
    # compute content loss
    weight_per_layer =1.0/ float(number_content)
    for a,b in zip(generated_content_feature,content_features):
        content_loss+=weight_per_layer*get_content_loss(a[0],b)
    
    # apply α and β in equation 1
    style_loss *= style_weight
    content_loss *= content_weight
    # get total loss -- equation 1
    total_loss = content_loss + style_loss
    
    
    return total_loss,style_loss,content_loss
def compute_grads(dictionary):
    with tf.GradientTape() as tape:
        all_loss=compute_loss(**dictionary)
        
    total_loss=all_loss[0]
    # return gradients and loss
    return tape.gradient(total_loss,dictionary['image']),all_loss<jupyter_output><empty_output><jupyter_text>5. Train ModelFinally, let's put all the functions together to perform neural style transfer! Remember to turn on GPU for faster training. With GPU, it takes about 10 times faster to compose one image for me!<jupyter_code>def run_style_transfer(content_path,style_path,epochs=500, content_weight=1e3, style_weight=1e-3, show_img=False):
    
    model=get_model()
    
    for layer in model.layers:
        layer.trainable = False
    
    # get activations and style matrix
    content_feature,style_feature = get_features(model,content_path,style_path)
    # get gram matrix of style image
    style_gram_matrix=[gram_matrix(feature) for feature in style_feature]
    
    # initialize generated image
    generated = img_preprocess(content_path)
    generated = tf.Variable(generated,dtype=tf.float32)
    
    optimizer = tf.keras.optimizers.Adam(learning_rate=5, beta_1=0.99, epsilon=1e-1)
    
    best_loss,best_img=float('inf'),None
    
    # relative weightings of content and style image
    loss_weights = (style_weight, content_weight)
    dictionary={'model':model,
              'loss_weights':loss_weights,
              'image':generated,
              'style_gram_matrix':style_gram_matrix,
              'content_features':content_feature}
    
    # for clipping image
    norm_means = np.array([103.939, 116.779, 123.68])
    min_vals = -norm_means
    max_vals = 255 - norm_means   
  
    imgs = []
    for i in range(epochs):
        # run gradient descent and update generated image
        grad,all_loss = compute_grads(dictionary)
        total_loss,style_loss,content_loss = all_loss
        optimizer.apply_gradients([(grad,generated)])
        clipped=tf.clip_by_value(generated,min_vals,max_vals)
        generated.assign(clipped)
        
        if total_loss<best_loss:
            best_loss = total_loss
            best_img = deprocess_img(generated.numpy())
            
        # for visualization 
        # print out every 5 epochs
        
        if show_img:
            if i%5==0:
                plot_img = generated.numpy()
                plot_img = deprocess_img(plot_img)
                imgs.append(plot_img)
                IPython.display.clear_output(wait=True)
                IPython.display.display_png(Image.fromarray(plot_img))
                print('Epoch: {}'.format(i))        
                print('Total loss: {:.4e}, ' 
                  'style loss: {:.4e}, '
                  'content loss: {:.4e}, '.format(total_loss, style_loss, content_loss))
    
    IPython.display.clear_output(wait=True)
    
    
    return best_img,best_loss,imgs<jupyter_output><empty_output><jupyter_text>Feel free to experiment with some of the hyperparameters, such as `epochs`, `content_weight`(α) and `style_weight`(β) to compose the best artistic images! What I usually do is to try out a few `content_weights`, settle on one and then keep it constant while tuning `epochs` and `style_weights` to get a desired balanced effect. Here are some examples of performing NST on my pictures in Peru:<jupyter_code>best, best_loss,image = run_style_transfer(content_paths['rainbow'], 
                                           style_paths['face'], epochs=1000,
                                           style_weight=1e-4,
                                           show_img=True)
def plot_images(generated, content_path, style_path):
    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(15,5))
    ax[0].imshow(generated)
    ax[0].axis('off')
    ax[0].set_title("Generated Image", fontsize=20)
    
    image = imread(content_path)
    ax[1].imshow(image)
    ax[1].axis('off')
    ax[1].set_title("Content Image", fontsize=20)
    
    image = imread(style_path)
    ax[2].imshow(image)
    ax[2].axis('off')
    ax[2].set_title("Style Image", fontsize=20)
plot_images(best, content_paths['rainbow'], style_paths['face'])<jupyter_output><empty_output><jupyter_text>Animated Neural Style Transfer<jupyter_code>best, best_loss,image = run_style_transfer(content_paths['mp'], 
                                           style_paths['vg'], epochs=1000,
                                           style_weight=1e-3)
plot_images(best, content_paths['mp'], style_paths['vg'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09438762265718291, 'fraction_numerical': 0.01500441306266549, 'mean_word_length': 3.9746900826446283, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '556824', 'license': 'MIT', 'n_tokens_mistral': 5871, 'n_tokens_neox': 5608, 'n_words': 1951, 'path': 'Neural Style Transfer For Unique Artistic Photos! .ipynb', 'repo_name': 'teyang-lau/Neural_Style_Transfer', 'size': '19266'}