<jupyter_start><jupyter_text>The Joy of Neural Painting (2/4)[Neural Painters](https://arxiv.org/abs/1904.08410) [1] are a class of models that can be seen as a fully differentiable simulation of a particular non-differentiable painting program, in other words, the machine "paints" by successively generating brushstrokes (i.e., actions that defines a brushstrokes) and applying them on a canvas, as an artist would do.---[1] *Neural Painters: A Learned Differentiable Constraint for Generating Brushstroke Paintings*. <NAME>. arXiv preprint [arXiv:1904.08410](https://arxiv.org/abs/1904.08410), 2019.  Training the Critic as a Binary Classifier (Non-Adversarially)The goal in this stage is to train the `Critic` on a a binary classification task: to discriminate between true and generated (fake) brushstrokes. We will use the `Generator` trained previously to generate strokes. The `Generator`'s parameters will be frozen, that is, they will not be updated in this stage.<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms.functional as VF
import torch.optim as optim
import fastai
import numpy as np
import matplotlib.pyplot as plt
import time
import json
import PIL
import os

from torchvision import datasets, transforms
from torchvision.utils import save_image
from torch.autograd import Variable
from torch.utils.data import TensorDataset, DataLoader
from glob import glob
from torchvision import models
from pathlib import Path
from IPython.display import display, clear_output

# from fastai.vision import *
from fastai.torch_core import requires_grad, children
from fastai.vision.gan import basic_generator, gan_critic
from fastai.callbacks import hook_outputs

print('torch.__version__', torch.__version__)<jupyter_output>torch.__version__ 1.1.0<jupyter_text>Dataset For the purposes of this excercise we will use a collection of 100K brushstrokes, that is a single `episodes` file from the full `neural-painters` dataset available in Kaggle: https://www.kaggle.com/reiinakano/mypaint_brushstrokesWe have made available the file using Google Drive: [episodes_42.npz](https://drive.google.com/a/libreai.com/uc?id=1Fl9K6XbDr_5lOb0J-lzjPX9SoJ9M7j1x&export=download). Given its size (196M) `wget` or `curl` will fail because of the security warning from Google Drive, we will therefore go with [gdwon](https://pypi.org/project/gdown/) to download it.<jupyter_code>!pip install gdown
# create a folder called `data` and donwload the dataset there
!mkdir data; cd data; gdown https://drive.google.com/uc?id=1Fl9K6XbDr_5lOb0J-lzjPX9SoJ9M7j1x
data_fname = 'episodes_42.npz'
loaded = np.load('data/' + data_fname)
w=800
h=600

fig=plt.figure(figsize=(30, 10))
cols = 12

smaller_arr = loaded['strokes'][:cols]

for col in range(cols):
    img = smaller_arr[col][:, :, :3]
    fig.add_subplot(1, 20, col+1)
    plt.grid(False)
    plt.imshow(img)
plt.show()
loaded['actions'][:cols]<jupyter_output><empty_output><jupyter_text>Train a Critic as Binary Classifier: True or Fake images with Generator frozen<jupyter_code># a helper class to encapsulate some parameters
class Params():
    def __init__(self):
        super(Params)
params = Params()

params.condition_dim = 12
params.noise_dim = 0
params.image_size = 64
params.channels = 3
params.batch_size = 512
params.lr = 1e-3
params.train_epochs = 10
params.fc_dim = int(params.image_size * params.image_size)
critic = gan_critic(nf=64)
critic.cuda()<jupyter_output><empty_output><jupyter_text>Load the Pre-Trained Generator The `Generator` is defined as a [`Linear Layer`](https://pytorch.org/docs/stable/nn.htmllinear) followed by a [`Deep Convolutional Neural Network`](https://docs.fast.ai/vision.gan.htmlbasic_generator)<jupyter_code>class GeneratorX(nn.Module):

    def __init__(self, condition_dim, fc_dim, in_size, n_channels, n_extra_layers):
        super(GeneratorX, self).__init__()
        self.fc = nn.Linear(condition_dim, fc_dim)
        self.generator = basic_generator(in_size=in_size, n_channels=n_channels, n_extra_layers=n_extra_layers, noise_sz=fc_dim)

    def forward(self, x):
        x = self.fc(x)
        x = self.generator(x.view(x.size(0), x.size(1), 1, 1))
        return x


generator = GeneratorX(condition_dim=params.condition_dim,
                fc_dim=params.fc_dim,
                in_size=params.image_size,
                n_channels=params.channels,
                n_extra_layers=1
               )

generator<jupyter_output><empty_output><jupyter_text>Here you have to load the pre-trained `Generator` from the previous phase.<jupyter_code>!ls models
# Just in case you do not have your `Generator` at hand, you can uncomment the following lines 
# to download one we trained:
#!mkdir models
#!cd models; gdown https://drive.google.com/uc?id=1knOYFm-PBk0ovwPoVjV5ivc6-ts-8eHG<jupyter_output>Downloading...
From: https://drive.google.com/uc?id=1knOYFm-PBk0ovwPoVjV5ivc6-ts-8eHG
To: /content/models/neural_painter_x_non_adversarial_generator_param.pth
146MB [00:02, 58.5MB/s]<jupyter_text>No let's load the `Generator` and freeze its parameters:<jupyter_code>generator.load_state_dict(torch.load("models/neural_painter_x_non_adversarial_generator_param.pth"))
# freeze the generator parameters
for param in generator.parameters():
    generator.requires_grad = False
generator.cuda().eval();<jupyter_output><empty_output><jupyter_text>---  Learn the Critic<jupyter_code>critic_optimizer = optim.Adam(critic.parameters(), lr=params.lr, betas=(0.5, 0.999))
critic_loss = torch.nn.functional.binary_cross_entropy
strokes_tfms = transforms.Compose([transforms.Lambda(lambda x: x.float()),
                                   transforms.Lambda(lambda x: x.div_(255))
                                  ])

strokes = loaded['strokes']
conditions = loaded['actions']

strokes = torch.stack([torch.from_numpy(s).permute(2, 0, 1) for s in strokes])
conditions = torch.stack([torch.from_numpy(c).float() for c in conditions])

dataset = TensorDataset(strokes, conditions)
dataloader = DataLoader(dataset, batch_size=params.batch_size, shuffle=False)

root = 'models/'
model = 'neural_painter_x_'

if not os.path.isdir(root):
    os.mkdir(root)
    
critic_losses = []

print('training start!')


for epoch in range(params.train_epochs):

  losses = []

  for num_iter, (stroke, condition) in enumerate(dataloader):  
    condition = condition.cuda()
    stroke = torch.stack([strokes_tfms(s) for s in stroke]).cuda()

    # ground truths
    
    batch_size = stroke.shape[0]
    
    valid = torch.ones((batch_size, 1), dtype=torch.float, requires_grad=False).cuda()
    
    fake = torch.zeros((batch_size, 1), dtype=torch.float, requires_grad=False).cuda()

    # -----------------
    #  Train Critic
    # -----------------
    critic_optimizer.zero_grad()

    # computing the loss for true strokes
    critic_true_preds = torch.sigmoid(critic(stroke))
    critic_true_loss = critic_loss(critic_true_preds, valid)

    # computing the loss for generated (aka fake) strokes
    generated_stroke = generator(condition)
    critic_fake_preds = torch.sigmoid(critic(generated_stroke))
    critic_fake_loss = critic_loss(critic_fake_preds, fake)

    # taking the tarain loss as the average
    critic_train_loss = (critic_true_loss + critic_fake_loss) / 2.

    critic_train_loss.backward()
    critic_optimizer.step()

    critic_losses.append(critic_train_loss.data.item())

    if (num_iter % 25) == 0:
        print('epoch {}/{}  {}/{} complete!'.format((epoch + 1), 
                                                    params.train_epochs, 
                                                    (num_iter+1), 
                                                    len(dataloader)))
        print('loss: {}'.format(torch.mean(torch.FloatTensor(critic_losses))))
        # ---
        s = stroke[0].clone().cpu()
        print('real brushstroke')
        print('critic pred:', critic_true_preds[0].item(), '(true=1)')
        display(VF.to_pil_image(s))
        # ---
        g = generated_stroke[0].clone().cpu()
        print('generated brushstroke')
        print('critic pred: ', critic_fake_preds[0].item(), '(true=0)')
        display(VF.to_pil_image(g))
        print('---')
            
print('saving Critic ...')
torch.save(critic.state_dict(), root + model + 'non_adversarial_' + 'critic_param.pth')        
print('done.')
!ls models<jupyter_output>neural_painter_x_non_adversarial_critic_param.pth
neural_painter_x_non_adversarial_generator_param.pth<jupyter_text>~ Fin of this Part ~<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11278107887684959, 'fraction_numerical': 0.017476406850751487, 'mean_word_length': 4.019883040935673, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '579853', 'license': 'MIT', 'n_tokens_mistral': 2948, 'n_tokens_neox': 2804, 'n_words': 668, 'path': 'notebooks/02_neural_painter_x_training_Critic_Non_Adversarial.ipynb', 'repo_name': 'libreai/neural-painter-x', 'size': '8593'}