<jupyter_start><jupyter_text>Generative Adversarial Networks in Keras<jupyter_code>%matplotlib inline
import importlib
import utils2; importlib.reload(utils2)
from utils2 import *

from tqdm import tqdm<jupyter_output>Using TensorFlow backend.
/home/jhoward/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>The original GAN! See [this paper](https://arxiv.org/abs/1406.2661) for details of the approach we'll try first for our first GAN. We'll see if we can generate hand-drawn numbers based on MNIST, so let's load that dataset first.We'll be refering to the discriminator as 'D' and the generator as 'G'.<jupyter_code>from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train.shape
n = len(X_train)
X_train = X_train.reshape(n, -1).astype(np.float32)
X_test = X_test.reshape(len(X_test), -1).astype(np.float32)
X_train /= 255.; X_test /= 255.<jupyter_output><empty_output><jupyter_text>Train This is just a helper to plot a bunch of generated images.<jupyter_code>def plot_gen(G, n_ex=16):
    plot_multi(G.predict(noise(n_ex)).reshape(n_ex, 28,28), cmap='gray')<jupyter_output><empty_output><jupyter_text>Create some random data for the generator.<jupyter_code>def noise(bs): return np.random.rand(bs,100)<jupyter_output><empty_output><jupyter_text>Create a batch of some real and some generated data, with appropriate labels, for the discriminator.<jupyter_code>def data_D(sz, G):
    real_img = X_train[np.random.randint(0,n,size=sz)]
    X = np.concatenate((real_img, G.predict(noise(sz))))
    return X, [0]*sz + [1]*sz
def make_trainable(net, val):
    net.trainable = val
    for l in net.layers: l.trainable = val<jupyter_output><empty_output><jupyter_text>Train a few epochs, and return the losses for D and G. In each epoch we:1. Train D on one batch from data_D()2. Train G to create images that the discriminator predicts as real.<jupyter_code>def train(D, G, m, nb_epoch=5000, bs=128):
    dl,gl=[],[]
    for e in tqdm(range(nb_epoch)):
        X,y = data_D(bs//2, G)
        dl.append(D.train_on_batch(X,y))
        make_trainable(D, False)
        gl.append(m.train_on_batch(noise(bs), np.zeros([bs])))
        make_trainable(D, True)
    return dl,gl<jupyter_output><empty_output><jupyter_text>MLP GAN We'll keep thinks simple by making D & G plain ole' MLPs.<jupyter_code>MLP_G = Sequential([
    Dense(200, input_shape=(100,), activation='relu'),
    Dense(400, activation='relu'),
    Dense(784, activation='sigmoid'),
])
MLP_D = Sequential([
    Dense(300, input_shape=(784,), activation='relu'),
    Dense(300, activation='relu'),
    Dense(1, activation='sigmoid'),
])
MLP_D.compile(Adam(1e-4), "binary_crossentropy")
MLP_m = Sequential([MLP_G,MLP_D])
MLP_m.compile(Adam(1e-4), "binary_crossentropy")
dl,gl = train(MLP_D, MLP_G, MLP_m, 8000)<jupyter_output>100%|██████████| 4000/4000 [00:47<00:00, 84.49it/s]    | 8/4000 [00:00<00:55, 71.74it/s]<jupyter_text>The loss plots for most GANs are nearly impossible to interpret - which is one of the things that make them hard to train.<jupyter_code>plt.plot(dl[100:])
plt.plot(gl[100:])<jupyter_output><empty_output><jupyter_text>This is what's known in the literature as "mode collapse".<jupyter_code>plot_gen()<jupyter_output><empty_output><jupyter_text>OK, so that didn't work. Can we do better?...  DCGAN There's lots of ideas out there to make GANs train better, since they are notoriously painful to get working. The [paper introducing DCGANs](https://arxiv.org/abs/1511.06434) is the main basis for our next section. Add see https://github.com/soumith/ganhacks for many tips!Because we're using a CNN from now on, we'll reshape our digits into proper images.<jupyter_code>X_train = X_train.reshape(n, 28, 28, 1)
X_test = X_test.reshape(len(X_test), 28, 28, 1)<jupyter_output><empty_output><jupyter_text>Our generator uses a number of upsampling steps as suggested in the above papers. We use nearest neighbor upsampling rather than fractionally strided convolutions, as discussed in our style transfer notebook.<jupyter_code>CNN_G = Sequential([
    Dense(512*7*7, input_dim=100, activation=LeakyReLU()),
    BatchNormalization(mode=2),
    Reshape((7, 7, 512)),
    UpSampling2D(),
    Convolution2D(64, 3, 3, border_mode='same', activation=LeakyReLU()),
    BatchNormalization(mode=2),
    UpSampling2D(),
    Convolution2D(32, 3, 3, border_mode='same', activation=LeakyReLU()),
    BatchNormalization(mode=2),
    Convolution2D(1, 1, 1, border_mode='same', activation='sigmoid')
])<jupyter_output><empty_output><jupyter_text>The discriminator uses a few downsampling steps through strided convolutions.<jupyter_code>CNN_D = Sequential([
    Convolution2D(256, 5, 5, subsample=(2,2), border_mode='same', 
                  input_shape=(28, 28, 1), activation=LeakyReLU()),
    Convolution2D(512, 5, 5, subsample=(2,2), border_mode='same', activation=LeakyReLU()),
    Flatten(),
    Dense(256, activation=LeakyReLU()),
    Dense(1, activation = 'sigmoid')
])

CNN_D.compile(Adam(1e-3), "binary_crossentropy")<jupyter_output><empty_output><jupyter_text>We train D a "little bit" so it can at least tell a real image from random noise.<jupyter_code>sz = n//200
x1 = np.concatenate([np.random.permutation(X_train)[:sz], CNN_G.predict(noise(sz))])
CNN_D.fit(x1, [0]*sz + [1]*sz, batch_size=128, nb_epoch=1, verbose=2)
CNN_m = Sequential([CNN_G, CNN_D])
CNN_m.compile(Adam(1e-4), "binary_crossentropy")
K.set_value(CNN_D.optimizer.lr, 1e-3)
K.set_value(CNN_m.optimizer.lr, 1e-3)<jupyter_output><empty_output><jupyter_text>Now we can train D & G iteratively.<jupyter_code>dl,gl = train(CNN_D, CNN_G, CNN_m, 2500)
plt.plot(dl[10:])
plt.plot(gl[10:])<jupyter_output><empty_output><jupyter_text>Better than our first effort, but still a lot to be desired:...<jupyter_code>plot_gen(CNN_G)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13088921048400065, 'fraction_numerical': 0.03843061585463901, 'mean_word_length': 5.797814207650274, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 59, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '28215', 'license': 'Apache-2.0', 'n_tokens_mistral': 2464, 'n_tokens_neox': 2251, 'n_words': 559, 'path': 'deeplearning2/nbs/DCGAN.ipynb', 'repo_name': 'kaushikbar/ref-fastai-2016', 'size': '6219'}