<jupyter_start><jupyter_text>Deep Name GeneratorThis project is meant to be a proof-of-concept.  Showing "organic" first names can be generated using a [Generative Advasarial Network](https://en.wikipedia.org/wiki/Generative_adversarial_network). We are using a found dataset provided by [<NAME>](http://hadley.nz/) at RStudio.The goal will be to vectorize each of the names in the following format:| a_0 | b_0 | c_0 | ... | z_9 | a_10 | etc ||-----|-----|-----|-----|-----|------|-----||  1  |  0  |  0  | ... |  1  |  0   |  0  ||  0  |  0  |  1  | ... |  0  |  0   |  0  |Where the letter is the one-hot encoded representation of a character and the number the placeholder in string.For example, the name `Abby` would be represented with the following vector.| a_0 | ... | b_1 | ... | b_2 | ... | y_3 ||-----|-----|-----|-----|-----|-----|-----||  1  | ... |  1  | ... |  1  | ... |  1  |Given Wickham's dataset also includes:* `year`* `percent_[popularity]`* `sex`It may be interesting to add these as additional features to allow the model to learn first name contexts.  GANWorking off the following Keras GAN Example:https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0Another good article on GANS and text generationhttps://becominghuman.ai/generative-adversarial-networks-for-text-generation-part-1-2b886c8cab10And a good one on transformers (Attention is All You Need)https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04https://medium.com/datadriveninvestor/generative-adversarial-network-gan-using-keras-ce1c05cfdfd3https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/https://towardsdatascience.com/my-first-encounter-with-gans-6c0114f60cd7 Preventing Mode Collapsehttps://towardsdatascience.com/10-lessons-i-learned-training-generative-adversarial-networks-gans-for-a-year-c9071159628?https://medium.com/@jonathan_hui/gan-unrolled-gan-how-to-reduce-mode-collapse-af5f2f7b51cd**Good TF GAN build**https://towardsdatascience.com/gan-introduction-and-implementation-part1-implement-a-simple-gan-in-tf-for-mnist-handwritten-de00a759ae5c  Training Parameters<jupyter_code># Engineering parameters.
pad_character       = '~'
allowed_chars       = f'abcdefghijklmnopqrstuvwxyz{pad_character}'
len_allow_chars     = len(allowed_chars)
max_name_length     = 10 

# Parameters
optimizer_name        = 'adam'
g_learning_rate       = 0.001
d_learning_rate       = 0.00005
gan_learning_rate     = 0.001
epochs                = 5000
batch_size            = 16
num_samples           = 32

g_dropout             = 0.4
d_dropout             = 0.0

generator_inputs      = 900
d_width_modifier      = 0.8   # Discriminator deep-neuron multiplier.
g_width_modifier      = 0.01  # Generator deep-neuron multiplier.

# Discriminator accuracy threshold for retraining.
d_accuracy_threshold  = 0.90

params = {
    'epoch': epochs,
    'batch_size': batch_size,
    'g_learning_rate': g_learning_rate,
    'd_learning_rate': d_learning_rate,
    'gan_learning_rate': gan_learning_rate,
    'optimizer_name': optimizer_name,
    'generator_inputs': generator_inputs,
    'num_samples_per_step': num_samples,
    'allowed_chars': allowed_chars,
    'max_name_length': max_name_length,
    'g_dropout': g_dropout,
    'd_dropout': d_dropout,
    'd_width_modifier': d_width_modifier,
    'g_width_modifier': g_width_modifier,
    'd_accuracy_threshold': d_accuracy_threshold
}<jupyter_output><empty_output><jupyter_text>Prepared DataIf you'd like to skip to the fun part, I've vectorized the names already.But, if you want to grind it out, here's the code:* [deep_name_prep_data](https://github.com/Ladvien/gan_name_maker/blob/master/deep_name_prep_data.ipynb)  Load the Data<jupyter_code>import pandas as pd
import numpy as np
!git clone https://github.com/Ladvien/gan_name_maker
df = pd.read_csv('./gan_name_maker/vectorized_names.csv')
cols = list(df)

# Move the name column to the beginning.
cols.insert(0, cols.pop(cols.index('name')))
df = df.loc[:, cols]

# Drop the yucky columns.
df.drop('Unnamed: 0', axis = 1, inplace = True)

# Sort values by name
df.sort_values(by = 'name', ascending = True, inplace = True)
print(f'Vectorized data has {df.shape[0]} samples and {df.shape[1]} features.')
df.head()
# Randomize
df = df.sample(df.shape[0])<jupyter_output><empty_output><jupyter_text>Libraries<jupyter_code>import tensorflow as tf

from tensorflow.keras.layers import Dense, Dropout, Activation, Input, LeakyReLU,\
                                     BatchNormalization, ReLU
from tensorflow.keras import Sequential
from tensorflow.keras.models import Model

from tensorflow.keras.callbacks import History 

# Personal tools.
!pip install git+https://github.com/Ladvien/ladvien_ml.git
from ladvien_ml import FeatureModel

fm = FeatureModel()<jupyter_output><empty_output><jupyter_text>Setup Weights and Biases<jupyter_code>!pip install --upgrade wandb
!wandb login 186e8a3df54055bf2ce699bf0e3f5320c9bb29e6
import wandb
wandb.init(project = 'deep_name_generator',
           config = params)<jupyter_output><empty_output><jupyter_text>Discriminator<jupyter_code>def discriminator(input_shape, optimizer, dropout = 0.1, width_modifier = 0.5):
  
  D = Sequential()
  
  # Input layer
  input_layer_width = input_shape
  D.add(Dense(input_layer_width, input_shape = (input_layer_width,)))
  D.add(ReLU())
  # D.add(BatchNormalization())
  D.add(Dropout(dropout))
  
  # First Hidden Layer
  first_layer_width = input_shape * width_modifier
  D.add(Dense(first_layer_width))
  D.add(ReLU())
  # D.add(BatchNormalization()) 
  D.add(Dropout(dropout))

  # Second Hidden Layer
  second_layer_width = input_shape * width_modifier
  D.add(Dense(second_layer_width))
  D.add(ReLU())
  # D.add(BatchNormalization())
  D.add(Dropout(dropout))

  # Third Hidden Layer
  third_layer_width = input_shape * width_modifier
  D.add(Dense(third_layer_width))
  D.add(ReLU())
  # D.add(BatchNormalization())
  D.add(Dropout(dropout))
 
  # Output
  D.add(Dense(1, activation = 'sigmoid'))
  D._name = 'discriminator'
  D.compile(optimizer = optimizer, loss = 'binary_crossentropy', metrics = ['accuracy'])
  D.summary()

  return D<jupyter_output><empty_output><jupyter_text>Generator<jupyter_code>def generator(num_inputs, output_shape, optimizer, dropout = 0.1, width_modifier = 0.5):

  G = Sequential()

  # Input layer
  input_layer_width = num_inputs
  G.add(Dense(input_layer_width, input_shape = (input_layer_width,)))
  G.add(ReLU())
  # G.add(BatchNormalization())
  G.add(Dropout(dropout))
  
  # First Hidden Layer
  first_layer_width = num_inputs * width_modifier
  G.add(Dense(first_layer_width))
  G.add(ReLU())
  # G.add(BatchNormalization())
  G.add(Dropout(dropout))

  # Second Hidden Layer
  second_layer_width = num_inputs * width_modifier
  G.add(Dense(second_layer_width))
  G.add(ReLU())
  # G.add(BatchNormalization())
  G.add(Dropout(dropout))

  # Third Hidden Layer
  third_layer_width = num_inputs * width_modifier
  G.add(Dense(third_layer_width))
  G.add(ReLU())
  # G.add(BatchNormalization())
  G.add(Dropout(dropout))
  
  # Output layer 
  G.add(Dense(output_shape, activation = 'sigmoid'))
  G._name = 'generator'
  G.compile(optimizer = optimizer, loss = 'categorical_crossentropy')
  G.summary()
  return G<jupyter_output><empty_output><jupyter_text>GAN<jupyter_code>def create_gan(D, G, g_inputs):
    D.trainable = False
    gan_input = Input(shape = (g_inputs,))
    x = G(gan_input)
    gan_output = D(x)
    gan = Model(inputs = gan_input, outputs = gan_output)
    gan.compile(loss = 'binary_crossentropy', optimizer = 'rmsprop')
    return gan<jupyter_output><empty_output><jupyter_text>Compile<jupyter_code># Input shape will be the number of possible characters times 
# the maximum name length allowed.
vectorized_name_length = len_allow_chars * max_name_length

# Select optimizer.
g_optimizer = fm.select_optimizer(optimizer_name, g_learning_rate)
d_optimizer = fm.select_optimizer(optimizer_name, d_learning_rate)
gan_optimizer = fm.select_optimizer(optimizer_name, gan_learning_rate)

# Generator
G = generator(generator_inputs, vectorized_name_length, g_optimizer, dropout = g_dropout, width_modifier = g_width_modifier)

# Discriminator
D = discriminator(vectorized_name_length, d_optimizer, dropout = d_dropout, width_modifier = d_width_modifier)

# Build GAN
GAN = create_gan(D, G, generator_inputs)

GAN._name = 'GAN'
GAN.compile(loss='binary_crossentropy', optimizer = gan_optimizer, metrics=['accuracy'])
GAN.summary()<jupyter_output><empty_output><jupyter_text>Prepare Data<jupyter_code># Randomize inputs.
df = df.sample(df.shape[0])

# Create target label.
df['real'] = 1

# Drop the 'name' and 'real' columns.
X = df.iloc[:,1:-1]

# Get target
y = df.iloc[:,-1:]

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)<jupyter_output><empty_output><jupyter_text>Evaluation Method<jupyter_code>def retrieve_names_from_sparse_matrix(generated_names, pad_character):
  retrieved_names = []
  for name_index in range(len(generated_names)):
    generated_name = ''
    name_array = generated_names[name_index]
    for char_index in range(max_name_length):
      
      # Get A index.
      first_letter_index = (char_index * len_allow_chars)
      last_letter_index = (char_index * len_allow_chars + len_allow_chars)
      char_vector = list(name_array[first_letter_index:last_letter_index])
      
      char = allowed_chars[char_vector.index(max(char_vector))]

      if char == pad_character:
        break
        
      generated_name += char

    # print(generated_name) 
    retrieved_names.append(generated_name)

  return retrieved_names<jupyter_output><empty_output><jupyter_text>Training<jupyter_code># Loading the data
batch_count = x_train.shape[0] / batch_size
d_accuracy   = 0

for e in range(1, epochs + 1):
    print(f'Epoch: {e}')
    for step in range(batch_size):
        # Generate noise as input to initialize generator.
        noise = np.random.normal(0, 1, [batch_size, generator_inputs])
        
        # Generate fake names from noise.
        generated_names = G.predict(noise)
        
        # Get a random set of real names.
        real_names = x_train.iloc[np.random.randint(low = 0, high = x_train.shape[0], size = batch_size),:]

        #Construct different batches of  real and fake data 
        X = np.concatenate([real_names, generated_names])
        
        # Labels for generated and real data (first four rows are real)
        y_labels = np.zeros(2 * batch_size)
        y_labels[:batch_size] = 1
        
        if d_accuracy < d_accuracy_threshold:
          # Pre-train discriminator on fake and real data before starting the GAN. 
          D.trainable = True
          D.train_on_batch(X, y_labels)
          print(f'D_accuracy dropped too low: {d_accuracy}')

        # During the training of GAN, the weights of discriminator should be 
        # fixed. We can enforce that by setting the trainable flag.
        D.trainable = False

        # Tricking the noised input of the Generator as real data
        noise = np.random.normal(0, 1, [batch_size, generator_inputs])
        y_gen = np.ones(batch_size)
        
        # Train the GAN by alternating the training of the Discriminator 
        # and training the chained GAN model with Discriminator’s weights 
        # frozen.
        GAN_score = GAN.train_on_batch(noise, y_gen)
        D_score = D.evaluate(X, y_labels, verbose = 0)
        d_accuracy = D_score[1]


    # End of an epoch.
    print(f'GAN loss: {GAN_score[0]}')

    print(f'Disc. loss: {D_score[0]}')
    
    # Make Generator inputs.
    noise = np.random.normal(0, 1, [num_samples, generator_inputs])

    # Generate fake names from noise.
    generated_names = G.predict(noise)
    retrieved_names = retrieve_names_from_sparse_matrix(generated_names, pad_character)

    # Save generated names.
    table = wandb.Table(columns=['Name', 'Epoch'])

    for name in retrieved_names:
      table.add_data(name, e)
    
    # Log sample of generated names.
    wandb.log({"generated_names": table})

    # Log to Weights and Biases
    wandb.log({'GAN Loss': GAN_score[0], 
               'epoch': e, 
               'discriminator_loss': D_score[0],
               'discriminator_accuracy': D_score[1],
               'num_of_names': len(list(set(retrieved_names)))
    })
    
          
# Make Generator inputs.
noise = np.random.normal(0, 1, [20, generator_inputs])

# Generate fake names from noise.
generated_names = G.predict(noise)
retrieve_names_from_sparse_matrix(generated_names, pad_character)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11398271798900236, 'fraction_numerical': 0.015318146111547526, 'mean_word_length': 3.9135468930914703, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '276973', 'license': 'MIT', 'n_tokens_mistral': 4550, 'n_tokens_neox': 4321, 'n_words': 911, 'path': 'deep_name_generator.ipynb', 'repo_name': 'Ladvien/gan_name_maker', 'size': '12738'}