<jupyter_start><jupyter_text>Generating a Shrek movie script using a simple RNN  The scripts for [Shrek](https://www.imsdb.com/scripts/Shrek.html) and [Shrek The Third](https://www.imsdb.com/scripts/Shrek-the-Third.html) were downloaded and saved into a single text file.Using a simple RNN trained to predict the following character given an input sequence of characters from this file, we are able to generate our own Shrek 'scripts'.![alt text](https://s.wsj.net/public/resources/images/OB-IO273_shrekf_E_20100520084037.jpg)<jupyter_code>try:
  %tensorflow_version 2.x
except Exception:
  pass
import tensorflow as tf
import numpy as np
import os
import time
import re<jupyter_output>TensorFlow 2.x selected.<jupyter_text>Open the text file containing the two Shrek movie scripts.<jupyter_code>text = open('/content/shrek_script.txt', 'rb').read().decode(encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Preview the first 1000 characters of the text file, notice all the excess white space due to the format of the scripts.<jupyter_code>text[:100]<jupyter_output><empty_output><jupyter_text>We replace repeated white spaces with a single one using a regular expression.<jupyter_code>clean_text = re.sub(' +', ' ', text)
clean_text[:100]<jupyter_output><empty_output><jupyter_text>The vocabulary is simply a list of all unique characters present in the scripts.<jupyter_code>vocab = sorted(set(clean_text))
vocab[:5] # Print first 5 unique characters<jupyter_output><empty_output><jupyter_text>Since our model requires numerical representations of the data, we need to map each unique character to a number and be able to reverse said mapping.<jupyter_code>char2idx = {c:i for i, c in enumerate(vocab)} # Maps characters to a number
idx2char = np.array(vocab) # Maps numbers to a character<jupyter_output><empty_output><jupyter_text>Let us look at the first 10 character mappings<jupyter_code>for char in range(10):
  print('{:4s}: {:2d}'.format(repr(idx2char[char]), char2idx[idx2char[char]]))<jupyter_output>'\n':  0
' ' :  1
'!' :  2
'"' :  3
'#' :  4
'&' :  5
"'" :  6
'(' :  7
')' :  8
',' :  9<jupyter_text>We may now numericalise the text using `char2idx`.<jupyter_code>numerical_text = np.array([char2idx[c] for c in clean_text])<jupyter_output><empty_output><jupyter_text>The first 20 characters of the script and their numerical representation.<jupyter_code>print ('{} \n\n maps to \n\n {}'.format(repr(clean_text[:20]), numerical_text[:20]))<jupyter_output>'SHREK\n\n Written by\n\n' 

 maps to 

 [44 33 43 30 36  0  0  1 48 70 61 72 72 57 66  1 54 77  0  0]<jupyter_text>Given a sequence of `N` characters from the script, the aim of the model is to predict the character at `N+1`. The length of the character stream can be changed, and is controlled by the `seq_length` variable. We use the `tf.data.Dataset.from_tensor_slices` function to convert our numericalised text into a stream of character indices. We convert this into sequences of length `seq_length + 1` (given seq_length input, we wish to predict the char at `seq_length + 1`) using the batch method.<jupyter_code>seq_length = 100

char_dataset = tf.data.Dataset.from_tensor_slices(numerical_text) # Forms a tensor from each char in numerical_text

sequences = char_dataset.batch(seq_length+1, drop_remainder=True) # Converts character stream into input/target sequence where the last char is the target<jupyter_output><empty_output><jupyter_text>We now define a function to assign the first `seq_length` characters as the input and then shift over the characters by one place to form the target. We then map all sequences using this function to form our input/target pairs.<jupyter_code>def gen_splits(chunk):
    input_text = chunk[:-1] # First seq_length chars are input
    target_text = chunk[1:] # Final char is target
    return input_text, target_text

dataset = sequences.map(gen_splits)<jupyter_output><empty_output><jupyter_text>Example input and target pair. We can see the aim of the model is to predict the following character for a given stream of characters.<jupyter_code>for i, o in dataset.take(1):
  print('Input stream: ', repr(''.join(idx2char[i.numpy()])))
  print('Target stream: ', repr(''.join(idx2char[o.numpy()])))
BATCH_SIZE = 256
BUFFER_SIZE = 10000 # TF designed to deal with potentially infinite data, this is the buffer size on which it shuffles<jupyter_output><empty_output><jupyter_text>We now shuffle the data and create trainining batches.<jupyter_code>dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)<jupyter_output><empty_output><jupyter_text>We now define a function to build our model using the `tf.keras.Sequential` API.We begin with an embedding layer to map each character to an embedding space with `embedding_dim` dimensions. The recurrent layers follow, here we have a choice of [LSTMs](https://www.bioinf.jku.at/publications/older/2604.pdf) or [GRUs](https://arxiv.org/pdf/1412.3555.pdf), specified with the `cell_type` argument. We may add as many of these layers as we wish using the `rec_layers` argument. The number of units in each recurrent layer is controlled with the `rnn_units` argument.Two fully connected layers with dropout between form the head of the model, the dropout rate is specified using the `drop_rate` argument.<jupyter_code>def build_model(vocab_size, embedding_dim = 256, cell_type = 'lstm', rec_layers = 1, rnn_units = 1024, drop_rate = 0.2, batch_size = 256):

  cell_types = ['lstm', 'gru']

  model = tf.keras.Sequential()

  model.add(tf.keras.layers.Embedding(vocab_size, embedding_dim,
                              batch_input_shape=[batch_size, None]))

  if cell_type not in cell_types:
    raise ValueError("Invalid cell type. Expected one of: {}".format(repr(cell_types)))

  if cell_type == 'lstm':
    for _ in range(rec_layers):
      model.add(tf.keras.layers.LSTM(rnn_units, return_sequences=True, stateful=True))

  else:
    for _ in range(rec_layers):
      model.add(tf.keras.layers.GRU(rnn_units, return_sequences=True, stateful=True))

  model.add(tf.keras.layers.Dense(rnn_units))
  model.add(tf.keras.layers.Dropout(drop_rate))
  model.add(tf.keras.layers.Dense(vocab_size))

  return model<jupyter_output><empty_output><jupyter_text>We now build our model. Based on the results of this [paper](https://www.researchgate.net/publication/335158858_LSTM_vs_GRU_vs_Bidirectional_RNN_for_script_generation), we will use LSTM cells, as they found them to be better for script generation compared to GRUs.<jupyter_code>vocab_size = len(vocab)
rec_layers = 1

model = build_model(vocab_size, cell_type = 'lstm', rec_layers = rec_layers, batch_size = BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Since our model aims to predict a character, this is simply a standard classification problem, and we make use of the `tf.keras.losses.sparse_categorical_crossentropy` loss function.<jupyter_code>def loss(labels, logits):
  return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)<jupyter_output><empty_output><jupyter_text>We now compile our model using this loss and set Adam as the optimizer.<jupyter_code>model.compile(optimizer='Adam', loss=loss)<jupyter_output><empty_output><jupyter_text>The following callback allows checkpoints to be saved during training (We will use the final checkpoint to restore our model prior to generating a script).<jupyter_code>ckpt_dir = './training_checkpoints'
ckpt = os.path.join(ckpt_dir, "ckpt_{epoch}")

checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
    filepath=ckpt,
    save_weights_only=True,
    save_freq=20000) # Save model checkpoint only after 20000 samples have been seen - to reduce disk consumption on colab<jupyter_output><empty_output><jupyter_text>We are now ready to train our model.<jupyter_code>EPOCHS=750
model_history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback], verbose=1)<jupyter_output>Train for 7 steps
Epoch 1/750
7/7 [==============================] - 7s 967ms/step - loss: 4.2691
Epoch 2/750
7/7 [==============================] - 5s 659ms/step - loss: 3.9543
Epoch 3/750
7/7 [==============================] - 5s 657ms/step - loss: 3.5411
Epoch 4/750
7/7 [==============================] - 5s 662ms/step - loss: 3.3051
Epoch 5/750
7/7 [==============================] - 5s 659ms/step - loss: 3.2106
Epoch 6/750
7/7 [==============================] - 5s 655ms/step - loss: 3.1099
Epoch 7/750
7/7 [==============================] - 5s 666ms/step - loss: 2.9977
Epoch 8/750
7/7 [==============================] - 5s 657ms/step - loss: 2.8912
Epoch 9/750
7/7 [==============================] - 5s 663ms/step - loss: 2.7935
Epoch 10/750
7/7 [==============================] - 5s 656ms/step - loss: 2.7051
Epoch 11/750
7/7 [==============================] - 5s 663ms/step - loss: 2.6303
Epoch 12/750
7/7 [==============================] - 5s 686ms/step - loss: 2.5531
Epoch 13/750
7/7 [=[...]<jupyter_text>We need to rebuild our model, and load the weights saved from the checkpoint to allow our model to generate text - this is due to a change in batch size, which cannot be performed once a model has been built.<jupyter_code>writer = build_model(vocab_size, cell_type = 'lstm', rec_layers = rec_layers, batch_size = 1)

writer.load_weights(tf.train.latest_checkpoint(ckpt_dir))

writer.build(tf.TensorShape([1, None]))<jupyter_output><empty_output><jupyter_text>Since we wish to generate a Shrek script, we find the average length of the script, this is how many characters we will ask the model to generate.<jupyter_code>script_length = len(clean_text)//2<jupyter_output><empty_output><jupyter_text>We now define a function `generate_text` to generate our script.Initially, we pass a starting string to the model from which it must then predict the following character. The prediction distribution is captured and from this the index of the predicted character is calculated using a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution). This predicted character, and the previous state returned by the model, is then passed as the next input. This process is repeated until `gen_to` characters have been generated.<jupyter_code>def generate_text(model, start_string, gen_to):

  # Numericalise starting string
  input_eval = [char2idx[c] for c in start_string]
  input_eval = tf.expand_dims(input_eval, 0)

  text_generated = []

  # controls the degree of 'randomness' of the character generation
  temperature = 1.0

  model.reset_states()
  for i in range(gen_to):
      # predict next char
      predictions = model(input_eval)
      # remove the batch dimension
      predictions = tf.squeeze(predictions, 0)

      # categorical distribution to predict the character
      predictions = predictions / temperature
      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

      # Pass predicted char and previous hidden state
      input_eval = tf.expand_dims([predicted_id], 0)

      text_generated.append(idx2char[predicted_id])

  return (start_string + ''.join(text_generated))<jupyter_output><empty_output><jupyter_text>We are now ready to ask our model to generate a thrilling Shrek script for us to (hopefully) enjoy.<jupyter_code>print(generate_text(writer, start_string=u"THE SHREKINING\n\n Written by\n\n A Basic RNN\n\n\n\n\n", gen_to = script_length))<jupyter_output>THE SHREKINING

 Written by

 A Basic RNN




 They hold tou of
 home on 
 lf.)
 
 DONKEY
 What's your lork away.
 
 SHREK
 What?

 DONKEY
 Where do, uh, I sleep?

 SHREK
 (irritated) Outside!

 DONKEY
 Oh, well, I I will have - - (He CREATURES
 Avil does Donkey away. He falls, knocking over a
 guard holding an axe on his way down. The guard drops the
 logerbread Man is attending school.
 
 TEACHARES
 Sell the lavy bouthis plan.
 
 SHREK
 Donkey, we're dealing with
 amateurs.
 
 The guards are confused. They all ounder the 3 mice) What are 
 you doing in my house? (He gets bumped 
 from behind and he drops the mice.) 
 Hey, the boat and if the door buss him.
 
 ST I think that went perche Third - Final Screening Script 69.
 
 
 
 SHREK
 Thanks Artie.
 
 ARTIE
 The soap's because you start the Third - Final Screening Script 17.
 
 
 
 Shrek leans in closer after each "is," waiting in
 anyou.
 
 There is a montage of scement.
 
 The pirates aim the cannon at Puss, Donkey and Artie. Artie[...]
{'fraction_non_alphanumeric': 0.12029043969342477, 'fraction_numerical': 0.030092779346510688, 'mean_word_length': 5.058651026392962, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '452487', 'license': 'MIT', 'n_tokens_mistral': 4263, 'n_tokens_neox': 3892, 'n_words': 1288, 'path': 'Neural Networks/Shrekining.ipynb', 'repo_name': 'Dev-data-fun/TensorFlow-notebooks', 'size': '12395'}