<jupyter_start><jupyter_text>*Data Science Unit 4 Sprint 3 Assignment 1* Recurrent Neural Networks and Long Short Term Memory (LSTM)![Monkey at a typewriter](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Chimpanzee_seated_at_typewriter.jpg/603px-Chimpanzee_seated_at_typewriter.jpg)It is said that [infinite monkeys typing for an infinite amount of time](https://en.wikipedia.org/wiki/Infinite_monkey_theorem) will eventually type, among other things, the complete works of Wiliam Shakespeare. Let's see if we can get there a bit faster, with the power of Recurrent Neural Networks and LSTM.This text file contains the complete works of Shakespeare: https://www.gutenberg.org/files/100/100-0.txtUse it as training data for an RNN - you can keep it simple and train character level, and that is suggested as an initial approach.Then, use that trained RNN to generate Shakespearean-ish text. Your goal - a function that can take, as an argument, the size of text (e.g. number of characters or lines) to generate, and returns generated text of that size.Note - Shakespeare wrote an awful lot. It's OK, especially initially, to sample/use smaller data and parameters, so you can have a tighter feedback loop when you're trying to get things running. Then, once you've got a proof of concept - start pushing it more!<jupyter_code>import tensorflow as tf

import numpy as np
import os
import time
import re
path_to_shakespeare = tf.keras.utils.get_file('shakespeare.txt', 'https://www.gutenberg.org/files/100/100-0.txt')
text = open(path_to_shakespeare, 'rb').read().decode(encoding='utf-8')
# count number of chars
len(text)
def clean_shakespeare(text):
    # get contents
    contents = text[974:2893].replace("\r", "").replace("\n", "")
    contents = re.sub("\s{2,}", ",,", contents)
    contents = contents.split(",,")
    
    for idx,book in enumerate(contents):
        if book == "THE LIFE OF KING HENRY THE FIFTH":
            contents[idx] = "THE LIFE OF KING HENRY V"
        elif book == "THE TRAGEDY OF MACBETH":
            contents[idx] = "MACBETH"
        elif book == "THE TRAGEDY OF OTHELLO, MOOR OF VENICE":
            contents[idx] = "OTHELLO, THE MOOR OF VENICE"
        elif book == "TWELFTH NIGHT; OR, WHAT YOU WILL":
            contents[idx] = "TWELFTH NIGHT: OR, WHAT YOU WILL"
    
    # remove the project gutenberg info
    text = text[2893:-21529]
    
    shakespeare_dict = {}
    
    for idx,book in enumerate(contents):
        strpos = text.find(book) + len(book)
        
        if idx + 1 != len(contents):
            next_book = contents[idx + 1]
            endpos = text.find(next_book, strpos)
            shakespeare_dict[book] = text[strpos:endpos]
        else:
            shakespeare_dict[book] = text[strpos:]
    
    return shakespeare_dict, text
books, text = clean_shakespeare(text)
vocab = sorted(set(text))
len(vocab)
char2idx = {u:i for i, u in enumerate(vocab)}
idx2char = np.array(vocab)

text_as_int = np.array([char2idx[c] for c in text])
seq_length = 100
examples_per_epoch = len(text)//(seq_length+1)
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
sequences = char_dataset.batch(seq_length+1, drop_remainder=True)
def split_input_target(chunk):
    input_text = chunk[:-1]
    target_text = chunk[1:]
    return input_text, target_text

dataset = sequences.map(split_input_target)
for input_example, target_example in  dataset.take(1):
    print ('Input data: ', repr(''.join(idx2char[input_example.numpy()])))
    print ('Target data:', repr(''.join(idx2char[target_example.numpy()])))
for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):
    print("Step {:4d}".format(i))
    print("  input: {} ({:s})".format(input_idx, repr(idx2char[input_idx])))
    print("  expected output: {} ({:s})".format(target_idx, repr(idx2char[target_idx])))
BATCH_SIZE = 64

BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
vocab_size = len(vocab)

embedding_dim = 256

rnn_units = 1024
def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
    model = tf.keras.Sequential([
        tf.keras.layers.Embedding(vocab_size, embedding_dim,
                                  batch_input_shape=[batch_size, None]),
        tf.keras.layers.GRU(rnn_units,
                            return_sequences=True,
                            stateful=True,
                            recurrent_initializer='glorot_uniform'),
        tf.keras.layers.Dense(vocab_size)
    ])
    return model
model = build_model(
    vocab_size = len(vocab),
    embedding_dim=embedding_dim,
    rnn_units=rnn_units,
    batch_size=BATCH_SIZE)
def loss(labels, logits):
    return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
model.compile(optimizer='adam', loss=loss)
checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_prefix,
    save_weights_only=True)

EPOCHS=30
history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])
model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))

model.build(tf.TensorShape([1, None]))
model.summary()
def generate_text(model, start_string):
    # Evaluation step (generating text using the learned model)

    # Number of characters to generate
    num_generate = 1000

    # Converting our start string to numbers (vectorizing)
    input_eval = [char2idx[s] for s in start_string]
    input_eval = tf.expand_dims(input_eval, 0)

    # Empty string to store our results
    text_generated = []

    # Low temperatures results in more predictable text.
    # Higher temperatures results in more surprising text.
    # Experiment to find the best setting.
    temperature = 1.0

    # Here batch size == 1
    model.reset_states()
    for i in range(num_generate):
        predictions = model(input_eval)
        # remove the batch dimension
        predictions = tf.squeeze(predictions, 0)

        # using a categorical distribution to predict the character returned by the model
        predictions = predictions / temperature
        predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

        # We pass the predicted character as the next input to the model
        # along with the previous hidden state
        input_eval = tf.expand_dims([predicted_id], 0)

        text_generated.append(idx2char[predicted_id])

    return (start_string + ''.join(text_generated))
print(generate_text(model, start_string=u"A rose "))<jupyter_output>A rose fire;
  But for the hair of mothers, in the burs,
    Others to any day 'tis well. Grant things can water,
    I desire you attate yourself to gaze,
  And yet I had a broad, and pity life,
  And to be taken foolery that I met have beat, sirreiver, have I poison’d;
Reignith in how sbear thy way; and from—
Cold of my service, the youth does of the sea
That makes us not set in my best worth:
Her age with her by papient Boy, in all my fool,
I hie the salder by my death, I naked,
Look do’t barnes. Bo true, my lord you must.

Possessel out of a demuner rect,
Nor couplete as Anne, but we are entereat’st
than young in comfort, my lord!

VIOLA.
I say.

RODERIGO.
There, lady, since my true-piontine,
    As clap away. Let's not ask solicit,
And the most unfatificuate Romans of honourablic you,
Yet long keeps in a bark up witchcreful) else
That I am sure
That measur’d strangeness labours should be foulor,
  Procostant many Fantastic have founded.
My sentence, Catca, and
{'fraction_non_alphanumeric': 0.09790664412950202, 'fraction_numerical': 0.012092055649460408, 'mean_word_length': 3.768753874767514, 'pattern_counts': {'":': 4, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '64749', 'license': 'MIT', 'n_tokens_mistral': 2519, 'n_tokens_neox': 2418, 'n_words': 751, 'path': 'module1-rnn-and-lstm/LS_DS_431_RNN_and_LSTM_Assignment.ipynb', 'repo_name': 'spentaur/DS-Unit-4-Sprint-3-Deep-Learning', 'size': '7691'}