<jupyter_start><jupyter_text>*Data Science Unit 4 Sprint 3 Assignment 1* Recurrent Neural Networks and Long Short Term Memory (LSTM)![Monkey at a typewriter](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Chimpanzee_seated_at_typewriter.jpg/603px-Chimpanzee_seated_at_typewriter.jpg)It is said that [infinite monkeys typing for an infinite amount of time](https://en.wikipedia.org/wiki/Infinite_monkey_theorem) will eventually type, among other things, the complete works of Wiliam Shakespeare. Let's see if we can get there a bit faster, with the power of Recurrent Neural Networks and LSTM.This text file contains the complete works of Shakespeare: https://www.gutenberg.org/files/100/100-0.txtUse it as training data for an RNN - you can keep it simple and train character level, and that is suggested as an initial approach.Then, use that trained RNN to generate Shakespearean-ish text. Your goal - a function that can take, as an argument, the size of text (e.g. number of characters or lines) to generate, and returns generated text of that size.Note - Shakespeare wrote an awful lot. It's OK, especially initially, to sample/use smaller data and parameters, so you can have a tighter feedback loop when you're trying to get things running. Then, once you've got a proof of concept - start pushing it more!<jupyter_code># TODO - Words, words, mere words, no matter from the heart.
from tensorflow.keras.callbacks import LambdaCallback, EarlyStopping
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM
from tensorflow.keras.optimizers import RMSprop

import numpy as np
import random
import sys
import os

import pandas as pd
test = pd.read_fwf("https://www.gutenberg.org/files/100/100-0.txt")
test.shape
# Data cleaning

# Read in .txt url
data = pd.read_fwf("https://www.gutenberg.org/files/100/100-0.txt")

# Reset index because the actual text was the index
data = data.reset_index()

# Drop redundant column
data = data.drop(columns='Unnamed: 0')

# Filter out any NaN values
data = data[data['index'].notna()]

# Convert the df series into a python list
data = data['index'].tolist()
len(data)
data = data[0:int((len(data)/10))]
# Example text
data[0]
# transforms the list into a string
# the " " puts a space inbetween each word
text = " ".join(data)
# Looks through the entire text and grabs all the unique chars (list checks for uniques).
# Set is wrapped in a list so that it puts the unique characters in a list
chars = list(set(text))
# Creates a dict that gives each char a unique value
char_int = {c:i for i, c in enumerate(chars)}
# creates a dict that gives a number a unique char
int_char = {i:c for i, c in enumerate(chars)}
# Creates a list that looks through the char_int dict
# and matches the char from text to a number from the dict
encoded = [char_int[c] for c in text]
maxlen = 40
step = 5

sequences = []
next_char = []

# For loop that appends a list of 40 encoded numbers to the sequences list
# next_char gets the next encoded number of the next encoded numbers list in sequences
for i in range(0, len(encoded) - maxlen, step):
    sequences.append(encoded[i : i + maxlen])
    next_char.append(encoded[i + maxlen])
    
print('sequences', len(sequences))
# creates the x and y variables for the model

# Creates an array of bools (False's) that has 1059622 (len(sequences)) arrays that holds 40 arrays
# (maxlen) that each of those arrays holds 105 values (len(chars))
x = np.zeros((len(sequences), maxlen, len(chars)), dtype=np.bool)

# Creates an array of bools (False's) that has 1059622 arrays (len(sequences))
# with 105 values in each of them (len(chars))
y = np.zeros((len(sequences), len(chars)), dtype=np.bool)

# Everything is false since they are placeholders for now
# for loop that does a lot of stuff

# Grabs a single sequence
for i, sequence in enumerate(sequences):
    # grabs a single value in the list
    for t, char in enumerate(sequence):
        # assigns a 1 (True) to 
        x[i,t,char] = 1
        
    y[i, next_char[i]] = 1
(1059622, 40, 105)

x[0][0][0]
model = Sequential()
model.add(LSTM(128, input_shape=(maxlen, len(chars))))
model.add(Dense(len(chars), activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam')
def sample(preds):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / 1
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)
def on_epoch_end(epoch, _):
    
    print()
    print(f'------ Generating text after Epoch: {epoch}')
    
    start_index = random.randint(0, len(text) - maxlen - 1)
    
    generated = ''
    
    sentence = text[start_index: start_index + maxlen]
    
    generated += sentence
    
    print(f'------ Generating with seed: "" {sentence} "" ')
    sys.stdout.write(generated)
    
    for i in range(400):
        x_pred = np.zeros((1, maxlen, len(chars)))
        for t, char in enumerate(sentence):
            x_pred[0, t, char_int[char]] = 1
            
        preds = model.predict(x_pred, verbose=0)[0]
        next_index = sample(preds)
        next_char = int_char[next_index]
        
        sentence=sentence[1:] + next_char
        
        sys.stdout.write(next_char)
        sys.stdout.flush()
    
    print()
    
print_callback = LambdaCallback(on_epoch_end=on_epoch_end)    
model.fit(x, y,
         batch_size=256,
         epochs=10,
         callbacks=[print_callback])<jupyter_output>Train on 109754 samples
Epoch 1/10
109568/109754 [============================>.] - ETA: 0s - loss: 2.5932
------ Generating text after Epoch: 0
------ Generating with seed: "" w that a life was but a flower, In the s "" 
w that a life was but a flower, In the siclit in  Irhiserele gerdind mad an lith; Wham nuenlil ne liv oud  tewear mouch theederp? . cSEagr apke Rhainr and freit lereant non' yud fopther thils uettir'I f our to inh dols uy onniI pord iens! th thy Be win you hacpy Thel sald bo las hatk doke nor ste tithm amy Ther gpowlsechitna heu the beren virk fin. shas' she sayed Dr, thive Endee, lishe nkain, ond wereasit tDe. hame thate bu Ond soin pa
109754/109754 [==============================] - 41s 370us/sample - loss: 2.5928
Epoch 2/10
109568/109754 [============================>.] - ETA: 0s - loss: 2.3653
------ Generating text after Epoch: 1
------ Generating with seed: ""  That sings with piercing; do not touch  "" 
 That sings with piercing; do not touch nterekerr momyt fe[...]
{'fraction_non_alphanumeric': 0.11196969696969697, 'fraction_numerical': 0.02666666666666667, 'mean_word_length': 3.731899641577061, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '408368', 'license': 'MIT', 'n_tokens_mistral': 2226, 'n_tokens_neox': 2059, 'n_words': 790, 'path': 'module1-rnn-and-lstm/LS_DS_431_RNN_and_LSTM_Assignment.ipynb', 'repo_name': 'aklefebvere/DS-Unit-4-Sprint-3-Deep-Learning', 'size': '6600'}