<jupyter_start><jupyter_text>Develop a Small LSTM Recurrent Neural Network Let’s start off by importing the classes and functions we intend to use to train our model.<jupyter_code>import os
#os.environ["KERAS_BACKEND"] = "theano"
os.environ["KERAS_BACKEND"] = "tensorflow"

import sys
import time
import re
import numpy
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras.callbacks import ModelCheckpoint
from keras.utils import np_utils
import tensorflow as tf
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)<jupyter_output><empty_output><jupyter_text>First we'll develop a small ANN with Keras:<jupyter_code># load pima indians dataset
dataset = numpy.loadtxt("../datasets/CSVs/pima-indians-diabetes.data", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation="relu", kernel_initializer="uniform"))
model.add(Dense(8, activation="relu", kernel_initializer="uniform"))
model.add(Dense(1, activation="sigmoid", kernel_initializer="uniform"))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
history = model.fit(X, Y, validation_split=0.33, epochs=200, batch_size=10, verbose=1)
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.figure(figsize=(12,8))
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.figure(figsize=(12,8))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
# calculate predictions
predictions = model.predict(X)
print(predictions[0:10])

print()

# round predictions
rounded = [round(x[0]) for x in predictions]
print(rounded[0:10])<jupyter_output>[[0.6530575 ]
 [0.10395092]
 [0.8922678 ]
 [0.07589243]
 [0.8696196 ]
 [0.27980155]
 [0.20595627]
 [0.7305127 ]
 [0.90565926]
 [0.741504  ]]

[1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0]<jupyter_text>Develop a Small LSTM Recurrent Neural NetworkBased on this [post](https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/)In this section we will develop a simple LSTM network to learn sequences of characters from Alice in Wonderland. In the next section we will use this model to generate new sequences of characters. Next, we need to load the ASCII text for the book into memory and convert all of the characters to lowercase to reduce the vocabulary that the network must learn.<jupyter_code># load ascii text and covert to lowercase
filename = "../datasets/Texts/Gutenberg/wonderland.txt"
raw_text = open(filename).read()
raw_text = raw_text.lower()
raw_text.find('happy summer days.\n\n')
raw_text = raw_text[:145000]<jupyter_output><empty_output><jupyter_text>Cleaning the text:<jupyter_code>raw_text = raw_text[raw_text.find('\n\n\n\n'):]
raw_text = re.sub(r'[\n ]+',' ', raw_text)

import string
for char in string.punctuation:
    raw_text = raw_text.replace(char, ' ')

raw_text = raw_text.replace('  ', ' ')
raw_text = raw_text.replace('  ', ' ')<jupyter_output><empty_output><jupyter_text>Now that the book is loaded, we must prepare the data for modeling by the neural network. We cannot model the characters directly, instead we must convert the characters to integers.We can do this easily by first creating a set of all of the distinct characters in the book, then creating a map of each character to a unique integer.<jupyter_code># create mapping of unique chars to integers
chars = sorted(list(set(raw_text)))
char_to_int = dict((c, i) for i, c in enumerate(chars))<jupyter_output><empty_output><jupyter_text>For example, the list of unique sorted lowercase characters in the book is as follows:<jupyter_code>char_to_int<jupyter_output><empty_output><jupyter_text>You can see that there may be some characters that we could remove to further clean up the dataset that will reduce the vocabulary and may improve the modeling process.Now that the book has been loaded and the mapping prepared, we can summarize the dataset.<jupyter_code>n_chars = len(raw_text)
n_vocab = len(chars)
print("Total Characters: ", n_chars)
print("Total Vocab: ", n_vocab)<jupyter_output>Total Characters:  135172
Total Vocab:  29<jupyter_text>We can see that the book has just under 150,000 characters and that when converted to lowercase that there are only 37 distinct characters in the vocabulary for the network to learn. Much more than the 26 in the alphabet.We now need to define the training data for the network. There is a lot of flexibility in how you choose to break up the text and expose it to the network during training.In this tutorial we will split the book text up into subsequences with a fixed length of 100 characters, an arbitrary length. We could just as easily split the data up by sentences and pad the shorter sequences and truncate the longer ones.Each training pattern of the network is comprised of 100 time steps of one character (X) followed by one character output (y). When creating these sequences, we slide this window along the whole book one character at a time, allowing each character a chance to be learned from the 100 characters that preceded it (except the first 100 characters of course).For example, if the sequence length is 5 (for simplicity) then the first two training patterns would be as follows:  CHAPT -> EHAPTE -> RAs we split up the book into these sequences, we convert the characters to integers using our lookup table we prepared earlier.<jupyter_code># prepare the dataset of input to output pairs encoded as integers
seq_length = 100
dataX = []
dataY = []
for i in range(0, n_chars - seq_length, 1):
    seq_in = raw_text[i:i + seq_length]
    seq_out = raw_text[i + seq_length]
    dataX.append([char_to_int[char] for char in seq_in])
    dataY.append(char_to_int[seq_out])
n_patterns = len(dataX)
print("Total Patterns: ", n_patterns)<jupyter_output>Total Patterns:  135072<jupyter_text>Running the code to this point shows us that when we split up the dataset into training data for the network to learn that we have just under 150,000 training pattens. This makes sense as excluding the first 100 characters, we have one training pattern to predict each of the remaining characters.   Now that we have prepared our training data we need to transform it so that it is suitable for use with Keras.First we must transform the list of input sequences into the form [samples, time steps, features] expected by an LSTM network.Next we need to rescale the integers to the range 0-to-1 to make the patterns easier to learn by the LSTM network that uses the sigmoid activation function by default.Finally, we need to convert the output patterns (single characters converted to integers) into a one hot encoding. This is so that we can configure the network to predict the probability of each of the 47 different characters in the vocabulary (an easier representation) rather than trying to force it to predict precisely the next character. Each y value is converted into a sparse vector with a length of 47, full of zeros except with a 1 in the column for the letter (integer) that the pattern represents.We can implement these steps as below.<jupyter_code># reshape X to be [samples, time steps, features]
X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
# normalize
X = X / float(n_vocab)
# one hot encode the output variable
y = np_utils.to_categorical(dataY)<jupyter_output><empty_output><jupyter_text>We can now define our LSTM model. Here we define a single hidden LSTM layer with 256 memory units. The network uses dropout with a probability of 20. The output layer is a Dense layer using the softmax activation function to output a probability prediction for each of the 47 characters between 0 and 1.The problem is really a single character classification problem with 47 classes and as such is defined as optimizing the log loss (cross entropy), here using the ADAM optimization algorithm for speed.<jupyter_code># define the LSTM model
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')<jupyter_output><empty_output><jupyter_text>There is no test dataset. We are modeling the entire training dataset to learn the probability of each character in a sequence.We are not interested in the most accurate (classification accuracy) model of the training dataset. This would be a model that predicts each character in the training dataset perfectly. Instead we are interested in a generalization of the dataset that minimizes the chosen loss function. We are seeking a balance between generalization and overfitting but short of memorization.The network is slow to train (about 300 seconds per epoch on an Nvidia K520 GPU). Because of the slowness and because of our optimization requirements, we will use model checkpointing to record all of the network weights to file each time an improvement in loss is observed at the end of the epoch. We will use the best set of weights (lowest loss) to instantiate our generative model in the next section.<jupyter_code># define the checkpoint
filepath="weights-improvement-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]<jupyter_output><empty_output><jupyter_text>We can now fit our model to the data. Here we use a modest number of 20/30 epochs and a large batch size of 32/64/128 patterns.<jupyter_code>t0 = time.time()

history = model.fit(X, y, epochs=30, batch_size=32, callbacks=callbacks_list, verbose=1)

print(time.time() - t0)
# list all data in history
print(history.history.keys())
# summarize history for loss
plt.figure(figsize=(12,8))
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>The network loss decreased almost every epoch and I expect the network could benefit from training for many more epochs.In the next section we will look at using this model to generate new text sequences. You will see different results because of the stochastic nature of the model, and because it is hard to fix the random seed for LSTM models to get 100% reproducible results. This is not a concern for this generative model.After running the example, you should have a number of weight checkpoint files in the local directory.You can delete them all except the one with the smallest loss value.<jupyter_code>!ls *.hdf5
filename = "weights-improvement-29-1.3749.hdf5"<jupyter_output><empty_output><jupyter_text>Generating Text with an LSTM NetworkGenerating text using the trained LSTM network is relatively straightforward.Firstly, we load the data and define the network in exactly the same way, except the network weights are loaded from a checkpoint file and the network does not need to be trained.<jupyter_code># load the network weights
model.load_weights(filename)
model.compile(loss='categorical_crossentropy', optimizer='adam')<jupyter_output><empty_output><jupyter_text>Also, when preparing the mapping of unique characters to integers, we must also create a reverse mapping that we can use to convert the integers back to characters so that we can understand the predictions.<jupyter_code>int_to_char = dict((i, c) for i, c in enumerate(chars))<jupyter_output><empty_output><jupyter_text>Finally, we need to actually make predictions.The simplest way to use the Keras LSTM model to make predictions is to first start off with a seed sequence as input, generate the next character then update the seed sequence to add the generated character on the end and trim off the first character. This process is repeated for as long as we want to predict new characters (e.g. a sequence of 1,000 characters in length).We can pick a random input pattern as our seed sequence, then print generated characters as we generate them. Running this example first outputs the selected random seed, then each character as it is generated.For example, below are the results from one run of this text generator. The random seed was:<jupyter_code>#pick a random seed
start = numpy.random.randint(0, len(dataX)-1)
pattern = dataX[start]
print("Seed:")
print("\"", ''.join([int_to_char[value] for value in pattern]), "\"")
# generate characters
for i in range(1000):
    x = numpy.reshape(pattern, (1, len(pattern), 1))
    x = x / float(n_vocab)
    prediction = model.predict(x, verbose=0)
    index = numpy.argmax(prediction)
    result = int_to_char[index]
    seq_in = [int_to_char[value] for value in pattern]
    sys.stdout.write(result)
    pattern.append(index)
    pattern = pattern[1:len(pattern)]
print("\nDone.")<jupyter_output>Seed:
"  the fall never come to an end i wonder how many miles i ve fallen by this time she said aloud i mus "
t be gerting so she seies of the porr little thing sas how vhen the roeez ald the morte was she same as the sei gouse  no ohe was so the teite whrh ald the poeenrse ifd doon her hnot and the poeent temesbened hnro the door and the party was soie and the whrtes ald the dorrtuse who had never been io a coeaer of toike so bo sh the said this she woode of the dourt while she had never done tutn and shi said to herself the dorrouse shought alice i dar t be anle they re boeedt that she was not about a cat in a merure triek you dan t bente wou mow the rame thing alice said to the jury and the poeenr oe the tood she sei woud to alice hoowen the dante wou dou t luo thi said the datee it to the said this she was not about a far and taid alice i d neter tore th t she said this she dourd neter shen alice was not aaout the doom and the door and thin she house dnwn to the gadl i dan t be ai[...]<jupyter_text>We can note some observations about the generated text.  + It generally conforms to the line format observed in the original text of less than 80 characters before a new line.+ The characters are separated into word-like groups and most groups are actual English words (e.g. “the”, “little” and “was”), but many do not (e.g. “lott”, “tiie” and “taede”).+ Some of the words in sequence make sense(e.g. “and the white rabbit“), but many do not (e.g. “wese tilel“).The fact that this character based model of the book produces output like this is very impressive. It gives you a sense of the learning capabilities of LSTM networks.The results are not perfect. In the next section we look at improving the quality of results by developing a much larger LSTM network.  Larger LSTM Recurrent Neural NetworkWe got results, but not excellent results in the previous section. Now, we can try to improve the quality of the generated text by creating a much larger network.We will keep the number of memory units the same at 256, but add a second layer.<jupyter_code>model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(256))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')<jupyter_output><empty_output><jupyter_text>We will also change the filename of the checkpointed weights so that we can tell the difference between weights for this network and the previous (by appending the word “bigger” in the filename).<jupyter_code>filepath="weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5"<jupyter_output><empty_output><jupyter_text>Finally, we will increase the number of training epochs from 20 to 50 and use batch size 32/64 to give the network more of an opportunity to be updated and learn.<jupyter_code>t0 = time.time()

checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
history = model.fit(X, y, epochs=50, batch_size=32, callbacks=callbacks_list)

print(time.time() - t0)
# list all data in history
print(history.history.keys())
# summarize history for loss
plt.figure(figsize=(12,8))
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
!ls *.hdf5
filename = "weights-improvement-35-1.1737-bigger.hdf5"
# load the network weights
model.load_weights(filename)
model.compile(loss='categorical_crossentropy', optimizer='adam')
int_to_char = dict((i, c) for i, c in enumerate(chars))<jupyter_output><empty_output><jupyter_text>One example of running this text generation script produces the output below.The randomly chosen seed text was:<jupyter_code>#pick a random seed
start = numpy.random.randint(0, len(dataX)-1)
pattern = dataX[start]
print("Seed:")
print("\"", ''.join([int_to_char[value] for value in pattern]), "\"")
# generate characters
for i in range(1000):
    x = numpy.reshape(pattern, (1, len(pattern), 1))
    x = x / float(n_vocab)
    prediction = model.predict(x, verbose=0)
    index = numpy.argmax(prediction)
    result = int_to_char[index]
    seq_in = [int_to_char[value] for value in pattern]
    sys.stdout.write(result)
    pattern.append(index)
    pattern = pattern[1:len(pattern)]
print("\nDone.")<jupyter_output>Seed:
"  breath and said that s very curious it s all about as curious as it can be said the gryphon it all  "
the things he dound the mock turtle and the mouse we seady to the thing would be a little shing said the king the door said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse we mearn me think i won t said the king the dormouse w[...]
{'fraction_non_alphanumeric': 0.0813379346960446, 'fraction_numerical': 0.018157685160605255, 'mean_word_length': 5.206260296540362, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 78, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '314940', 'license': 'MIT', 'n_tokens_mistral': 5759, 'n_tokens_neox': 5417, 'n_words': 2470, 'path': 'notebooks/ML_SUP_LSTM_Text_Gen.ipynb', 'repo_name': 'hidroware/FGV_Intro_DS', 'size': '18835'}