<jupyter_start><jupyter_text>Learning the AlphabetThis is an example of a simple [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory) that is powerful enough to learn the alphabet. It is trained with strings that look like the alphabet.While this seems trivial, RNNs are capable of learning more complex text sequences, such as the works of Shakespeare or computer source code. Although training RNNs is computationally more expensive compared with other network structures, this example is simple enough to train on a CPU.<jupyter_code># See https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/
# Small LSTM Network to Generate Text for Alice in Wonderland
import sys
import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras.callbacks import ModelCheckpoint
from keras.utils import np_utils
from matplotlib import pyplot as plt<jupyter_output>Using TensorFlow backend.<jupyter_text>Training DataThe training data is generated using characters from 'a' to 'z'. Several sequences are concatenated to produce a longer sequence of characters.<jupyter_code># load ascii text and covert to lowercase
raw_text = ''.join([chr(x) for x in range(ord('a'), ord('z')+1)])
raw_text += raw_text
raw_text += raw_text
raw_text += raw_text
# create mapping of unique chars to integers
chars = sorted(list(set(raw_text)))
char_to_int = dict((c, i) for i, c in enumerate(chars))
int_to_char = dict((i, c) for i, c in enumerate(chars))

print("Subset of training data: %s" % raw_text[0:64])
# summarize the loaded data
n_chars = len(raw_text)
n_vocab = len(chars)
print("Total Characters: ", n_chars)
print("Total Vocab: ", n_vocab)<jupyter_output>Subset of training data: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijkl
Total Characters:  208
Total Vocab:  26<jupyter_text>Prepare SequencesInput and expected output sequences are prepared and encoded as integers.<jupyter_code>seq_length = 10
dataX = []
dataY = []
for i in range(0, n_chars - seq_length, 1):
    seq_in = raw_text[i:i + seq_length]
    seq_out = raw_text[i + seq_length]
    dataX.append([char_to_int[char] for char in seq_in])
    dataY.append(char_to_int[seq_out])
n_patterns = len(dataX)
print("Total Patterns: ", n_patterns)<jupyter_output>Total Patterns:  198<jupyter_text>Train Model<jupyter_code># reshape X to be [samples, time steps, features]
X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
# normalize
X = X / float(n_vocab)
# one hot encode the output variable
y = np_utils.to_categorical(dataY)
# define the LSTM model
model = Sequential()
model.add(LSTM(16, input_shape=(X.shape[1], X.shape[2])))
#model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
# define the checkpoint
#filepath="weights-improvement-{epoch:02d}-{loss:.4f}.hdf5"
#checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
#callbacks_list = [checkpoint]
# fit the model
#model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)
history = model.fit(X, y, epochs=100, batch_size=1, verbose=2)<jupyter_output>WARNING:tensorflow:From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/100
 - 2s - loss: 3.2691
Epoch 2/100
 - 1s - loss: 3.2344
Epoch 3/100
 - 2s - loss: 3.1111
Epoch 4/100
 - 2s - loss: 2.7656
Epoch 5/100
 - 2s - loss: 2.4020
Epoch 6/100
 - 2s - loss: 2.1680
Epoch 7/100
 - 2s - loss: 2.0238
Epoch 8/100
 - 2s - loss: 1.9023
Epoch 9/100
 - 2s - loss: 1.8063
Epoch 10/100
 - 1s - loss: 1.7181
Epoch 11/100
 - 1s - loss: 1.6487
Epoch 12/100
 - 2s - loss: 1.5739
Epoch 13/100
 - 2s - loss: 1.5074
Epoch 14/100[...]<jupyter_text>Model Loss<jupyter_code>plt.plot(history.history['loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train'], loc='upper left')
plt.show()
print("Minimum loss: %.3f" % history.history['loss'][-1])<jupyter_output><empty_output><jupyter_text>PredictionA random starting sequence is used to seed prediction. The output is almost perfect.<jupyter_code># pick a random seed
start = numpy.random.randint(0, len(dataX)-1)
pattern = dataX[start]
#pattern = [start]
print("Seed:")
print("\"", ''.join([int_to_char[value] for value in pattern]), "\"")
# generate characters
for i in range(100):
    x = numpy.reshape(pattern, (1, len(pattern), 1))
    x = x / float(n_vocab)
    prediction = model.predict(x, verbose=0)
    index = numpy.argmax(prediction)
    result = int_to_char[index]
    seq_in = [int_to_char[value] for value in pattern]
    sys.stdout.write(result)
    pattern.append(index)
    pattern = pattern[1:len(pattern)]
print("\nAll done!")<jupyter_output>Seed:
" stuvwxyzab "
cdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwx
All done!
{'fraction_non_alphanumeric': 0.11617565314063369, 'fraction_numerical': 0.03705762460626274, 'mean_word_length': 5.480192076830733, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '63790', 'license': 'MIT', 'n_tokens_mistral': 2057, 'n_tokens_neox': 1876, 'n_words': 514, 'path': 'Text/Alphabet.ipynb', 'repo_name': 'cas10ph/MachineLearning', 'size': '5397'}