<jupyter_start><jupyter_text>BabblingLets write a stupid LSTM RNN which learns to generate text based on a corpus fed to it. Keras has a lovely API so we'll use that, backed up by the brunt of Tensorflow.<jupyter_code>import math
import pandas as pd
import numpy as np
import nltk

from numpy.random import choice
from keras.layers import *
from keras.models import Sequential<jupyter_output><empty_output><jupyter_text>Let's load in a big lump of text for the LSTM to read<jupyter_code>book_path = './data/hp_philosophers_stone.txt'

with open(book_path) as f:
    text = f.read().lower()
print('corpus length:', len(text))<jupyter_output>corpus length: 439400<jupyter_text>Then get a set of the unique characters in the text, and call it our vocabulary. Even in normal text the vocabulary is likely to be quite large - 26 upper case characters, 26 lower case characters, and loads of punctuation<jupyter_code>characters = sorted(list(set(text)))
vocab_size = len(characters)

vocab_size<jupyter_output><empty_output><jupyter_text>To make our data computationally interpretable, we should make some kind of index mapping each character to a unique numeric id. We can then represent our full book text as a list of character indicies. In other words, the output will be a long sequence of numbers which spell out the book.<jupyter_code>character_to_index = dict((c, i) for i, c in enumerate(characters))
index_to_character = dict((i, c) for i, c in enumerate(characters))

text_as_indicies = [character_to_index[c] for c in text]<jupyter_output><empty_output><jupyter_text>Now we can start splitting that massively long series of numbers into a load of training sequences. We'll use a sequence length of 40, because, having tested this with a bunch of lengths, 40 is a nice round number that seems to work well. It also gives us enough context to start picking up on grammar and sentence cadence without being excessive.<jupyter_code>sequence_length = 40
num_sequences = len(text) - sequence_length + 1

sequences = [text_as_indicies[i : i + sequence_length] 
             for i in range(num_sequences)]

next_characters = [text_as_indicies[i + 1 : i + sequence_length + 1] 
                   for i in range(num_sequences)]

len(sequences)<jupyter_output><empty_output><jupyter_text>Now we need to come up with the series of next-characters that follow each sequence.<jupyter_code>sequences = np.concatenate([[np.array(seq)] for seq in sequences[:-2]])
next_characters = np.concatenate([[np.array(char)] for char in next_characters[:-2]])<jupyter_output><empty_output><jupyter_text>Here's an example of the two things we'll be using to train the network<jupyter_code>print('sequence:\n' + str(sequences[0]) + '\n')
print('next characters:\n' + str(sequences[1]))<jupyter_output>sequence:
[44 32 29  1 26 39 49  1 47 32 39  1 36 33 46 29 28 52 52 37 42 10  1 25 38
 28  1 37 42 43 10  1 28 45 42 43 36 29 49  8]

next characters:
[32 29  1 26 39 49  1 47 32 39  1 36 33 46 29 28 52 52 37 42 10  1 25 38 28
  1 37 42 43 10  1 28 45 42 43 36 29 49  8  1]<jupyter_text>Building the modelWe're going to use a pretty generic model structure: - embedding- lstm- dropout- lstm - dropout- dense (time distributed)- softmaxWe're also going to use the ADAM optimizer because it's the best and most clever mashup of things (AdaGrad and RMSProp) ever, sparse categorical cross entropy as our loss function, and the mean average error as our metric.<jupyter_code>model = Sequential([Embedding(vocab_size, 
                              24, 
                              input_length=sequence_length),
                    LSTM(512, 
                         input_dim=24, 
                         return_sequences=True, 
                         dropout_U=0.2, 
                         dropout_W=0.2, 
                         consume_less='gpu'),
                    Dropout(0.2),
                    LSTM(512, 
                         return_sequences=True, 
                         dropout_U=0.2, 
                         dropout_W=0.2, 
                         consume_less='gpu'),
                    Dropout(0.2),
                    TimeDistributed(Dense(vocab_size)),
                    Activation('softmax')])
model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['mae'])<jupyter_output><empty_output><jupyter_text>Training the model<jupyter_code>model.optimizer.lr = 0.001

model.fit(sequences, 
          np.expand_dims(next_characters,-1), 
          batch_size=64, 
          nb_epoch=1)<jupyter_output>/home/ubuntu/anaconda2/envs/py36/lib/python3.6/site-packages/keras/models.py:851: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  warnings.warn('The `nb_epoch` argument in `fit` '<jupyter_text>Now that we've trained the model and optimised all of the weights in the network, we can save them to an `.h5` file.<jupyter_code>model.save_weights('models/weights.h5')<jupyter_output><empty_output><jupyter_text>Reloading a pretrained modelIf you've build the model and trained it elsewhere, you can reload it by calling `.load_weights()` with the path to the `.h5` file, as follows<jupyter_code>model.load_weights('models/weights.h5')<jupyter_output><empty_output><jupyter_text>Babbling<jupyter_code>def babble(seed_string=' '*40, output_length=500):
    '''
    Say a lot of stupid stuff based on all of the input text 
    that we trained the model on
    
    Parameters
    ----------
    seed_string : string (optional)
        The story that you want your idiot network to be 
        inspired by
        default = 40 spaces
    
    output_length : int (optional)
        how long do you want the idiot network to talk for
        default = 500
        
    Returns
    -------
    seed_string : string
        the original seed string with 500 characters of new 
        stuff attached to the end of it
    '''
    for i in range(output_length):
        x = np.array([character_to_index[c] for c in seed_string[-40:]])[np.newaxis,:]
        preds = model.predict(x, verbose=0)[0][-1]
        preds = preds / np.sum(preds)
        next_character = choice(characters, p=preds)
        seed_string += next_character
    print(seed_string)
babble()<jupyter_output>in it and walked up by trouble glumping on his tricks as harry left harry's broom and back.　　"let's everyone else had to go bit to look at each other.　　"just then," harry. but harry, too, ron, and ron fruffled so back for us," ron sighed, as they telling himself against the stone.　　"then the armchairs wouldn't over his mouth. the flash of the days to give us them id it, just a wafd."　　this is it must be sort. i dungeon had left professor mcgonagall noticing making the first i've got to said. "co
{'fraction_non_alphanumeric': 0.08230940044411547, 'fraction_numerical': 0.03375277572168764, 'mean_word_length': 3.1755253399258345, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '665268', 'license': 'MIT', 'n_tokens_mistral': 2268, 'n_tokens_neox': 2010, 'n_words': 747, 'path': 'babble.ipynb', 'repo_name': 'harrisonpim/babbling', 'size': '6755'}