<jupyter_start><jupyter_text><NAME> - 13 June 2017Practical Deep Learning ILesson 5 - RNNs, NLPCode along of [char-rnn.ipynb](https://github.com/fastai/courses/blob/master/deeplearning1/nbs/char-rnn.ipynb)<jupyter_code>import theano
%matplotlib inline
import os, sys
sys.path.insert(1, os.path.join('utils'))
import utils; reload(utils)
from utils import *
from __future__ import print_function, division
from keras.layers import TimeDistributed, Activation
# https://keras.io/layers/wrappers/
# [Doc:TimeDistributed] this wrapper allows to apply a layer to every temporal slice of an input
# https://keras.io/activations/
# [Doc:Activation] activations can be used through an Activation layer
from numpy.random import choice<jupyter_output><empty_output><jupyter_text>SetupWe haven't really looked into the detail of how this works yet - so this is provided for self-study for those who're interested. We'll look at it closely next week.<jupyter_code>path = get_file('nietzsche.txt', origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt')
text = open(path).read().lower()
print('corpus length:', len(text))
!tail {path} -n25
# unique characters
chars = sorted(list(set(text)))
vocab_size = len(chars) + 1
print('total chars:', vocab_size)
chars.insert(0, "\0")
# the unique characters (in UpLoCase corpus, add 26)
''.join(chars[1:-6])
# create a mapping from char to index in which it appears
char_indices = dict((c, i) for i, c in enumerate(chars))
# create a mapping from index to char
indices_char = dict((i, c) for i, c in enumerate(chars))<jupyter_output><empty_output><jupyter_text>^ This allows us to take the text & convert into a list of numbers, where the number represents the index in which the char appears in the unique-char list.<jupyter_code>idx = [char_indices[c] for c in text]
idx[:10]
''.join(indices_char[i] for i in idx[:70])<jupyter_output><empty_output><jupyter_text>Preprocess and create model<jupyter_code>maxlen = 40
sentences = []
next_chars = []
for i in xrange(0, len(idx) - maxlen + 1):
    sentences.append(idx[i: i + maxlen])
    next_chars.append(idx[i + 1: i + maxlen + 1])
print('nb sequences:', len(sentences))
sentences = np.concatenate([[np.array(o)] for o in sentences[:-2]])
next_chars = np.concatenate([[np.array(o)] for o in next_chars[:-2]])
sentences.shape, next_chars.shape
n_fac = 24<jupyter_output><empty_output><jupyter_text>In Lesson 6: improved: an RNN feeding into an RNN. See [lecture at ~ 1:10:00](https://youtu.be/ll9y1U0SoVY?t=4254)<jupyter_code>model = Sequential([
            Embedding(vocab_size, n_fac, input_length=maxlen),
            LSTM(512, input_dim=n_fac, return_sequences=True, dropout_U=0.2, dropout_W=0.2,
                 consume_less='gpu'),
            Dropout(0.2),
            LSTM(512, return_sequences=True, dropout_U=0.2, dropout_W=0.2,
                 consume_less='gpu'),
            Dropout(0.2),
            TimeDistributed(Dense(vocab_size)),
            Activation('softmax')
        ])
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>def print_example():
    seed_string="ethics is a basic foundation of all that"
    for i in range(320):
        x=np.array([char_indices[c] for c in seed_string[-40:]])[np.newaxis,:]
        preds = model.predict(x, verbose=0)[0][-1]
        preds = preds/np.sum(preds)
        next_char = choice(chars, p=preds)
        seed_string = seed_string + next_char
    print(seed_string)
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=64, nb_epoch=1)
print_example()
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=64, nb_epoch=1)
print_example()
model.optimizer.lr=1e-3
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=128, nb_epoch=1)
print_example()
model.optimizer.lr=1e-4
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=256, nb_epoch=1)
print_example()
%mkdir -p 'data/char_rnn/'
model.save_weights('data/char_rnn.h5')
model.optimizer.lr=1e-5
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=256, nb_epoch=1)
print_example()
model.fit(sentences, np.expand_dims(next_chars, -1), batch_size=128, nb_epoch=1)
print_example()
print_example()
model.save_weights('data/char_rnn.h5')
def print_example(seed_string=''):
    if not seed_string:
        seed_string="ethics is a basic foundation of all that"
    for i in range(320):
        x=np.array([char_indices[c] for c in seed_string[-40:]])[np.newaxis,:]
        preds = model.predict(x, verbose=0)[0][-1]
        preds = preds/np.sum(preds)
        next_char = choice(chars, p=preds)
        seed_string = seed_string + next_char
    print(seed_string)
text ='so um first i was afraid i was petrified'
print_example(text)<jupyter_output>so um first i was afraid i was petrified, and with achievement and
deeper synthesis, in short, a yea and the
great historical problem of things.

35. there is neither a hammer of few of the suffering of cheerful mediocrity is comforted to the "principle"!--it is the last great spoctise only even in him. he discovers much flung part of its claim, the outside
{'fraction_non_alphanumeric': 0.1266290604940673, 'fraction_numerical': 0.022952732931336317, 'mean_word_length': 4.810169491525424, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '305835', 'license': 'MIT', 'n_tokens_mistral': 1861, 'n_tokens_neox': 1798, 'n_words': 456, 'path': 'FAI_old/char_rnn_CodeAlong.ipynb', 'repo_name': 'WNoxchi/Kawkasos', 'size': '5149'}