<jupyter_start><jupyter_text>Text generation using RNN - Character LevelTo generate text using RNN, we need a to convert raw text to a supervised learning problem format.Take, for example, the following corpus:"Her brother shook his head incredulously"First we need to divide the data into tabular format containing input (X) and output (y) sequences. In case of a character level model, the X and y will look like this:|      X     |  Y  ||------------|-----||    Her b   |  r  ||    er br   |  o  ||    r bro   |  t  ||     brot   |  h  ||    broth   |  e  ||    .....   |  .  ||    .....   |  .  ||    ulous   |  l  ||    lousl   |  y  |Note that in the above problem, the sequence length of X is five characters and that of y is one character. Hence, this is a many-to-one architecture. We can, however, change the number of input characters to any number of characters depending on the type of problem.A model is trained on such data. To generate text, we simply give the model any five characters using which it predicts the next character. Then it appends the predicted character to the input sequence (on the extreme right of the sequence) and discards the first character (character on extreme left of the sequence). Then it predicts again using the new sequence and the cycle continues until a fix number of iterations. An example is shown below:Seed text: "incre"|      X                                            |  Y                       ||---------------------------------------------------|--------------------------||                        incre                      |      ||               ncre               |      ||       cre    |      ||       re    |      ||                      ...                          |            ...           |  Notebook Overview1. Preprocess data2. LSTM model3. Generate code<jupyter_code># import libraries
import warnings
warnings.filterwarnings("ignore")

import os
import re
import numpy as np
import random
import sys
import io
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Activation, LSTM
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import get_file<jupyter_output><empty_output><jupyter_text>1. Preprocess data We're going to build a C code generator by training an RNN on a huge corpus of C code (the linux kernel code). You can download the C code used as source text from the following link:https://github.com/torvalds/linux/tree/master/kernelWe have already downloaded the entire kernel folder and stored in a local directory  Load C code<jupyter_code>cd ..
# set path where C files reside

path = r"linux-master/kernel"

os.chdir(path)

file_names = os.listdir()
print(file_names)
# use regex to filter .c files
import re
c_names = ".*\.c$"

c_files = list()

for file in file_names:
    if re.match(c_names, file):
        c_files.append(file)

print(c_files)
# load all c code in a list
full_code = list()
for file in c_files:
    code = open(file, "r", encoding='utf-8')
    full_code.append(code.read())
    code.close()
# let's look at how a typical C code looks like
print(full_code[20])
# merge different c codes into one big c code
text = "\n".join(full_code)
print("Total number of characters in entire code: {}".format(len(text)))
# top_n: only consider first top_n characters and discard the rest for memory and computational efficiency
top_n = 400000
text = text[:top_n]<jupyter_output><empty_output><jupyter_text>Convert characters to integers<jupyter_code># create character to index mapping
chars = sorted(list(set(text)))
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
print("Vocabulary size: {}".format(len(chars)))<jupyter_output>Vocabulary size: 97<jupyter_text>Divide data in input (X) and output (y)  Create sequences<jupyter_code># define length for each sequence
MAX_SEQ_LENGTH = 50          # number of input characters (X) in each sequence 
STEP           = 3           # increment between each sequence
VOCAB_SIZE     = len(chars)  # total number of unique characters in dataset

sentences  = []              # X
next_chars = []              # y

for i in range(0, len(text) - MAX_SEQ_LENGTH, STEP):
    sentences.append(text[i: i + MAX_SEQ_LENGTH])
    next_chars.append(text[i + MAX_SEQ_LENGTH])
print('Number of training samples: {}'.format(len(sentences)))<jupyter_output>Number of training samples: 133317<jupyter_text>Create input and output using the created sequencesWhen you're not using the Embedding layer of the Keras as the very first layer, you need to convert your data in the following format: input shape should be of the form :  (samples, timesteps, features) output shape should be of the form :  (samples, timesteps, features)![Tensor shape](./jupyter resources/rnn_tensor.png)samples: the number of data points (or sequences)timesteps: It's the length of the sequence of your data (the MAX_SEQ_LENGTH variable).features: Number of features depends on the type of problem. In this problem, features is the vocabulary size, that is, the dimensionality of the one-hot encoding matrix using which each character is being represented. If you're working with **images**, features size will be equal to: (height, width, channels), and the input shape will be (training_samples, timesteps, height, width, channels)<jupyter_code># create X and y
X = np.zeros((len(sentences), MAX_SEQ_LENGTH, VOCAB_SIZE), dtype=np.bool)
y = np.zeros((len(sentences), VOCAB_SIZE), dtype=np.bool)
for i, sentence in enumerate(sentences):
    for t, char in enumerate(sentence):
        X[i, t, char_indices[char]] = 1
    y[i, char_indices[next_chars[i]]] = 1
print("Shape of X: {}".format(X.shape))
print("Shape of y: {}".format(y.shape))<jupyter_output>Shape of X: (133317, 50, 97)
Shape of y: (133317, 97)<jupyter_text>Here, X is reshaped to (samples, timesteps, features). We have explicitly mentioned the third dimension (features) because we won't use the Embedding() layer of Keras in this case since there are only 97 characters. Characters can be represented as one-hot encoded vector. There are no word embeddings for characters.  2. LSTM<jupyter_code># define model architecture - using a two-layer LSTM with 128 LSTM cells in each layer
model = Sequential()
model.add(LSTM(128, input_shape=(MAX_SEQ_LENGTH, VOCAB_SIZE), return_sequences=True, dropout=1.0))
model.add(LSTM(128, dropout=1.0))
model.add(Dense(VOCAB_SIZE, activation = "softmax"))

optimizer = Adam(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics = ['acc'])
# check model summary
model.summary()
# fit model
model.fit(X, y, batch_size=128, epochs=20)<jupyter_output><empty_output><jupyter_text>3. Generate code Create a function that will make next character predictions based on temperature. If temperature is greater than 1, the generated characters will be more versatile and diverse. On the other hand, if temperature is less than one, the generated characters will be much more conservative.<jupyter_code># define function to sample next word from a probability array based on temperature
def sample(preds, temperature=1.0):
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)
np.random.multinomial(10, [0.05, 0.9, 0.05], size=2)
# generate code

start_index = random.randint(0, len(text) - MAX_SEQ_LENGTH - 1) # pick random code to start text generation

for diversity in [0.5, 1.0, 1.5]:
        print('-'*50, 'diversity:', diversity)

        generated = ''
        sentence = text[start_index: start_index + MAX_SEQ_LENGTH]
        generated += sentence
        print('----- Generating with seed: "' + sentence + '"')
        sys.stdout.write(generated)

        for i in range(1000):
            x_pred = np.zeros((1, MAX_SEQ_LENGTH, VOCAB_SIZE))
            for t, char in enumerate(sentence):
                x_pred[0, t, char_indices[char]] = 1.

            preds = model.predict(x_pred, verbose=0)[0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            generated += next_char
            sentence = sentence[1:] + next_char

            sys.stdout.write(next_char)
            sys.stdout.flush()
# generate code

start_index = random.randint(0, len(text) - MAX_SEQ_LENGTH - 1) # pick random seed

for diversity in [0.5, 1.0, 1.5]:
        print('-'*50, 'diversity:', diversity)

        generated = ''
        sentence = text[start_index: start_index + MAX_SEQ_LENGTH]
        generated += sentence
        print('----- Generating with seed: "' + sentence + '"')
        sys.stdout.write(generated)

        for i in range(1000):
            x_pred = np.zeros((1, MAX_SEQ_LENGTH, VOCAB_SIZE))
            for t, char in enumerate(sentence):
                x_pred[0, t, char_indices[char]] = 1.

            preds = model.predict(x_pred, verbose=0)[0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            generated += next_char
            sentence = sentence[1:] + next_char

            sys.stdout.write(next_char)
            sys.stdout.flush()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10627345844504021, 'fraction_numerical': 0.013726541554959785, 'mean_word_length': 3.1578243423985732, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '25594', 'license': 'Apache-2.0', 'n_tokens_mistral': 2829, 'n_tokens_neox': 2679, 'n_words': 996, 'path': '14. RNN/.ipynb_checkpoints/2. rnn_code_generator-checkpoint.ipynb', 'repo_name': 'vshubh24/machineLearning', 'size': '9325'}