<jupyter_start><jupyter_text>Masnavi mANNavi This tutorial demonstrates how to generate text using a character-based RNN. We will work with a dataset of Rumi's Masnavi Manavi Given a sequence of characters from this data , train a model to predict the next character in the sequence. Longer sequences of text can be generated by calling the model repeatedly.This tutorial includes runnable code implemented using [tf.keras](https://www.tensorflow.org/programmers_guide/keras) and [eager execution](https://www.tensorflow.org/programmers_guide/eager). The following is sample output when the model in this tutorial trained for 30 epochs, and started with the word "به":به درد و سردهم مخنث را نهان پيدا شود	سنگ و كوه و ماهشان بد پيش تواين كرامت يافت گردش دم به دست	آتشى خواهين فقير شاه مردمست آن كه خوش شدند آن كردگار	بى‏اسب زين سايه در جست او قماچشم داند جست و باطن زير چاد	كان به چاهى مى‏كنى هم از خداخويشى و ديوار تن از دست تست	پس ندانستيم اندر مشرق است‏صبر كن با اين دو معنيهاى نور	كه رود جز بس عدو ضريرو آن پيمبر گفت او را حصرت زدند	هيچ گشتيم از دروغ آن جا روى‏روز روشن گردد آن در دود تو	سوى موسى زيركى رنجور رفت‏گفت دارم در گذار اين اعتقان	آمديم اندر هلا در نوع دان‏هم تو بر با هم از آن آموخته است	فايم آن فرحود خود را از كل‏ايمن آب بود آخر زمان	كه دمى سازن مرا پيش كشيدكيست ما زين افتضاط فقل را	جفت در خشم آمد و رفته توارنه به دست آمد كه اى من سرخ گشت	گنج نوع و خواب را ياريك شوكى كنند آن پير انسبان مان خوى رو سوى خانه‏ى زردها را بر تنت	با سرشتن را بود هم در پذير	كه نباشد طبع شير و جهل شيرپا خلوص است از ناودان فر من خوشند يك ذوت آب از جبيله‏ى قوتى است	عاقلى بر صد معرف اين ستوخ‏اين ندارد جان مادر مى‏فتد	چون نهاد او شير مردانه بجسWhile some of the sentences are grammatical, most do not make sense. The model has not learned the meaning of words, but consider:* The model is character-based. When training started, the model did not know how to spell a word, or that words were even a unit of text.* As demonstrated below, the model is trained on small batches of text (100 characters each), and is still able to generate a longer sequence of text with coherent structure.  Setup  Import TensorFlow and other libraries<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals

!pip install tensorflow-gpu==2.0.0-alpha0
import tensorflow as tf

import numpy as np
import os
import time<jupyter_output>Collecting tensorflow-gpu==2.0.0-alpha0
[?25l  Downloading https://files.pythonhosted.org/packages/1a/66/32cffad095253219d53f6b6c2a436637bbe45ac4e7be0244557210dc3918/tensorflow_gpu-2.0.0a0-cp36-cp36m-manylinux1_x86_64.whl (332.1MB)
[K    100% |████████████████████████████████| 332.1MB 52kB/s 
[?25hRequirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.7.1)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.2.2)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.15.0)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.33.1)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.0.7)
Requirement already satisfied: ker[...]<jupyter_text>Load Masnavi datasetOriginally Masnavi is a series of six books of poetry that together amount to around 25,000 verses or 50,000 line. hover to the books folder where you can find them seperately in doc format with extra credits and organized. also there is a Masnavi.txt file that contains all six books plain poets one after another.<jupyter_code>path_to_file = '/Books/Masnavi.txt'<jupyter_output><empty_output><jupyter_text>Read the dataFirst, look in the text:<jupyter_code># Read, then decode for py2 compat.
text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
# length of text is the number of characters in it
print ('Length of text: {} characters'.format(len(text)))
# Take a look at the first 250 characters in text
print(text[:225])
# The unique characters in the file
vocab = sorted(set(text))
print ('{} unique characters'.format(len(vocab)))<jupyter_output>59 unique characters<jupyter_text>Process the text  Vectorize the textBefore training, we need to map strings to a numerical representation. Create two lookup tables: one mapping characters to numbers, and another for numbers to characters.<jupyter_code># Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
idx2char = np.array(vocab)

text_as_int = np.array([char2idx[c] for c in text])<jupyter_output><empty_output><jupyter_text>Now we have an integer representation for each character. Notice that we mapped the character as indexes from 0 to `len(unique)`.<jupyter_code>print('{')
for char,_ in zip(char2idx, range(20)):
    print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))
print('  ...\n}')
# Show how the first 13 characters from the text are mapped to integers
print ('{} ---- characters mapped to int ---- > {}'.format(repr(text[:13]), text_as_int[:13]))<jupyter_output>'بشنو از نى چو' ---- characters mapped to int ---- > [16 28 40 42  3 15 26  3 40 43  3 54 42]<jupyter_text>The prediction task Given a character, or a sequence of characters, what is the most probable next character? This is the task we're training the model to perform. The input to the model will be a sequence of characters, and we train the model to predict the output—the following character at each time step.Since RNNs maintain an internal state that depends on the previously seen elements, given all the characters computed until this moment, what is the next character?  Create training examples and targetsNext divide the text into example sequences. Each input sequence will contain `seq_length` characters from the text.For each input sequence, the corresponding targets contain the same length of text, except shifted one character to the right.To do this first use the `tf.data.Dataset.from_tensor_slices` function to convert the text vector into a stream of character indices.<jupyter_code># The maximum length sentence we want for a single input in characters
seq_length = 100
examples_per_epoch = len(text)//seq_length

# Create training examples / targets
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)

for i in char_dataset.take(5):
  print(idx2char[i.numpy()])<jupyter_output>ب
ش
ن
و<jupyter_text>The `batch` method lets us easily convert these individual characters to sequences of the desired size.<jupyter_code>sequences = char_dataset.batch(seq_length+1, drop_remainder=True)

for item in sequences.take(5):
  print(repr(''.join(idx2char[item.numpy()])))<jupyter_output>'بشنو از نى چون حكايت مى\u200fكند\tاز جدايى\u200fها شكايت مى\u200fكند\r\nكز نيستان تا مرا ببريده\u200fاند\tدر نفيرم مرد و زن ن'
'اليده\u200fاند\r\nسينه خواهم شرحه شرحه از فراق\tتا بگويم شرح درد اشتياق\u200f\r\nهر كسى كاو دور ماند از اصل خويش\tباز'
' جويد روزگار وصل خويش\u200f\r\nمن به هر جمعيتى نالان شدم\tجفت بد حالان و خوش حالان شدم\u200f\r\nهر كسى از ظن خود شد '
'يار من\tاز درون من نجست اسرار من\u200f\r\nسر من از ناله\u200fى من دور نيست\tليك چشم و گوش را آن نور نيست\u200f\r\nتن ز جان'
' و جان ز تن مستور نيست\tليك كس را ديد جان دستور نيست\u200f\r\nآتش است اين بانگ ناى و نيست باد\tهر كه اين آتش ن'<jupyter_text>For each sequence, duplicate and shift it to form the input and target text by using the `map` method to apply a simple function to each batch:<jupyter_code>def split_input_target(chunk):
    input_text = chunk[:-1]
    target_text = chunk[1:]
    return input_text, target_text

dataset = sequences.map(split_input_target)<jupyter_output><empty_output><jupyter_text>Each index of these vectors are processed as one time step. For the input at time step 0, the model receives the index for "F" and trys to predict the index for "i" as the next character. At the next timestep, it does the same thing but the `RNN` considers the previous step context in addition to the current input character.<jupyter_code>for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):
    print("Step {:4d}".format(i))
    print("  input: {} ({:s})".format(input_idx, repr(idx2char[input_idx])))
    print("  expected output: {} ({:s})".format(target_idx, repr(idx2char[target_idx])))<jupyter_output>Step    0
  input: 16 ('ب')
  expected output: 28 ('ش')
Step    1
  input: 28 ('ش')
  expected output: 40 ('ن')
Step    2
  input: 40 ('ن')
  expected output: 42 ('و')
Step    3
  input: 42 ('و')
  expected output: 3 (' ')
Step    4
  input: 3 (' ')
  expected output: 15 ('ا')<jupyter_text>Create training batchesWe used `tf.data` to split the text into manageable sequences. But before feeding this data into the model, we need to shuffle the data and pack it into batches.<jupyter_code># Batch size
BATCH_SIZE = 64

# Buffer size to shuffle the dataset
# (TF data is designed to work with possibly infinite sequences,
# so it doesn't attempt to shuffle the entire sequence in memory. Instead,
# it maintains a buffer in which it shuffles elements).
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

dataset<jupyter_output><empty_output><jupyter_text>Build The Model Use `tf.keras.Sequential` to define the model. For this simple example three layers are used to define our model:* `tf.keras.layers.Embedding`: The input layer. A trainable lookup table that will map the numbers of each character to a vector with `embedding_dim` dimensions;* `tf.keras.layers.GRU`: A type of RNN with size `units=rnn_units` (You can also use a LSTM layer here.)* `tf.keras.layers.Dense`: The output layer, with `vocab_size` outputs.<jupyter_code># Length of the vocabulary in chars
vocab_size = len(vocab)

# The embedding dimension
embedding_dim = 256

# Number of RNN units
rnn_units = 1024
def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
  model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim,
                              batch_input_shape=[batch_size, None]),
    tf.keras.layers.LSTM(rnn_units,
                        return_sequences=True,
                        stateful=True,
                        recurrent_initializer='glorot_uniform'),
    tf.keras.layers.Dense(vocab_size)
  ])
  return model
model = build_model(
  vocab_size = len(vocab),
  embedding_dim=embedding_dim,
  rnn_units=rnn_units,
  batch_size=BATCH_SIZE)
for input_example_batch, target_example_batch in dataset.take(1):
  example_batch_predictions = model(input_example_batch)
  print(example_batch_predictions.shape, "# (batch_size, sequence_length, vocab_size)")<jupyter_output>(64, 100, 59) # (batch_size, sequence_length, vocab_size)<jupyter_text>In the above example the sequence length of the input is `100` but the model can be run on inputs of any length:<jupyter_code>model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (64, None, 256)           15104     
_________________________________________________________________
unified_lstm (UnifiedLSTM)   (64, None, 1024)          5246976   
_________________________________________________________________
dense (Dense)                (64, None, 59)            60475     
=================================================================
Total params: 5,322,555
Trainable params: 5,322,555
Non-trainable params: 0
_________________________________________________________________<jupyter_text>To get actual predictions from the model we need to sample from the output distribution, to get actual character indices. This distribution is defined by the logits over the character vocabulary.Note: It is important to _sample_ from this distribution as taking the _argmax_ of the distribution can easily get the model stuck in a loop.Try it for the first example in the batch:  Train the model At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character.  Attach an optimizer, and a loss function The standard `tf.keras.losses.sparse_softmax_crossentropy` loss function works in this case because it is applied across the last dimension of the predictions.Because our model returns logits, we need to set the `from_logits` flag.<jupyter_code>def loss(labels, logits):
  return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)

example_batch_loss  = loss(target_example_batch, example_batch_predictions)
print("Prediction shape: ", example_batch_predictions.shape, " # (batch_size, sequence_length, vocab_size)")
print("scalar_loss:      ", example_batch_loss.numpy().mean())<jupyter_output>Prediction shape:  (64, 100, 59)  # (batch_size, sequence_length, vocab_size)
scalar_loss:       4.0772667<jupyter_text>Configure the training procedure using the `tf.keras.Model.compile` method. We'll use `tf.keras.optimizers.Adam` with default arguments and the loss function.<jupyter_code>model.compile(optimizer='adam', loss=loss)<jupyter_output><empty_output><jupyter_text>Configure checkpoints<jupyter_code># Directory where the checkpoints will be saved
checkpoint_dir = './checkpoints'
# Name of the checkpoint files
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_prefix,
    save_weights_only=True)<jupyter_output><empty_output><jupyter_text>Execute the training In Colab, set the runtime to GPU for faster training.<jupyter_code>EPOCHS=30
history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])<jupyter_output>Epoch 1/30
236/236 [==============================] - 18s 77ms/step - loss: 1.6055
Epoch 2/30
236/236 [==============================] - 19s 79ms/step - loss: 1.5528
Epoch 3/30
236/236 [==============================] - 18s 76ms/step - loss: 1.5034
Epoch 4/30
236/236 [==============================] - 18s 76ms/step - loss: 1.4559
Epoch 5/30
236/236 [==============================] - 18s 76ms/step - loss: 1.4088
Epoch 6/30
236/236 [==============================] - 18s 77ms/step - loss: 1.3609
Epoch 7/30
236/236 [==============================] - 18s 78ms/step - loss: 1.3115
Epoch 8/30
236/236 [==============================] - 18s 78ms/step - loss: 1.2648
Epoch 9/30
236/236 [==============================] - 18s 78ms/step - loss: 1.2191
Epoch 10/30
236/236 [==============================] - 18s 76ms/step - loss: 1.1727
Epoch 11/30
236/236 [==============================] - 18s 77ms/step - loss: 1.1313
Epoch 12/30
236/236 [==============================] - 18s 77ms/step - loss: 1.0971
E[...]<jupyter_text>Generate text  Restore the latest checkpoint<jupyter_code>tf.train.latest_checkpoint(checkpoint_dir)
model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))

model.build(tf.TensorShape([1, None]))
model.summary()
def generate_text(model, start_string):
  # Evaluation step (generating text using the learned model)

  # Number of characters to generate
  num_generate = 1000

  # Converting our start string to numbers (vectorizing)
  input_eval = [char2idx[s] for s in start_string]
  input_eval = tf.expand_dims(input_eval, 0)

  # Empty string to store our results
  text_generated = []

  # Low temperatures results in more predictable text.
  # Higher temperatures results in more surprising text.
  # Experiment to find the best setting.
  temperature = 1.0

  # Here batch size == 1
  model.reset_states()
  for i in range(num_generate):
      predictions = model(input_eval)
      # remove the batch dimension
      predictions = tf.squeeze(predictions, 0)

      # using a categorical distribution to predict the word returned by the model
      predictions = predictions / temperature
      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

      # We pass the predicted word as the next input to the model
      # along with the previous hidden state
      input_eval = tf.expand_dims([predicted_id], 0)

      text_generated.append(idx2char[predicted_id])

  return (start_string + ''.join(text_generated))
print(generate_text(model, start_string=u"به"))<jupyter_output>به درد و سرد
هم مخنث را نهان پيدا شود	سنگ و كوه و ماهشان بد پيش تو
اين كرامت يافت گردش دم به دست	آتشى خواهين فقير شاه مرد
مست آن كه خوش شدند آن كردگار	بى‏اسب زين سايه در جست او قما
چشم داند جست و باطن زير چاد	كان به چاهى مى‏كنى هم از خدا
خويشى و ديوار تن از دست تست	پس ندانستيم اندر مشرق است‏
صبر كن با اين دو معنيهاى نور	كه رود جز بس عدو ضرير
و آن پيمبر گفت او را حصرت زدند	هيچ گشتيم از دروغ آن جا روى‏
روز روشن گردد آن در دود تو	سوى موسى زيركى رنجور رفت‏
گفت دارم در گذار اين اعتقان	آمديم اندر هلا در نوع دان‏
هم تو بر با هم از آن آموخته است	فايم آن فرحود خود را از كل‏
ايمن آب بود آخر زمان	كه دمى سازن مرا پيش كشيد
كيست ما زين افتضاط فقل را	جفت در خشم آمد و رفته توار
نه به دست آمد كه اى من سرخ گشت	گنج نوع و خواب را ياريك شو
كى كنند آن پير انسبان مان خوى رو سوى خانه‏ى زردها را بر تنت	با سرشتن را بود هم در پذير	كه نباشد طبع شير و جهل شير
پا خلوص است از ناودان فر من خوشند يك ذوت آب از جبيله‏ى قوتى است	عاقلى بر صد معرف اين ستوخ‏
اين ندارد جان مادر مى‏فتد	چون نهاد او شير مردانه ب[...]
{'fraction_non_alphanumeric': 0.10953058321479374, 'fraction_numerical': 0.03453769559032717, 'mean_word_length': 4.671506937721846, 'pattern_counts': {'":': 1, '<': 58, '<?xml version=': 0, '>': 65, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '254429', 'license': 'MIT', 'n_tokens_mistral': 7434, 'n_tokens_neox': 6439, 'n_words': 2079, 'path': 'masnavi_mANNavi.ipynb', 'repo_name': 'AryanSh98/Molana-High-On-ANNs', 'size': '17575'}