<jupyter_start><jupyter_text>Generation of mexican municipalitiesMexico has over 2400 municipalities with names where the root of their names come from nahuatl (Aztec's language), spanish, Maya, and many more regional languages. And because of that Mexico has a very rich diversity when it comes to names. This notebook tries to use a RNN to learn and generate names that seems from mexican municipalities names.I'm using the Tensorflow's notebook that generate Shakespeare's style of writing that can be found [here](https://www.tensorflow.org/tutorials/sequences/text_generation). But I'm changing the corpus to be the mexican municipalities names and changing the n-gram of characters to be only 1 because usually the names of the municipalities are just 1 word. Load packages<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf
tf.enable_eager_execution()

from google.colab import files
import numpy as np
import os
import time
import urllib.request<jupyter_output><empty_output><jupyter_text>I load the dataset from my github and display it<jupyter_code>url = 'https://raw.githubusercontent.com/RicardoHE97/RN-Unison/master/modelo_de_generacion_de_texto_con_redes_recurrentes/municipios.txt'
response = urllib.request.urlopen(url)
data = response.read()      # a `bytes` object
text = data.decode('utf-8')
text<jupyter_output><empty_output><jupyter_text>Display the lenght of the whole corpus.<jupyter_code>print ('Length of text: {} characters'.format(len(text)))<jupyter_output>Length of text: 35530 characters<jupyter_text>Display the lenght of unique characters from the corpus<jupyter_code># The unique characters in the file
vocab = sorted(set(text))
print ('{} unique characters'.format(len(vocab)))<jupyter_output>68 unique characters<jupyter_text>Process the text  Vectorize the textBefore training, we need to map strings to a numerical representation. Create two lookup tables: one mapping characters to numbers, and another for numbers to characters.<jupyter_code># Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
idx2char = np.array(vocab)

text_as_int = np.array([char2idx[c] for c in text])
print('{')
for char,_ in zip(char2idx, range(20)):
    print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))
print('  ...\n}')<jupyter_output>{
  '\n':   0,
  ' ' :   1,
  ',' :   2,
  '-' :   3,
  '.' :   4,
  '0' :   5,
  '2' :   6,
  '6' :   7,
  '8' :   8,
  'A' :   9,
  'B' :  10,
  'C' :  11,
  'D' :  12,
  'E' :  13,
  'F' :  14,
  'G' :  15,
  'H' :  16,
  'I' :  17,
  'J' :  18,
  'K' :  19,
  ...
}<jupyter_text>Create training examples and targets<jupyter_code># The maximum length sentence we want for a single input in characters
seq_length = 1
examples_per_epoch = len(text)//seq_length

# Create training examples / targets
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)

for i in char_dataset.take(5):
  print(idx2char[i.numpy()])
  
print(examples_per_epoch)
sequences = char_dataset.batch(seq_length+1, drop_remainder=True)

for item in sequences.take(5):
  print(repr(''.join(idx2char[item.numpy()])))
def split_input_target(chunk):
    input_text = chunk[:-1]
    target_text = chunk[1:]
    return input_text, target_text

dataset = sequences.map(split_input_target)
for input_example, target_example in  dataset.take(1):
  print ('Input data: ', repr(''.join(idx2char[input_example.numpy()])))
  print ('Target data:', repr(''.join(idx2char[target_example.numpy()])))
for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):
    print("Step {:4d}".format(i))
    print("  input: {} ({:s})".format(input_idx, repr(idx2char[input_idx])))
    print("  expected output: {} ({:s})".format(target_idx, repr(idx2char[target_idx])))
# Batch size
BATCH_SIZE = 64
steps_per_epoch = examples_per_epoch//BATCH_SIZE

# Buffer size to shuffle the dataset
# (TF data is designed to work with possibly infinite sequences,
# so it doesn't attempt to shuffle the entire sequence in memory. Instead,
# it maintains a buffer in which it shuffles elements).
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

dataset
# Length of the vocabulary in chars
vocab_size = len(vocab)

# The embedding dimension
embedding_dim = 256

# Number of RNN units
rnn_units = 1024<jupyter_output><empty_output><jupyter_text>Try to use GPUs if we can<jupyter_code>if tf.test.is_gpu_available():
  rnn = tf.keras.layers.CuDNNGRU
else:
  import functools
  rnn = functools.partial(
    tf.keras.layers.GRU, recurrent_activation='sigmoid')
def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
  model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim,
                              batch_input_shape=[batch_size, None]),
    rnn(rnn_units,
        return_sequences=True,
        recurrent_initializer='glorot_uniform',
        stateful=True),
    tf.keras.layers.Dense(vocab_size)
  ])
  return model
model = build_model(
  vocab_size = len(vocab),
  embedding_dim=embedding_dim,
  rnn_units=rnn_units,
  batch_size=BATCH_SIZE)
for input_example_batch, target_example_batch in dataset.take(1):
  example_batch_predictions = model(input_example_batch)
  print(example_batch_predictions.shape, "# (batch_size, sequence_length, vocab_size)")
model.summary()
sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)
sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()
sampled_indices
#print(repr("".join(idx2char[tuple(input_example_batch[0])])))
print("Input: \n", repr("".join(idx2char[tuple(input_example_batch[0])])))
print()
print("Next Char Predictions: \n", repr("".join(idx2char[sampled_indices ])))<jupyter_output>Input: 
 '.'

Next Char Predictions: 
 'B'<jupyter_text>The standard tf.keras.losses.sparse_categorical_crossentropy loss function works in this case because it is applied across the last dimension of the predictions.<jupyter_code>def loss(labels, logits):
  return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)

example_batch_loss  = loss(target_example_batch, example_batch_predictions)
print("Prediction shape: ", example_batch_predictions.shape, " # (batch_size, sequence_length, vocab_size)")
print("scalar_loss:      ", example_batch_loss.numpy().mean())
model.compile(
    optimizer = tf.train.AdamOptimizer(),
    loss = loss)
# Directory where the checkpoints will be saved
checkpoint_dir = './training_checkpoints'
# Name of the checkpoint files
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_prefix,
    save_weights_only=True)<jupyter_output><empty_output><jupyter_text>We train the model<jupyter_code>EPOCHS=20
history = model.fit(dataset.repeat(), epochs=EPOCHS, steps_per_epoch=steps_per_epoch, callbacks=[checkpoint_callback])<jupyter_output>Epoch 1/20
553/555 [============================>.] - ETA: 0s - loss: 2.5796WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py:1436: update_checkpoint_state (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.train.CheckpointManager to manage checkpoints rather than manually editing the Checkpoint proto.
555/555 [==============================] - 6s 10ms/step - loss: 2.5791
Epoch 2/20
555/555 [==============================] - 4s 8ms/step - loss: 2.3519
Epoch 3/20
555/555 [==============================] - 4s 8ms/step - loss: 2.3387
Epoch 4/20
555/555 [==============================] - 4s 8ms/step - loss: 2.3303
Epoch 5/20
555/555 [==============================] - 4s 8ms/step - loss: 2.3265
Epoch 6/20
555/555 [==============================] - 4s 8ms/step - loss: 2.3228
Epoch 7/20
555/555 [==============================] - 5s 9ms/step - los[...]<jupyter_text>Load latestest checkpoint<jupyter_code>tf.train.latest_checkpoint(checkpoint_dir)
model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))

model.build(tf.TensorShape([1, None]))<jupyter_output><empty_output><jupyter_text>Create a function that use de model<jupyter_code>def generate_text(model, start_string):
  # Evaluation step (generating text using the learned model)

  # Number of characters to generate
  num_generate = 1000

  # Converting our start string to numbers (vectorizing)
  input_eval = [char2idx[s] for s in start_string]
  input_eval = tf.expand_dims(input_eval, 0)

  # Empty string to store our results
  text_generated = []

  # Low temperatures results in more predictable text.
  # Higher temperatures results in more surprising text.
  # Experiment to find the best setting.
  temperature = 1.0

  # Here batch size == 1
  model.reset_states()
  for i in range(num_generate):
      predictions = model(input_eval)
      # remove the batch dimension
      predictions = tf.squeeze(predictions, 0)

      # using a multinomial distribution to predict the word returned by the model
      predictions = predictions / temperature
      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

      # We pass the predicted word as the next input to the model
      # along with the previous hidden state
      input_eval = tf.expand_dims([predicted_id], 0)

      text_generated.append(idx2char[predicted_id])

  return (start_string + ''.join(text_generated))
print(generate_text(model, start_string=u"A")) # Doesn't matter which word is started<jupyter_output>A. Pe
Fe
Tin Mel lo Ixlóbobl Chingro dechuca
Sa Mamantatlahepelán
Chutí
Jorosane
Renotixtachitlolamal
Zil Abarían dexépan
Jon d
Sol Juoco Naro Rotlaltinris Llaha Margo Il -Daxco
Ron Or
Sastl Pe Atall Lues
Arapes Moraruapaorasisitepar Mapan Carín
Canacáracínco
Salvanchto Ravaranuan
Elcalaroyánan Pén Hote Ojuelilosin
Emónco
Amora Oco
Callo
Canagos
Alva
Atléha
Chuiántl Medez Vipel Jitaz
Catlántarindepa Mas
Cilahumpue Dían Gosalápec
Buixtitl
Domucía
Lue Charla -Donc Ac
C Coré depusqun den
Con
Sixteltaz
Undan Nu Jistlan An Co
Alos
Zo <NAME>
Hocosí
Tlgasénggoran Cos Caz Xitlahavan Ja
Peontlápoto
Zas Biran dacon desporé Milecotlumbare do doblteroltlo
Tulac
Lo Ro Cocostán
Flántan lanjas
H<NAME> <NAME>
San Buihujo Malchaz Ayateoran
Sastl Repimos Bechitl
Pe
Marrés
Lopa Do
Huepetlote Ma Ma Quan Sat dalán Tranla Labiaracáro
Gonterietloparma
Ma Mo
San Caqui. Ba Alan Luiber Combrin Comos Juia
<NAME>
Tiazuhueza
<NAME>
Min de Jo Textlachua
Istlánto Az[...]
{'fraction_non_alphanumeric': 0.12256371814092953, 'fraction_numerical': 0.022769865067466265, 'mean_word_length': 5.345422116527943, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '290312', 'license': 'MIT', 'n_tokens_mistral': 3848, 'n_tokens_neox': 3618, 'n_words': 953, 'path': 'Generation_of_mexican_municipalities_names_tensorflow.ipynb', 'repo_name': 'RicardoHE97/UC-NLP-2019-1', 'size': '10723'}