<jupyter_start><jupyter_text>Dataset数据集为莎士比亚数据集，包含了一些莎士比亚文章<jupyter_code>path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')
# Read, then decode for py2 compat.
text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
# length of text is the number of characters in it
print('Length of text: {} characters'.format(len(text)))
# Take a look at the first 250 characters in text
print(text[:250])
# The unique characters in the file
vocab = sorted(set(text))
print('{} unique characters'.format(len(vocab)))<jupyter_output>65 unique characters<jupyter_text>Process Text 我们需要构建一个字母到数字id的映射，首先需要把所有单词拆分成字母列表。可以使用`tf.strings.unicode_split`<jupyter_code>example_texts = ['abcdefg', 'xyz']

chars = tf.strings.unicode_split(example_texts, input_encoding='UTF-8')
chars<jupyter_output><empty_output><jupyter_text>可以使用vocab来构建这个字典preprocessing.StringLookup<jupyter_code>ids_from_chars = preprocessing.StringLookup(vocabulary=list(vocab))<jupyter_output><empty_output><jupyter_text>用上面的chars来试试<jupyter_code>ids=ids_from_chars(chars)
ids<jupyter_output><empty_output><jupyter_text>最终我们要将网络的输出转为文字，因此还需要一个id到字母的映射<jupyter_code>chars_from_ids = tf.keras.layers.experimental.preprocessing.StringLookup(
    vocabulary=ids_from_chars.get_vocabulary(), invert=True)
chars = chars_from_ids(ids)
chars<jupyter_output><empty_output><jupyter_text>建个函数把字母拼回去<jupyter_code>def text_from_ids(ids):
  return tf.strings.reduce_join(chars_from_ids(ids), axis=-1)
text_from_ids(ids)<jupyter_output><empty_output><jupyter_text>Build Dataset我们现在有的东西只有文章的txt。需要自己构建数据集。最终期望的效果是，给模型一个至多个起始字符，模型要给出后续出现的字符，字符组成词，词再组成句子。因为我们是字符级别的预测，所以模型的输入应当是一个字符序列，模型的输出应该也是序列，包含每次输入字符之后应该出现的字符。举个例子，模型的输入是[H,E,L,L]，那么输出应该是[E,L,L,O]。<jupyter_code>all_ids = ids_from_chars(tf.strings.unicode_split(text, 'UTF-8'))
all_ids
ids_dataset = tf.data.Dataset.from_tensor_slices(all_ids)
for ids in ids_dataset.take(10):
    print(chars_from_ids(ids).numpy().decode('utf-8'))
seq_length = 100
examples_per_epoch = len(text)//(seq_length+1)
examples_per_epoch
sequences = ids_dataset.batch(seq_length+1, drop_remainder=True)

for seq in sequences.take(1):
  print(chars_from_ids(seq))<jupyter_output>tf.Tensor(
[b'F' b'i' b'r' b's' b't' b' ' b'C' b'i' b't' b'i' b'z' b'e' b'n' b':'
 b'\n' b'B' b'e' b'f' b'o' b'r' b'e' b' ' b'w' b'e' b' ' b'p' b'r' b'o'
 b'c' b'e' b'e' b'd' b' ' b'a' b'n' b'y' b' ' b'f' b'u' b'r' b't' b'h'
 b'e' b'r' b',' b' ' b'h' b'e' b'a' b'r' b' ' b'm' b'e' b' ' b's' b'p'
 b'e' b'a' b'k' b'.' b'\n' b'\n' b'A' b'l' b'l' b':' b'\n' b'S' b'p' b'e'
 b'a' b'k' b',' b' ' b's' b'p' b'e' b'a' b'k' b'.' b'\n' b'\n' b'F' b'i'
 b'r' b's' b't' b' ' b'C' b'i' b't' b'i' b'z' b'e' b'n' b':' b'\n' b'Y'
 b'o' b'u' b' '], shape=(101,), dtype=string)<jupyter_text>构建（input, label）对<jupyter_code>def split_input_target(sequence):
    input_text = sequence[:-1]
    target_text = sequence[1:]
    return input_text, target_text
split_input_target(list("Tensorflow"))
dataset = sequences.map(split_input_target)
for input_example, target_example in  dataset.take(1):
    print("Input :", text_from_ids(input_example).numpy())
    print("Target:", text_from_ids(target_example).numpy())
# Batch size
BATCH_SIZE = 64

# Buffer size to shuffle the dataset
# (TF data is designed to work with possibly infinite sequences,
# so it doesn't attempt to shuffle the entire sequence in memory. Instead,
# it maintains a buffer in which it shuffles elements).
BUFFER_SIZE = 10000
train_ds=dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True).prefetch(tf.data.experimental.AUTOTUNE)
train_ds<jupyter_output><empty_output><jupyter_text>Build Model<jupyter_code># Length of the vocabulary in chars
vocab_size = len(vocab)

# The embedding dimension
embedding_dim = 256

# Number of RNN units
rnn_units = 1024<jupyter_output><empty_output><jupyter_text>因为网络内部有一些逻辑判断，所以这次我们没用Sequantial的方式快速构建网络，而是继承tf.keras.Model搭建网络<jupyter_code>class MyModel(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, rnn_units):
    super().__init__(self)
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(rnn_units,
                                   return_sequences=True, 
                                   return_state=True)
    self.dense = tf.keras.layers.Dense(vocab_size)

  def call(self, inputs, states=None, return_state=False, training=False):
    x = inputs
    x = self.embedding(x, training=training)
    if states is None:
      states = self.gru.get_initial_state(x)
    x, states = self.gru(x, initial_state=states, training=training)
    x = self.dense(x, training=training)

    if return_state:
      return x, states
    else: 
      return x
model = MyModel(
    # Be sure the vocabulary size matches the `StringLookup` layers.
    vocab_size=len(ids_from_chars.get_vocabulary()),
    embedding_dim=embedding_dim,
    rnn_units=rnn_units)<jupyter_output><empty_output><jupyter_text>试一下模型前馈。因为没训练，现在的输出是很混乱的。<jupyter_code>for input_example_batch, target_example_batch in train_ds.take(1):
    example_batch_predictions = model(input_example_batch)
    print(example_batch_predictions.shape, "# (batch_size, sequence_length, vocab_size)")
model.summary()
sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)
sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()
print("Input:\n", text_from_ids(input_example_batch[0]).numpy())
print()
print("Next Char Predictions:\n", text_from_ids(sampled_indices).numpy())<jupyter_output>Input:
 b' to thy cold\nbed, and warm thee.\n\nHostess:\nI know my remedy; I must go fetch the\nthird--borough.\n\nSL'

Next Char Predictions:
 b'XbW CczqomFLYY?auB!b?VdfEqwFgY\ngAQJQqawC-ZmRBZxf A3zwN[UNK]DtEIRyRXmxFkBMg?nTAphO.BHQxmstE.\nokurQGxslc'<jupyter_text>Compile Model<jupyter_code>model.compile(optimizer='adam', loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True))<jupyter_output><empty_output><jupyter_text>Training 配一个checkpoint的callback，用来在训练过程中保存每一个epoch得到的模型。这样因为意外停止训练后还可以重新load模型继续训练。<jupyter_code># Directory where the checkpoints will be saved
checkpoint_dir = './training_checkpoints'
# Name of the checkpoint files
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_prefix,
    save_weights_only=True)
EPOCHS = 20
history = model.fit(train_ds, epochs=EPOCHS, callbacks=[checkpoint_callback])<jupyter_output>Epoch 1/20
172/172 [==============================] - 11s 51ms/step - loss: 3.3362
Epoch 2/20
172/172 [==============================] - 10s 51ms/step - loss: 2.1015
Epoch 3/20
172/172 [==============================] - 10s 52ms/step - loss: 1.7782
Epoch 4/20
172/172 [==============================] - 10s 53ms/step - loss: 1.5855
Epoch 5/20
172/172 [==============================] - 10s 53ms/step - loss: 1.4677
Epoch 6/20
172/172 [==============================] - 10s 54ms/step - loss: 1.3944
Epoch 7/20
172/172 [==============================] - 10s 54ms/step - loss: 1.3369
Epoch 8/20
172/172 [==============================] - 10s 55ms/step - loss: 1.2891
Epoch 9/20
172/172 [==============================] - 10s 55ms/step - loss: 1.2452
Epoch 10/20
172/172 [==============================] - 11s 56ms/step - loss: 1.2035
Epoch 11/20
172/172 [==============================] - 11s 56ms/step - loss: 1.1616
Epoch 12/20
172/172 [==============================] - 11s 56ms/step - loss: 1.1189
E[...]<jupyter_text>Generate Text![](https://www.tensorflow.org/tutorials/text/images/text_generation_sampling.png?authuser=1)<jupyter_code>class OneStep(tf.keras.Model):
  def __init__(self, model, chars_from_ids, ids_from_chars, temperature=1.0):
    super().__init__()
    self.temperature=temperature
    self.model = model
    self.chars_from_ids = chars_from_ids
    self.ids_from_chars = ids_from_chars

    # Create a mask to prevent "" or "[UNK]" from being generated.
    skip_ids = self.ids_from_chars(['','[UNK]'])[:, None]
    sparse_mask = tf.SparseTensor(
        # Put a -inf at each bad index.
        values=[-float('inf')]*len(skip_ids),
        indices = skip_ids,
        # Match the shape to the vocabulary
        dense_shape=[len(ids_from_chars.get_vocabulary())]) 
    self.prediction_mask = tf.sparse.to_dense(sparse_mask)

  @tf.function
  def generate_one_step(self, inputs, states=None):
    # Convert strings to token IDs.
    input_chars = tf.strings.unicode_split(inputs, 'UTF-8')
    input_ids = self.ids_from_chars(input_chars).to_tensor()

    # Run the model.
    # predicted_logits.shape is [batch, char, next_char_logits] 
    predicted_logits, states =  self.model(inputs=input_ids, states=states, 
                                          return_state=True)
    # Only use the last prediction.
    predicted_logits = predicted_logits[:, -1, :]
    predicted_logits = predicted_logits/self.temperature
    # Apply the prediction mask: prevent "" or "[UNK]" from being generated.
    predicted_logits = predicted_logits + self.prediction_mask

    # Sample the output logits to generate token IDs.
    predicted_ids = tf.random.categorical(predicted_logits, num_samples=1)
    predicted_ids = tf.squeeze(predicted_ids, axis=-1)

    # Convert from token ids to characters
    predicted_chars = self.chars_from_ids(predicted_ids)

    # Return the characters and model state.
    return predicted_chars, states
one_step_model = OneStep(model, chars_from_ids, ids_from_chars)
start = time.time()
states = None
next_char = tf.constant(['ROMEO:'])
result = [next_char]

for n in range(1000):
  next_char, states = one_step_model.generate_one_step(next_char, states=states)
  result.append(next_char)

result = tf.strings.join(result)
end = time.time()

print(result[0].numpy().decode('utf-8'), '\n\n' + '_'*80)

print(f"\nRun time: {end - start}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1670024174053183, 'fraction_numerical': 0.028605962933118453, 'mean_word_length': 5.52365308804205, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '630498', 'license': 'MIT', 'n_tokens_mistral': 4160, 'n_tokens_neox': 3970, 'n_words': 670, 'path': 'notebooks/4_text_generation.ipynb', 'repo_name': 'nofacer/TensorCertificate', 'size': '9928'}