<jupyter_start><jupyter_text>训练样本的创建将整个文本拆分成 `seq_length+1` 个字符的文本块，每个输入序列和目标序列均有 `seq_length` 个字符。目标序列为输入序列向右顺移一个字符。文本块 "Hello",输入序列 "Hell",目标序列 "ello".<jupyter_code>seq_length = 100

text_as_int_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
text_as_int_dataset = text_as_int_dataset.batch(seq_length+1, drop_remainder=True)
text_as_int_dataset
def split_input_target(chunk):
    """
    ("Hello") -> ("Hell", "ello")
    """
    return chunk[:-1], chunk[1:]

dataset = text_as_int_dataset.map(split_input_target)
dataset
dataset = dataset.shuffle(10000).batch(64, drop_remainder=True)
dataset
# 嵌入的维度
embedding_dimension = 256

# RNN 单元数量
rnn_units = 1024

model = keras.Sequential([
    keras.layers.Embedding(len(vocab), embedding_dimension,
                           batch_input_shape=[64, None]),
    keras.layers.GRU(rnn_units,
                     return_sequences=True,
                     stateful=True,
                     recurrent_initializer='glorot_uniform'),
    keras.layers.Dense(len(vocab))
])

model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (64, None, 256)           16640     
_________________________________________________________________
gru (GRU)                    (64, None, 1024)          3938304   
_________________________________________________________________
dense (Dense)                (64, None, 65)            66625     
=================================================================
Total params: 4,021,569
Trainable params: 4,021,569
Non-trainable params: 0
_________________________________________________________________<jupyter_text>![text generation RNN](text_generation_training.png)<jupyter_code>def loss(labels, logits):
    # 因为我们的模型返回逻辑回归，所以我们需要设定命令行参数 from_logits
    return keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)

model.compile(optimizer='adam', loss=loss)
history = model.fit(dataset, epochs=10)
model.save_weights('{}.h5'.format(datetime.datetime.now().strftime('%m-%d-%H-%M')))
for input_example_batch, target_example_batch in dataset.take(1):
  print(input_example_batch)
  example_batch_predictions = model(input_example_batch)
  print(example_batch_predictions.shape, "# (batch_size, sequence_length, vocab_size)")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15156375300721733, 'fraction_numerical': 0.02967121090617482, 'mean_word_length': 4.884433962264151, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '189885', 'license': 'MIT', 'n_tokens_mistral': 875, 'n_tokens_neox': 821, 'n_words': 111, 'path': 'Tensorflow_official/RNN/text_generation/train.ipynb', 'repo_name': 'starkidstory/OmegaTensor', 'size': '2494'}