<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "K2s1A9eLRPEj"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "VRLVEKiTEn04"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Cffg2i257iMS"
      },
      "source": [
        "# 使用视觉注意力生成图像描述\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/text/image_captioning\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" class=\"\">在 TensorFlow.org 上查看</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/text/image_captioning.ipynb\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" class=\"\">在 Google Colab 中运行</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/text/image_captioning.ipynb\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" class=\"\">在 GitHub 上查看源代码</a></td>\n",
        "  <td>\n",
        "<img src=\"https://www.tensorflow.org/images/download_logo_32px.png\"><a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/image_captioning.ipynb\">Download notebook</a>\n",
        "</td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QASbY_HGo4Lq"
      },
      "source": [
        "给定一个类似以下示例的图像，我们的目标是生成一个类似“a surfer riding on a wave（一名正在冲浪的冲浪者）”的描述。\n",
        "\n",
        "![Man Surfing](https://tensorflow.org/images/surf.jpg)\n",
        "\n",
        "*[图像来源](https://commons.wikimedia.org/wiki/Surfing#/media/File:Surfing_in_Hawaii.jpg)；许可：公共领域*\n",
        "\n",
        "为此，您将使用基于注意力的模型，它使我们能够看到在生成描述时模型聚焦在图像的哪个部分。\n",
        "\n",
        "![Prediction](https://tensorflow.org/images/imcap_prediction.png)\n",
        "\n",
        "此模型架构类似于论文 [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://arxiv.org/abs/1502.03044)。\n",
        "\n",
        "此笔记本是一个端到端示例。当您运行此笔记本时，它会下载 [MS-COCO](http://cocodataset.org/#home) 数据集、使用 Inception V3 缓存图像的子集、训练编解码器模型，并使用训练后的模型在新的图像上生成描述。\n",
        "\n",
        "在此示例中，您将在相对较少的数据上训练模型——对应约 20,000 个图像的前 30,000 个描述（因为数据集中的每个图像有多个描述）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "U8l4RJ0XRPEm"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# You'll generate plots of attention in order to see which parts of an image\n",
        "# our model focuses on during captioning\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Scikit-learn includes many helpful utilities\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.utils import shuffle\n",
        "\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import time\n",
        "import json\n",
        "from glob import glob\n",
        "from PIL import Image\n",
        "import pickle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "b6qbGw8MRPE5"
      },
      "source": [
        "## 下载并准备 MS-COCO 数据集\n",
        "\n",
        "您将使用 [MS-COCO 数据集](http://cocodataset.org/#home)来训练模型。此数据集包含了超过 82,000 个图像，每个图像至少具有 5 个不同的描述注解。下面的代码将自动下载并提取数据集。\n",
        "\n",
        "**小心：前方含有大量下载**。您将使用的训练集包含 13GB 的文件。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "krQuPYTtRPE7"
      },
      "outputs": [],
      "source": [
        "# Download caption annotation files\n",
        "annotation_folder = '/annotations/'\n",
        "if not os.path.exists(os.path.abspath('.') + annotation_folder):\n",
        "  annotation_zip = tf.keras.utils.get_file('captions.zip',\n",
        "                                          cache_subdir=os.path.abspath('.'),\n",
        "                                          origin = 'http://images.cocodataset.org/annotations/annotations_trainval2014.zip',\n",
        "                                          extract = True)\n",
        "  annotation_file = os.path.dirname(annotation_zip)+'/annotations/captions_train2014.json'\n",
        "  os.remove(annotation_zip)\n",
        "\n",
        "# Download image files\n",
        "image_folder = '/train2014/'\n",
        "if not os.path.exists(os.path.abspath('.') + image_folder):\n",
        "  image_zip = tf.keras.utils.get_file('train2014.zip',\n",
        "                                      cache_subdir=os.path.abspath('.'),\n",
        "                                      origin = 'http://images.cocodataset.org/zips/train2014.zip',\n",
        "                                      extract = True)\n",
        "  PATH = os.path.dirname(image_zip) + image_folder\n",
        "  os.remove(image_zip)\n",
        "else:\n",
        "  PATH = os.path.abspath('.') + image_folder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aANEzb5WwSzg"
      },
      "source": [
        "## 可选：限制训练集的大小\n",
        "\n",
        "为了加快此教程的训练速度，您将使用 30,000 个描述及其对应图像的子集来训练模型。选择使用更多数据将提高生成描述的质量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4G3b8x8_RPFD"
      },
      "outputs": [],
      "source": [
        "# Read the json file\n",
        "with open(annotation_file, 'r') as f:\n",
        "    annotations = json.load(f)\n",
        "\n",
        "# Store captions and image names in vectors\n",
        "all_captions = []\n",
        "all_img_name_vector = []\n",
        "\n",
        "for annot in annotations['annotations']:\n",
        "    caption = '&lt;start&gt; ' + annot['caption'] + ' &lt;end&gt;'\n",
        "    image_id = annot['image_id']\n",
        "    full_coco_image_path = PATH + 'COCO_train2014_' + '%012d.jpg' % (image_id)\n",
        "\n",
        "    all_img_name_vector.append(full_coco_image_path)\n",
        "    all_captions.append(caption)\n",
        "\n",
        "# Shuffle captions and image_names together\n",
        "# Set a random state\n",
        "train_captions, img_name_vector = shuffle(all_captions,\n",
        "                                          all_img_name_vector,\n",
        "                                          random_state=1)\n",
        "\n",
        "# Select the first 30000 captions from the shuffled set\n",
        "num_examples = 30000\n",
        "train_captions = train_captions[:num_examples]\n",
        "img_name_vector = img_name_vector[:num_examples]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mPBMgK34RPFL"
      },
      "outputs": [],
      "source": [
        "len(train_captions), len(all_captions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8cSW4u-ORPFQ"
      },
      "source": [
        "## 使用 InceptionV3 预处理图像\n",
        "\n",
        "接下来，您将使用 InceptionV3（已在 Imagenet 上进行了预训练）对每个图像进行分类。您将从最后一个卷积层中提取特征。\n",
        "\n",
        "首先，您需要将图像转换为 InceptionV3 的预期格式：\n",
        "\n",
        "- 将图像大小调整为 299 x 299 像素\n",
        "- 使用 [preprocess_input](https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_v3/preprocess_input) 方法[预处理图像](https://cloud.google.com/tpu/docs/inception-v3-advanced#preprocessing_stage)，以对其进行归一化，以便使其包含范围在 -1 和 1 内的像素，从而与用于训练 InceptionV3 的图像格式相匹配。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zXR0217aRPFR"
      },
      "outputs": [],
      "source": [
        "def load_image(image_path):\n",
        "    img = tf.io.read_file(image_path)\n",
        "    img = tf.image.decode_jpeg(img, channels=3)\n",
        "    img = tf.image.resize(img, (299, 299))\n",
        "    img = tf.keras.applications.inception_v3.preprocess_input(img)\n",
        "    return img, image_path"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MDvIu4sXRPFV"
      },
      "source": [
        "## 初始化 InceptionV3 并加载经过预训练的 Imagenet 权重\n",
        "\n",
        "现在，您将创建一个 tf.keras 模型，其中输出层是 InceptionV3 架构中的最后一个卷积层。此层的输出形状为 `8x8x2048`。之所以使用最后一个卷积层，是因为我们在此示例中使用了注意力。您无法在训练期间执行此初始化，因为它可能会成为瓶颈。\n",
        "\n",
        "- 您将通过网络对每个图像进行前向传递，并将得到的向量存储在字典中（image_name --> feature_vector）。\n",
        "- 当通过网络传递所有图像后，对字典进行 pickle 操作，然后将其保存到磁盘。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RD3vW4SsRPFW"
      },
      "outputs": [],
      "source": [
        "image_model = tf.keras.applications.InceptionV3(include_top=False,\n",
        "                                                weights='imagenet')\n",
        "new_input = image_model.input\n",
        "hidden_layer = image_model.layers[-1].output\n",
        "\n",
        "image_features_extract_model = tf.keras.Model(new_input, hidden_layer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rERqlR3WRPGO"
      },
      "source": [
        "## 缓存从 InceptionV3 中提取的特征\n",
        "\n",
        "您将使用 InceptionV3 预处理每个图像，然后将输出缓存到磁盘。将输出缓存到 RAM 会更快，但也会占用大量内存（每个图像需要 8 * 8 * 2048 个浮点数）。写入时，这将超出 Colab 的内存限制（目前为 12GB 内存）。\n",
        "\n",
        "使用更复杂的缓存策略（例如，将图像分片来减少对磁盘 I/O 的随机访问）可以提高性能，但这需要编写更多代码。\n",
        "\n",
        "使用 GPU 在 Colab 中运行缓存大约需要 10 分钟。如果您想查看进度条，可以进行以下操作：\n",
        "\n",
        "1. 安装 [tqdm](https://github.com/tqdm/tqdm)：\n",
        "\n",
        "    `!pip install tqdm`\n",
        "\n",
        "2. 导入 tqdm：\n",
        "\n",
        "    `from tqdm import tqdm`\n",
        "\n",
        "3. 将下面的行：\n",
        "\n",
        "    `for img, path in image_dataset:`\n",
        "\n",
        "    改为：\n",
        "\n",
        "    `for img, path in tqdm(image_dataset):`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Dx_fvbVgRPGQ"
      },
      "outputs": [],
      "source": [
        "# Get unique images\n",
        "encode_train = sorted(set(img_name_vector))\n",
        "\n",
        "# Feel free to change batch_size according to your system configuration\n",
        "image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)\n",
        "image_dataset = image_dataset.map(\n",
        "  load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE).batch(16)\n",
        "\n",
        "for img, path in image_dataset:\n",
        "  batch_features = image_features_extract_model(img)\n",
        "  batch_features = tf.reshape(batch_features,\n",
        "                              (batch_features.shape[0], -1, batch_features.shape[3]))\n",
        "\n",
        "  for bf, p in zip(batch_features, path):\n",
        "    path_of_feature = p.numpy().decode(\"utf-8\")\n",
        "    np.save(path_of_feature, bf.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nyqH3zFwRPFi"
      },
      "source": [
        "## 预处理描述并进行分词\n",
        "\n",
        "- 首先，您需要对描述进行分词（例如，按照空格进行切分）。这会为我们提供一个包含数据中的所有唯一单词的词汇表（例如，“surfing”、“football”等）。\n",
        "- 接下来，将词汇表的大小限制为前 5,000 个单词（以节省内存）。将所有其他单词替换为“UNK”（未知）。\n",
        "- 然后，创建“词到索引”和“索引到词”的映射。\n",
        "- 最后，将所有序列填充为与最长序列相同的长度。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HZfK8RhQRPFj"
      },
      "outputs": [],
      "source": [
        "# Find the maximum length of any caption in our dataset\n",
        "def calc_max_length(tensor):\n",
        "    return max(len(t) for t in tensor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "oJGE34aiRPFo"
      },
      "outputs": [],
      "source": [
        "# Choose the top 5000 words from the vocabulary\n",
        "top_k = 5000\n",
        "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,\n",
        "                                                  oov_token=\"&lt;unk&gt;\",\n",
        "                                                  filters='!\"#$%&amp;()*+.,-/:;=?@[\\]^_`{|}~ ')\n",
        "tokenizer.fit_on_texts(train_captions)\n",
        "train_seqs = tokenizer.texts_to_sequences(train_captions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "8Q44tNQVRPFt"
      },
      "outputs": [],
      "source": [
        "tokenizer.word_index['&lt;pad&gt;'] = 0\n",
        "tokenizer.index_word[0] = '&lt;pad&gt;'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0fpJb5ojRPFv"
      },
      "outputs": [],
      "source": [
        "# Create the tokenized vectors\n",
        "train_seqs = tokenizer.texts_to_sequences(train_captions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AidglIZVRPF4"
      },
      "outputs": [],
      "source": [
        "# Pad each vector to the max_length of the captions\n",
        "# If you do not provide a max_length value, pad_sequences calculates it automatically\n",
        "cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gL0wkttkRPGA"
      },
      "outputs": [],
      "source": [
        "# Calculates the max_length, which is used to store the attention weights\n",
        "max_length = calc_max_length(train_seqs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "M3CD75nDpvTI"
      },
      "source": [
        "## 将数据拆分为训练和测试"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "iS7DDMszRPGF"
      },
      "outputs": [],
      "source": [
        "# Create training and validation sets using an 80-20 split\n",
        "img_name_train, img_name_val, cap_train, cap_val = train_test_split(img_name_vector,\n",
        "                                                                    cap_vector,\n",
        "                                                                    test_size=0.2,\n",
        "                                                                    random_state=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XmViPkRFRPGH"
      },
      "outputs": [],
      "source": [
        "len(img_name_train), len(cap_train), len(img_name_val), len(cap_val)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uEWM9xrYcg45"
      },
      "source": [
        "## 创建用于训练的 tf.data 数据集\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "horagNvhhZiy"
      },
      "source": [
        "我们的图像和描述已准备就绪！接下来，让我们创建一个 tf.data 数据集，用于训练我们的模型。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Q3TnZ1ToRPGV"
      },
      "outputs": [],
      "source": [
        "# Feel free to change these parameters according to your system's configuration\n",
        "\n",
        "BATCH_SIZE = 64\n",
        "BUFFER_SIZE = 1000\n",
        "embedding_dim = 256\n",
        "units = 512\n",
        "vocab_size = top_k + 1\n",
        "num_steps = len(img_name_train) // BATCH_SIZE\n",
        "# Shape of the vector extracted from InceptionV3 is (64, 2048)\n",
        "# These two variables represent that vector shape\n",
        "features_shape = 2048\n",
        "attention_features_shape = 64"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "SmZS2N0bXG3T"
      },
      "outputs": [],
      "source": [
        "# Load the numpy files\n",
        "def map_func(img_name, cap):\n",
        "  img_tensor = np.load(img_name.decode('utf-8')+'.npy')\n",
        "  return img_tensor, cap"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FDF_Nm3tRPGZ"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))\n",
        "\n",
        "# Use map to load the numpy files in parallel\n",
        "dataset = dataset.map(lambda item1, item2: tf.numpy_function(\n",
        "          map_func, [item1, item2], [tf.float32, tf.int32]),\n",
        "          num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "# Shuffle and batch\n",
        "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n",
        "dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nrvoDphgRPGd"
      },
      "source": [
        "## 模型\n",
        "\n",
        "有趣的事实：下面的解码器与[使用注意力机制的神经网络机器翻译](../sequences/nmt_with_attention.ipynb)示例中的解码器相同。\n",
        "\n",
        "此模型架构的灵感来自论文 [Show, Attend and Tell](https://arxiv.org/pdf/1502.03044.pdf)。\n",
        "\n",
        "- 在此示例中，您将从 InceptionV3 的较低卷积层中提取特征，从而得到一个形状为 (8, 8, 2048) 的向量。\n",
        "- 将上述形状挤压为 (64, 2048) 的形状。\n",
        "- 然后通过 CNN 编码器（包含一个全连接层）传递此向量。\n",
        "- RNN（此处为 GRU）会将注意力放在图像上以预测下一个单词。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ja2LFTMSdeV3"
      },
      "outputs": [],
      "source": [
        "class BahdanauAttention(tf.keras.Model):\n",
        "  def __init__(self, units):\n",
        "    super(BahdanauAttention, self).__init__()\n",
        "    self.W1 = tf.keras.layers.Dense(units)\n",
        "    self.W2 = tf.keras.layers.Dense(units)\n",
        "    self.V = tf.keras.layers.Dense(1)\n",
        "\n",
        "  def call(self, features, hidden):\n",
        "    # features(CNN_encoder output) shape == (batch_size, 64, embedding_dim)\n",
        "\n",
        "    # hidden shape == (batch_size, hidden_size)\n",
        "    # hidden_with_time_axis shape == (batch_size, 1, hidden_size)\n",
        "    hidden_with_time_axis = tf.expand_dims(hidden, 1)\n",
        "\n",
        "    # score shape == (batch_size, 64, hidden_size)\n",
        "    score = tf.nn.tanh(self.W1(features) + self.W2(hidden_with_time_axis))\n",
        "\n",
        "    # attention_weights shape == (batch_size, 64, 1)\n",
        "    # you get 1 at the last axis because you are applying score to self.V\n",
        "    attention_weights = tf.nn.softmax(self.V(score), axis=1)\n",
        "\n",
        "    # context_vector shape after sum == (batch_size, hidden_size)\n",
        "    context_vector = attention_weights * features\n",
        "    context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "    return context_vector, attention_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AZ7R1RxHRPGf"
      },
      "outputs": [],
      "source": [
        "class CNN_Encoder(tf.keras.Model):\n",
        "    # Since you have already extracted the features and dumped it using pickle\n",
        "    # This encoder passes those features through a Fully connected layer\n",
        "    def __init__(self, embedding_dim):\n",
        "        super(CNN_Encoder, self).__init__()\n",
        "        # shape after fc == (batch_size, 64, embedding_dim)\n",
        "        self.fc = tf.keras.layers.Dense(embedding_dim)\n",
        "\n",
        "    def call(self, x):\n",
        "        x = self.fc(x)\n",
        "        x = tf.nn.relu(x)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "V9UbGQmERPGi"
      },
      "outputs": [],
      "source": [
        "class RNN_Decoder(tf.keras.Model):\n",
        "  def __init__(self, embedding_dim, units, vocab_size):\n",
        "    super(RNN_Decoder, self).__init__()\n",
        "    self.units = units\n",
        "\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc1 = tf.keras.layers.Dense(self.units)\n",
        "    self.fc2 = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "    self.attention = BahdanauAttention(self.units)\n",
        "\n",
        "  def call(self, x, features, hidden):\n",
        "    # defining attention as a separate model\n",
        "    context_vector, attention_weights = self.attention(features, hidden)\n",
        "\n",
        "    # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
        "    x = self.embedding(x)\n",
        "\n",
        "    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
        "    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
        "\n",
        "    # passing the concatenated vector to the GRU\n",
        "    output, state = self.gru(x)\n",
        "\n",
        "    # shape == (batch_size, max_length, hidden_size)\n",
        "    x = self.fc1(output)\n",
        "\n",
        "    # x shape == (batch_size * max_length, hidden_size)\n",
        "    x = tf.reshape(x, (-1, x.shape[2]))\n",
        "\n",
        "    # output shape == (batch_size * max_length, vocab)\n",
        "    x = self.fc2(x)\n",
        "\n",
        "    return x, state, attention_weights\n",
        "\n",
        "  def reset_state(self, batch_size):\n",
        "    return tf.zeros((batch_size, self.units))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Qs_Sr03wRPGk"
      },
      "outputs": [],
      "source": [
        "encoder = CNN_Encoder(embedding_dim)\n",
        "decoder = RNN_Decoder(embedding_dim, units, vocab_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-bYN7xA0RPGl"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6A3Ni64joyab"
      },
      "source": [
        "## 检查点"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "PpJAqPMWo0uE"
      },
      "outputs": [],
      "source": [
        "checkpoint_path = \"./checkpoints/train\"\n",
        "ckpt = tf.train.Checkpoint(encoder=encoder,\n",
        "                           decoder=decoder,\n",
        "                           optimizer = optimizer)\n",
        "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fUkbqhc_uObw"
      },
      "outputs": [],
      "source": [
        "start_epoch = 0\n",
        "if ckpt_manager.latest_checkpoint:\n",
        "  start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])\n",
        "  # restoring the latest checkpoint in checkpoint_path\n",
        "  ckpt.restore(ckpt_manager.latest_checkpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PHod7t72RPGn"
      },
      "source": [
        "## 训练\n",
        "\n",
        "- 提取存储在相应 `.npy` 文件中的特征，然后通过编码器传递这些特征。\n",
        "- 将编码器输出、隐藏状态（已初始化为 0）和解码器输入（即起始词例）传递给解码器。\n",
        "- 解码器会返回预测和解码器隐藏状态。\n",
        "- 然后将解码器隐藏状态传回模型，并使用预测计算损失。\n",
        "- 使用 Teacher Forcing 方法决定解码器的下一个输入。\n",
        "- Teacher Forcing 是一种将目标词作为下一个输入传递给解码器的技术。\n",
        "- 最后一步是计算梯度并将其应用于优化器和反向传播。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Vt4WZ5mhJE-E"
      },
      "outputs": [],
      "source": [
        "# adding this in a separate cell because if you run the training cell\n",
        "# many times, the loss_plot array will be reset\n",
        "loss_plot = []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sqgyz2ANKlpU"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_step(img_tensor, target):\n",
        "  loss = 0\n",
        "\n",
        "  # initializing the hidden state for each batch\n",
        "  # because the captions are not related from image to image\n",
        "  hidden = decoder.reset_state(batch_size=target.shape[0])\n",
        "\n",
        "  dec_input = tf.expand_dims([tokenizer.word_index['&lt;start&gt;']] * target.shape[0], 1)\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "      features = encoder(img_tensor)\n",
        "\n",
        "      for i in range(1, target.shape[1]):\n",
        "          # passing the features through the decoder\n",
        "          predictions, hidden, _ = decoder(dec_input, features, hidden)\n",
        "\n",
        "          loss += loss_function(target[:, i], predictions)\n",
        "\n",
        "          # using teacher forcing\n",
        "          dec_input = tf.expand_dims(target[:, i], 1)\n",
        "\n",
        "  total_loss = (loss / int(target.shape[1]))\n",
        "\n",
        "  trainable_variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, trainable_variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, trainable_variables))\n",
        "\n",
        "  return loss, total_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "UlA4VIQpRPGo"
      },
      "outputs": [],
      "source": [
        "EPOCHS = 20\n",
        "\n",
        "for epoch in range(start_epoch, EPOCHS):\n",
        "    start = time.time()\n",
        "    total_loss = 0\n",
        "\n",
        "    for (batch, (img_tensor, target)) in enumerate(dataset):\n",
        "        batch_loss, t_loss = train_step(img_tensor, target)\n",
        "        total_loss += t_loss\n",
        "\n",
        "        if batch % 100 == 0:\n",
        "            print ('Epoch {} Batch {} Loss {:.4f}'.format(\n",
        "              epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))\n",
        "    # storing the epoch end loss value to plot later\n",
        "    loss_plot.append(total_loss / num_steps)\n",
        "\n",
        "    if epoch % 5 == 0:\n",
        "      ckpt_manager.save()\n",
        "\n",
        "    print ('Epoch {} Loss {:.6f}'.format(epoch + 1,\n",
        "                                         total_loss/num_steps))\n",
        "    print ('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1Wm83G-ZBPcC"
      },
      "outputs": [],
      "source": [
        "plt.plot(loss_plot)\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.title('Loss Plot')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xGvOcLQKghXN"
      },
      "source": [
        "## 说明！\n",
        "\n",
        "- 评估函数与训练循环类似，不同之处在于，此处不使用 Teacher Forcing。每一个时间步骤的解码器输入都是其前一步的预测结果、隐藏状态和编码器输出。\n",
        "- 当模型预测到最后一个词例时停止。\n",
        "- 存储每一个时间步骤的注意力权重。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RCWpDtyNRPGs"
      },
      "outputs": [],
      "source": [
        "def evaluate(image):\n",
        "    attention_plot = np.zeros((max_length, attention_features_shape))\n",
        "\n",
        "    hidden = decoder.reset_state(batch_size=1)\n",
        "\n",
        "    temp_input = tf.expand_dims(load_image(image)[0], 0)\n",
        "    img_tensor_val = image_features_extract_model(temp_input)\n",
        "    img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))\n",
        "\n",
        "    features = encoder(img_tensor_val)\n",
        "\n",
        "    dec_input = tf.expand_dims([tokenizer.word_index['&lt;start&gt;']], 0)\n",
        "    result = []\n",
        "\n",
        "    for i in range(max_length):\n",
        "        predictions, hidden, attention_weights = decoder(dec_input, features, hidden)\n",
        "\n",
        "        attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()\n",
        "\n",
        "        predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()\n",
        "        result.append(tokenizer.index_word[predicted_id])\n",
        "\n",
        "        if tokenizer.index_word[predicted_id] == '&lt;end&gt;':\n",
        "            return result, attention_plot\n",
        "\n",
        "        dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "    attention_plot = attention_plot[:len(result), :]\n",
        "    return result, attention_plot"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fD_y7PD6RPGt"
      },
      "outputs": [],
      "source": [
        "def plot_attention(image, result, attention_plot):\n",
        "    temp_image = np.array(Image.open(image))\n",
        "\n",
        "    fig = plt.figure(figsize=(10, 10))\n",
        "\n",
        "    len_result = len(result)\n",
        "    for l in range(len_result):\n",
        "        temp_att = np.resize(attention_plot[l], (8, 8))\n",
        "        ax = fig.add_subplot(len_result//2, len_result//2, l+1)\n",
        "        ax.set_title(result[l])\n",
        "        img = ax.imshow(temp_image)\n",
        "        ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7x8RiPHe_4qI"
      },
      "outputs": [],
      "source": [
        "# captions on the validation set\n",
        "rid = np.random.randint(0, len(img_name_val))\n",
        "image = img_name_val[rid]\n",
        "real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])\n",
        "result, attention_plot = evaluate(image)\n",
        "\n",
        "print ('Real Caption:', real_caption)\n",
        "print ('Prediction Caption:', ' '.join(result))\n",
        "plot_attention(image, result, attention_plot)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Rprk3HEvZuxb"
      },
      "source": [
        "## 在自己的图像上进行尝试\n",
        "\n",
        "为了增加趣味性，我们在下面提供了一个方法，可让您通过我们刚才训练的模型为您自己的图像生成描述。请记住，这个模型是使用较少数据训练的，而您的图像可能与训练数据不同（因此请做好心理准备，您可能会得到奇怪的结果）。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9Psd1quzaAWg"
      },
      "outputs": [],
      "source": [
        "image_url = 'https://tensorflow.org/images/surf.jpg'\n",
        "image_extension = image_url[-4:]\n",
        "image_path = tf.keras.utils.get_file('image'+image_extension,\n",
        "                                     origin=image_url)\n",
        "\n",
        "result, attention_plot = evaluate(image_path)\n",
        "print ('Prediction Caption:', ' '.join(result))\n",
        "plot_attention(image_path, result, attention_plot)\n",
        "# opening the image\n",
        "Image.open(image_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VJZXyJco6uLO"
      },
      "source": [
        "# 后续步骤\n",
        "\n",
        "恭喜您！您刚刚使用注意力机制训练了一个图像描述生成模型。接下来，您可以看看这个示例：[使用注意力机制的神经网络机器翻译](../sequences/nmt_with_attention.ipynb)。它使用了类似架构对西班牙语和英语的句子进行互译。您还可以尝试在不同的数据集上训练此笔记本中的代码。"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "image_captioning.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
<commit_msg>Translate image_captioning.ipynb via GitLocalize<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "K2s1A9eLRPEj"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "VRLVEKiTEn04"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Cffg2i257iMS"
      },
      "source": [
        "# 使用视觉注意力生成图像描述\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/text/image_captioning\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" class=\"\">在 TensorFlow.org 上查看</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/text/image_captioning.ipynb\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" class=\"\">在 Google Colab 中运行</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/text/image_captioning.ipynb\" class=\"\">     <img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" class=\"\">在 GitHub 上查看源代码</a></td>\n",
        "  <td> <img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\" class=\"\"><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/tutorials/text/image_captioning.ipynb\" class=\"\">下载笔记本</a> </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QASbY_HGo4Lq"
      },
      "source": [
        "给定一个类似以下示例的图像，我们的目标是生成一个类似“a surfer riding on a wave（一名正在冲浪的冲浪者）”的描述。\n",
        "\n",
        "![Man Surfing](https://tensorflow.org/images/surf.jpg)\n",
        "\n",
        "*[图像来源](https://commons.wikimedia.org/wiki/Surfing#/media/File:Surfing_in_Hawaii.jpg)；许可：公共领域*\n",
        "\n",
        "为此，您将使用基于注意力的模型，它使我们能够看到在生成描述时模型聚焦在图像的哪个部分。\n",
        "\n",
        "![Prediction](https://tensorflow.org/images/imcap_prediction.png)\n",
        "\n",
        "此模型架构类似于论文 [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://arxiv.org/abs/1502.03044)。\n",
        "\n",
        "此笔记本是一个端到端示例。当您运行此笔记本时，它会下载 [MS-COCO](http://cocodataset.org/#home) 数据集、使用 Inception V3 缓存图像的子集、训练编解码器模型，并使用训练后的模型在新的图像上生成描述。\n",
        "\n",
        "在此示例中，您将在相对较少的数据上训练模型——对应约 20,000 个图像的前 30,000 个描述（因为数据集中的每个图像有多个描述）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "U8l4RJ0XRPEm"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# You'll generate plots of attention in order to see which parts of an image\n",
        "# our model focuses on during captioning\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Scikit-learn includes many helpful utilities\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.utils import shuffle\n",
        "\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import time\n",
        "import json\n",
        "from glob import glob\n",
        "from PIL import Image\n",
        "import pickle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "b6qbGw8MRPE5"
      },
      "source": [
        "## 下载并准备 MS-COCO 数据集\n",
        "\n",
        "您将使用 [MS-COCO 数据集](http://cocodataset.org/#home)来训练模型。此数据集包含了超过 82,000 个图像，每个图像至少具有 5 个不同的描述注解。下面的代码将自动下载并提取数据集。\n",
        "\n",
        "**小心：前方含有大量下载**。您将使用的训练集包含 13GB 的文件。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "krQuPYTtRPE7"
      },
      "outputs": [],
      "source": [
        "# Download caption annotation files\n",
        "annotation_folder = '/annotations/'\n",
        "if not os.path.exists(os.path.abspath('.') + annotation_folder):\n",
        "  annotation_zip = tf.keras.utils.get_file('captions.zip',\n",
        "                                          cache_subdir=os.path.abspath('.'),\n",
        "                                          origin = 'http://images.cocodataset.org/annotations/annotations_trainval2014.zip',\n",
        "                                          extract = True)\n",
        "  annotation_file = os.path.dirname(annotation_zip)+'/annotations/captions_train2014.json'\n",
        "  os.remove(annotation_zip)\n",
        "\n",
        "# Download image files\n",
        "image_folder = '/train2014/'\n",
        "if not os.path.exists(os.path.abspath('.') + image_folder):\n",
        "  image_zip = tf.keras.utils.get_file('train2014.zip',\n",
        "                                      cache_subdir=os.path.abspath('.'),\n",
        "                                      origin = 'http://images.cocodataset.org/zips/train2014.zip',\n",
        "                                      extract = True)\n",
        "  PATH = os.path.dirname(image_zip) + image_folder\n",
        "  os.remove(image_zip)\n",
        "else:\n",
        "  PATH = os.path.abspath('.') + image_folder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aANEzb5WwSzg"
      },
      "source": [
        "## 可选：限制训练集的大小\n",
        "\n",
        "为了加快此教程的训练速度，您将使用 30,000 个描述及其对应图像的子集来训练模型。选择使用更多数据将提高生成描述的质量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4G3b8x8_RPFD"
      },
      "outputs": [],
      "source": [
        "# Read the json file\n",
        "with open(annotation_file, 'r') as f:\n",
        "    annotations = json.load(f)\n",
        "\n",
        "# Store captions and image names in vectors\n",
        "all_captions = []\n",
        "all_img_name_vector = []\n",
        "\n",
        "for annot in annotations['annotations']:\n",
        "    caption = '&lt;start&gt; ' + annot['caption'] + ' &lt;end&gt;'\n",
        "    image_id = annot['image_id']\n",
        "    full_coco_image_path = PATH + 'COCO_train2014_' + '%012d.jpg' % (image_id)\n",
        "\n",
        "    all_img_name_vector.append(full_coco_image_path)\n",
        "    all_captions.append(caption)\n",
        "\n",
        "# Shuffle captions and image_names together\n",
        "# Set a random state\n",
        "train_captions, img_name_vector = shuffle(all_captions,\n",
        "                                          all_img_name_vector,\n",
        "                                          random_state=1)\n",
        "\n",
        "# Select the first 30000 captions from the shuffled set\n",
        "num_examples = 30000\n",
        "train_captions = train_captions[:num_examples]\n",
        "img_name_vector = img_name_vector[:num_examples]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mPBMgK34RPFL"
      },
      "outputs": [],
      "source": [
        "len(train_captions), len(all_captions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8cSW4u-ORPFQ"
      },
      "source": [
        "## 使用 InceptionV3 预处理图像\n",
        "\n",
        "接下来，您将使用 InceptionV3（已在 Imagenet 上进行了预训练）对每个图像进行分类。您将从最后一个卷积层中提取特征。\n",
        "\n",
        "首先，您需要将图像转换为 InceptionV3 的预期格式：\n",
        "\n",
        "- 将图像大小调整为 299 x 299 像素\n",
        "- 使用 [preprocess_input](https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_v3/preprocess_input) 方法[预处理图像](https://cloud.google.com/tpu/docs/inception-v3-advanced#preprocessing_stage)，以对其进行归一化，以便使其包含范围在 -1 和 1 内的像素，从而与用于训练 InceptionV3 的图像格式相匹配。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zXR0217aRPFR"
      },
      "outputs": [],
      "source": [
        "def load_image(image_path):\n",
        "    img = tf.io.read_file(image_path)\n",
        "    img = tf.image.decode_jpeg(img, channels=3)\n",
        "    img = tf.image.resize(img, (299, 299))\n",
        "    img = tf.keras.applications.inception_v3.preprocess_input(img)\n",
        "    return img, image_path"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MDvIu4sXRPFV"
      },
      "source": [
        "## 初始化 InceptionV3 并加载经过预训练的 Imagenet 权重\n",
        "\n",
        "现在，您将创建一个 tf.keras 模型，其中输出层是 InceptionV3 架构中的最后一个卷积层。此层的输出形状为 `8x8x2048`。之所以使用最后一个卷积层，是因为我们在此示例中使用了注意力。您无法在训练期间执行此初始化，因为它可能会成为瓶颈。\n",
        "\n",
        "- 您将通过网络对每个图像进行前向传递，并将得到的向量存储在字典中（image_name --> feature_vector）。\n",
        "- 当通过网络传递所有图像后，对字典进行 pickle 操作，然后将其保存到磁盘。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RD3vW4SsRPFW"
      },
      "outputs": [],
      "source": [
        "image_model = tf.keras.applications.InceptionV3(include_top=False,\n",
        "                                                weights='imagenet')\n",
        "new_input = image_model.input\n",
        "hidden_layer = image_model.layers[-1].output\n",
        "\n",
        "image_features_extract_model = tf.keras.Model(new_input, hidden_layer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rERqlR3WRPGO"
      },
      "source": [
        "## 缓存从 InceptionV3 中提取的特征\n",
        "\n",
        "您将使用 InceptionV3 预处理每个图像，然后将输出缓存到磁盘。将输出缓存到 RAM 会更快，但也会占用大量内存（每个图像需要 8 * 8 * 2048 个浮点数）。写入时，这将超出 Colab 的内存限制（目前为 12GB 内存）。\n",
        "\n",
        "使用更复杂的缓存策略（例如，将图像分片来减少对磁盘 I/O 的随机访问）可以提高性能，但这需要编写更多代码。\n",
        "\n",
        "使用 GPU 在 Colab 中运行缓存大约需要 10 分钟。如果您想查看进度条，可以进行以下操作：\n",
        "\n",
        "1. 安装 [tqdm](https://github.com/tqdm/tqdm)：\n",
        "\n",
        "    `!pip install tqdm`\n",
        "\n",
        "2. 导入 tqdm：\n",
        "\n",
        "    `from tqdm import tqdm`\n",
        "\n",
        "3. 将下面的行：\n",
        "\n",
        "    `for img, path in image_dataset:`\n",
        "\n",
        "    改为：\n",
        "\n",
        "    `for img, path in tqdm(image_dataset):`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Dx_fvbVgRPGQ"
      },
      "outputs": [],
      "source": [
        "# Get unique images\n",
        "encode_train = sorted(set(img_name_vector))\n",
        "\n",
        "# Feel free to change batch_size according to your system configuration\n",
        "image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)\n",
        "image_dataset = image_dataset.map(\n",
        "  load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE).batch(16)\n",
        "\n",
        "for img, path in image_dataset:\n",
        "  batch_features = image_features_extract_model(img)\n",
        "  batch_features = tf.reshape(batch_features,\n",
        "                              (batch_features.shape[0], -1, batch_features.shape[3]))\n",
        "\n",
        "  for bf, p in zip(batch_features, path):\n",
        "    path_of_feature = p.numpy().decode(\"utf-8\")\n",
        "    np.save(path_of_feature, bf.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nyqH3zFwRPFi"
      },
      "source": [
        "## 预处理描述并进行分词\n",
        "\n",
        "- 首先，您需要对描述进行分词（例如，按照空格进行切分）。这会为我们提供一个包含数据中的所有唯一单词的词汇表（例如，“surfing”、“football”等）。\n",
        "- 接下来，将词汇表的大小限制为前 5,000 个单词（以节省内存）。将所有其他单词替换为“UNK”（未知）。\n",
        "- 然后，创建“词到索引”和“索引到词”的映射。\n",
        "- 最后，将所有序列填充为与最长序列相同的长度。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HZfK8RhQRPFj"
      },
      "outputs": [],
      "source": [
        "# Find the maximum length of any caption in our dataset\n",
        "def calc_max_length(tensor):\n",
        "    return max(len(t) for t in tensor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "oJGE34aiRPFo"
      },
      "outputs": [],
      "source": [
        "# Choose the top 5000 words from the vocabulary\n",
        "top_k = 5000\n",
        "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,\n",
        "                                                  oov_token=\"&lt;unk&gt;\",\n",
        "                                                  filters='!\"#$%&amp;()*+.,-/:;=?@[\\]^_`{|}~ ')\n",
        "tokenizer.fit_on_texts(train_captions)\n",
        "train_seqs = tokenizer.texts_to_sequences(train_captions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "8Q44tNQVRPFt"
      },
      "outputs": [],
      "source": [
        "tokenizer.word_index['&lt;pad&gt;'] = 0\n",
        "tokenizer.index_word[0] = '&lt;pad&gt;'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0fpJb5ojRPFv"
      },
      "outputs": [],
      "source": [
        "# Create the tokenized vectors\n",
        "train_seqs = tokenizer.texts_to_sequences(train_captions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AidglIZVRPF4"
      },
      "outputs": [],
      "source": [
        "# Pad each vector to the max_length of the captions\n",
        "# If you do not provide a max_length value, pad_sequences calculates it automatically\n",
        "cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gL0wkttkRPGA"
      },
      "outputs": [],
      "source": [
        "# Calculates the max_length, which is used to store the attention weights\n",
        "max_length = calc_max_length(train_seqs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "M3CD75nDpvTI"
      },
      "source": [
        "## 将数据拆分为训练和测试"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "iS7DDMszRPGF"
      },
      "outputs": [],
      "source": [
        "# Create training and validation sets using an 80-20 split\n",
        "img_name_train, img_name_val, cap_train, cap_val = train_test_split(img_name_vector,\n",
        "                                                                    cap_vector,\n",
        "                                                                    test_size=0.2,\n",
        "                                                                    random_state=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XmViPkRFRPGH"
      },
      "outputs": [],
      "source": [
        "len(img_name_train), len(cap_train), len(img_name_val), len(cap_val)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uEWM9xrYcg45"
      },
      "source": [
        "## 创建用于训练的 tf.data 数据集\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "horagNvhhZiy"
      },
      "source": [
        "我们的图像和描述已准备就绪！接下来，让我们创建一个 tf.data 数据集，用于训练我们的模型。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Q3TnZ1ToRPGV"
      },
      "outputs": [],
      "source": [
        "# Feel free to change these parameters according to your system's configuration\n",
        "\n",
        "BATCH_SIZE = 64\n",
        "BUFFER_SIZE = 1000\n",
        "embedding_dim = 256\n",
        "units = 512\n",
        "vocab_size = top_k + 1\n",
        "num_steps = len(img_name_train) // BATCH_SIZE\n",
        "# Shape of the vector extracted from InceptionV3 is (64, 2048)\n",
        "# These two variables represent that vector shape\n",
        "features_shape = 2048\n",
        "attention_features_shape = 64"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "SmZS2N0bXG3T"
      },
      "outputs": [],
      "source": [
        "# Load the numpy files\n",
        "def map_func(img_name, cap):\n",
        "  img_tensor = np.load(img_name.decode('utf-8')+'.npy')\n",
        "  return img_tensor, cap"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FDF_Nm3tRPGZ"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))\n",
        "\n",
        "# Use map to load the numpy files in parallel\n",
        "dataset = dataset.map(lambda item1, item2: tf.numpy_function(\n",
        "          map_func, [item1, item2], [tf.float32, tf.int32]),\n",
        "          num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "# Shuffle and batch\n",
        "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n",
        "dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nrvoDphgRPGd"
      },
      "source": [
        "## 模型\n",
        "\n",
        "有趣的事实：下面的解码器与[使用注意力机制的神经网络机器翻译](../sequences/nmt_with_attention.ipynb)示例中的解码器相同。\n",
        "\n",
        "此模型架构的灵感来自论文 [Show, Attend and Tell](https://arxiv.org/pdf/1502.03044.pdf)。\n",
        "\n",
        "- 在此示例中，您将从 InceptionV3 的较低卷积层中提取特征，从而得到一个形状为 (8, 8, 2048) 的向量。\n",
        "- 将上述形状挤压为 (64, 2048) 的形状。\n",
        "- 然后通过 CNN 编码器（包含一个全连接层）传递此向量。\n",
        "- RNN（此处为 GRU）会将注意力放在图像上以预测下一个单词。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ja2LFTMSdeV3"
      },
      "outputs": [],
      "source": [
        "class BahdanauAttention(tf.keras.Model):\n",
        "  def __init__(self, units):\n",
        "    super(BahdanauAttention, self).__init__()\n",
        "    self.W1 = tf.keras.layers.Dense(units)\n",
        "    self.W2 = tf.keras.layers.Dense(units)\n",
        "    self.V = tf.keras.layers.Dense(1)\n",
        "\n",
        "  def call(self, features, hidden):\n",
        "    # features(CNN_encoder output) shape == (batch_size, 64, embedding_dim)\n",
        "\n",
        "    # hidden shape == (batch_size, hidden_size)\n",
        "    # hidden_with_time_axis shape == (batch_size, 1, hidden_size)\n",
        "    hidden_with_time_axis = tf.expand_dims(hidden, 1)\n",
        "\n",
        "    # score shape == (batch_size, 64, hidden_size)\n",
        "    score = tf.nn.tanh(self.W1(features) + self.W2(hidden_with_time_axis))\n",
        "\n",
        "    # attention_weights shape == (batch_size, 64, 1)\n",
        "    # you get 1 at the last axis because you are applying score to self.V\n",
        "    attention_weights = tf.nn.softmax(self.V(score), axis=1)\n",
        "\n",
        "    # context_vector shape after sum == (batch_size, hidden_size)\n",
        "    context_vector = attention_weights * features\n",
        "    context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "    return context_vector, attention_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AZ7R1RxHRPGf"
      },
      "outputs": [],
      "source": [
        "class CNN_Encoder(tf.keras.Model):\n",
        "    # Since you have already extracted the features and dumped it using pickle\n",
        "    # This encoder passes those features through a Fully connected layer\n",
        "    def __init__(self, embedding_dim):\n",
        "        super(CNN_Encoder, self).__init__()\n",
        "        # shape after fc == (batch_size, 64, embedding_dim)\n",
        "        self.fc = tf.keras.layers.Dense(embedding_dim)\n",
        "\n",
        "    def call(self, x):\n",
        "        x = self.fc(x)\n",
        "        x = tf.nn.relu(x)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "V9UbGQmERPGi"
      },
      "outputs": [],
      "source": [
        "class RNN_Decoder(tf.keras.Model):\n",
        "  def __init__(self, embedding_dim, units, vocab_size):\n",
        "    super(RNN_Decoder, self).__init__()\n",
        "    self.units = units\n",
        "\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc1 = tf.keras.layers.Dense(self.units)\n",
        "    self.fc2 = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "    self.attention = BahdanauAttention(self.units)\n",
        "\n",
        "  def call(self, x, features, hidden):\n",
        "    # defining attention as a separate model\n",
        "    context_vector, attention_weights = self.attention(features, hidden)\n",
        "\n",
        "    # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
        "    x = self.embedding(x)\n",
        "\n",
        "    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
        "    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
        "\n",
        "    # passing the concatenated vector to the GRU\n",
        "    output, state = self.gru(x)\n",
        "\n",
        "    # shape == (batch_size, max_length, hidden_size)\n",
        "    x = self.fc1(output)\n",
        "\n",
        "    # x shape == (batch_size * max_length, hidden_size)\n",
        "    x = tf.reshape(x, (-1, x.shape[2]))\n",
        "\n",
        "    # output shape == (batch_size * max_length, vocab)\n",
        "    x = self.fc2(x)\n",
        "\n",
        "    return x, state, attention_weights\n",
        "\n",
        "  def reset_state(self, batch_size):\n",
        "    return tf.zeros((batch_size, self.units))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Qs_Sr03wRPGk"
      },
      "outputs": [],
      "source": [
        "encoder = CNN_Encoder(embedding_dim)\n",
        "decoder = RNN_Decoder(embedding_dim, units, vocab_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-bYN7xA0RPGl"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6A3Ni64joyab"
      },
      "source": [
        "## 检查点"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "PpJAqPMWo0uE"
      },
      "outputs": [],
      "source": [
        "checkpoint_path = \"./checkpoints/train\"\n",
        "ckpt = tf.train.Checkpoint(encoder=encoder,\n",
        "                           decoder=decoder,\n",
        "                           optimizer = optimizer)\n",
        "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fUkbqhc_uObw"
      },
      "outputs": [],
      "source": [
        "start_epoch = 0\n",
        "if ckpt_manager.latest_checkpoint:\n",
        "  start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])\n",
        "  # restoring the latest checkpoint in checkpoint_path\n",
        "  ckpt.restore(ckpt_manager.latest_checkpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PHod7t72RPGn"
      },
      "source": [
        "## 训练\n",
        "\n",
        "- 提取存储在相应 `.npy` 文件中的特征，然后通过编码器传递这些特征。\n",
        "- 将编码器输出、隐藏状态（已初始化为 0）和解码器输入（即起始词例）传递给解码器。\n",
        "- 解码器会返回预测和解码器隐藏状态。\n",
        "- 然后将解码器隐藏状态传回模型，并使用预测计算损失。\n",
        "- 使用 Teacher Forcing 方法决定解码器的下一个输入。\n",
        "- Teacher Forcing 是一种将目标词作为下一个输入传递给解码器的技术。\n",
        "- 最后一步是计算梯度并将其应用于优化器和反向传播。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Vt4WZ5mhJE-E"
      },
      "outputs": [],
      "source": [
        "# adding this in a separate cell because if you run the training cell\n",
        "# many times, the loss_plot array will be reset\n",
        "loss_plot = []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sqgyz2ANKlpU"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_step(img_tensor, target):\n",
        "  loss = 0\n",
        "\n",
        "  # initializing the hidden state for each batch\n",
        "  # because the captions are not related from image to image\n",
        "  hidden = decoder.reset_state(batch_size=target.shape[0])\n",
        "\n",
        "  dec_input = tf.expand_dims([tokenizer.word_index['&lt;start&gt;']] * target.shape[0], 1)\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "      features = encoder(img_tensor)\n",
        "\n",
        "      for i in range(1, target.shape[1]):\n",
        "          # passing the features through the decoder\n",
        "          predictions, hidden, _ = decoder(dec_input, features, hidden)\n",
        "\n",
        "          loss += loss_function(target[:, i], predictions)\n",
        "\n",
        "          # using teacher forcing\n",
        "          dec_input = tf.expand_dims(target[:, i], 1)\n",
        "\n",
        "  total_loss = (loss / int(target.shape[1]))\n",
        "\n",
        "  trainable_variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, trainable_variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, trainable_variables))\n",
        "\n",
        "  return loss, total_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "UlA4VIQpRPGo"
      },
      "outputs": [],
      "source": [
        "EPOCHS = 20\n",
        "\n",
        "for epoch in range(start_epoch, EPOCHS):\n",
        "    start = time.time()\n",
        "    total_loss = 0\n",
        "\n",
        "    for (batch, (img_tensor, target)) in enumerate(dataset):\n",
        "        batch_loss, t_loss = train_step(img_tensor, target)\n",
        "        total_loss += t_loss\n",
        "\n",
        "        if batch % 100 == 0:\n",
        "            print ('Epoch {} Batch {} Loss {:.4f}'.format(\n",
        "              epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))\n",
        "    # storing the epoch end loss value to plot later\n",
        "    loss_plot.append(total_loss / num_steps)\n",
        "\n",
        "    if epoch % 5 == 0:\n",
        "      ckpt_manager.save()\n",
        "\n",
        "    print ('Epoch {} Loss {:.6f}'.format(epoch + 1,\n",
        "                                         total_loss/num_steps))\n",
        "    print ('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1Wm83G-ZBPcC"
      },
      "outputs": [],
      "source": [
        "plt.plot(loss_plot)\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.title('Loss Plot')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xGvOcLQKghXN"
      },
      "source": [
        "## 说明！\n",
        "\n",
        "- 评估函数与训练循环类似，不同之处在于，此处不使用 Teacher Forcing。每一个时间步骤的解码器输入都是其前一步的预测结果、隐藏状态和编码器输出。\n",
        "- 当模型预测到最后一个词例时停止。\n",
        "- 存储每一个时间步骤的注意力权重。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RCWpDtyNRPGs"
      },
      "outputs": [],
      "source": [
        "def evaluate(image):\n",
        "    attention_plot = np.zeros((max_length, attention_features_shape))\n",
        "\n",
        "    hidden = decoder.reset_state(batch_size=1)\n",
        "\n",
        "    temp_input = tf.expand_dims(load_image(image)[0], 0)\n",
        "    img_tensor_val = image_features_extract_model(temp_input)\n",
        "    img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))\n",
        "\n",
        "    features = encoder(img_tensor_val)\n",
        "\n",
        "    dec_input = tf.expand_dims([tokenizer.word_index['&lt;start&gt;']], 0)\n",
        "    result = []\n",
        "\n",
        "    for i in range(max_length):\n",
        "        predictions, hidden, attention_weights = decoder(dec_input, features, hidden)\n",
        "\n",
        "        attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()\n",
        "\n",
        "        predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()\n",
        "        result.append(tokenizer.index_word[predicted_id])\n",
        "\n",
        "        if tokenizer.index_word[predicted_id] == '&lt;end&gt;':\n",
        "            return result, attention_plot\n",
        "\n",
        "        dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "    attention_plot = attention_plot[:len(result), :]\n",
        "    return result, attention_plot"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fD_y7PD6RPGt"
      },
      "outputs": [],
      "source": [
        "def plot_attention(image, result, attention_plot):\n",
        "    temp_image = np.array(Image.open(image))\n",
        "\n",
        "    fig = plt.figure(figsize=(10, 10))\n",
        "\n",
        "    len_result = len(result)\n",
        "    for l in range(len_result):\n",
        "        temp_att = np.resize(attention_plot[l], (8, 8))\n",
        "        ax = fig.add_subplot(len_result//2, len_result//2, l+1)\n",
        "        ax.set_title(result[l])\n",
        "        img = ax.imshow(temp_image)\n",
        "        ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7x8RiPHe_4qI"
      },
      "outputs": [],
      "source": [
        "# captions on the validation set\n",
        "rid = np.random.randint(0, len(img_name_val))\n",
        "image = img_name_val[rid]\n",
        "real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])\n",
        "result, attention_plot = evaluate(image)\n",
        "\n",
        "print ('Real Caption:', real_caption)\n",
        "print ('Prediction Caption:', ' '.join(result))\n",
        "plot_attention(image, result, attention_plot)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Rprk3HEvZuxb"
      },
      "source": [
        "## 在自己的图像上进行尝试\n",
        "\n",
        "为了增加趣味性，我们在下面提供了一个方法，可让您通过我们刚才训练的模型为您自己的图像生成描述。请记住，这个模型是使用较少数据训练的，而您的图像可能与训练数据不同（因此请做好心理准备，您可能会得到奇怪的结果）。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9Psd1quzaAWg"
      },
      "outputs": [],
      "source": [
        "image_url = 'https://tensorflow.org/images/surf.jpg'\n",
        "image_extension = image_url[-4:]\n",
        "image_path = tf.keras.utils.get_file('image'+image_extension,\n",
        "                                     origin=image_url)\n",
        "\n",
        "result, attention_plot = evaluate(image_path)\n",
        "print ('Prediction Caption:', ' '.join(result))\n",
        "plot_attention(image_path, result, attention_plot)\n",
        "# opening the image\n",
        "Image.open(image_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VJZXyJco6uLO"
      },
      "source": [
        "# 后续步骤\n",
        "\n",
        "恭喜您！您刚刚使用注意力机制训练了一个图像描述生成模型。接下来，您可以看看这个示例：[使用注意力机制的神经网络机器翻译](../sequences/nmt_with_attention.ipynb)。它使用了类似架构对西班牙语和英语的句子进行互译。您还可以尝试在不同的数据集上训练此笔记本中的代码。"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "image_captioning.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.17461054298704312, 'fraction_numerical': 0.010949851862976666, 'mean_word_length': 1.8227223678125481, 'pattern_counts': {'":': 752, '<': 47, '<?xml version=': 0, '>': 49, 'https://': 36, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 95, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3826378', 'n_tokens_mistral': 28154, 'n_tokens_neox': 28240, 'n_words': 4102}