<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEL3NlTTDlSX"
      },
      "source": [
        "##### Copyright 2021 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "FlUw7tSKbtg4"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77z2OchJTk0l"
      },
      "source": [
        "# 调试 TF2 迁移的训练流水线\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/migrate/migration_debugging\">     <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">     View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/migrate/migration_debugging.ipynb\">     <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">     Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate/migration_debugging.ipynb\">     <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">     View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/migrate/migration_debugging.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zTwPu-w6M5sz"
      },
      "source": [
        "此笔记本演示了如何在迁移到 TF2 时调试训练流水线。它由以下组件组成：\n",
        "\n",
        "1. 调试训练流水线的建议步骤和代码示例\n",
        "2. 用于调试的工具\n",
        "3. 其他相关资源\n",
        "\n",
        "一个假设是您有用于比较的 TF1.x 代码和已训练模型，并且您希望构建一个 TF2 模型来实现类似的验证准确率。\n",
        "\n",
        "此笔记本**不**涵盖有关训练/推断速度或内存使用的调试性能问题。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fKm9R4CtOAP3"
      },
      "source": [
        "## 调试工作流\n",
        "\n",
        "下面是调试 TF2 训练流水线的一般工作流。请注意，您不需要按顺序执行这些步骤。您也可以使用二分查找方法，在中间步骤中测试模型并缩小调试范围。\n",
        "\n",
        "1. 修复编译和运行时错误\n",
        "\n",
        "2. 单次前向传递验证（在单独的[指南](./validate_correctness.ipynb)中）\n",
        "\n",
        "    a. 在单个 CPU 设备上\n",
        "\n",
        "    - 验证变量是否只创建一次\n",
        "    - 检查变量计数、名称和形状是否匹配\n",
        "    - 重置所有变量，在停用所有随机性的情况下检查数值等价性\n",
        "    - 对齐随机数生成，检查推断中的数值等价性\n",
        "    - （可选）检查检查点已正确加载，TF1.x/TF2 模型生成相同的输出\n",
        "\n",
        "    b. 在单个 GPU/TPU 设备上\n",
        "\n",
        "    c. 采用多设备策略\n",
        "\n",
        "3. 几个步骤的模型训练数值等价性验证（下面提供代码示例）\n",
        "\n",
        "    a. 在单个 CPU 设备上使用少量固定数据进行单次训练步骤验证。具体来说，检查以下组件的数值等价性\n",
        "\n",
        "    - 损失计算\n",
        "    - 指标\n",
        "    - 学习率\n",
        "    - 梯度计算和更新\n",
        "\n",
        "    b. 在训练 3 个或更多步骤后检查统计数据，验证优化器的行为（如动量）在单个 CPU 设备上是否仍然使用固定数据\n",
        "\n",
        "    c. 在单个 GPU/TPU 设备上\n",
        "\n",
        "    d. 使用多设备策略（查看底部 [MultiProcessRunner](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/multi_process_runner.py#L108) 的介绍）\n",
        "\n",
        "4. 对真实数据集的端到端覆盖测试\n",
        "\n",
        "    a. 使用 TensorBoard 检查训练行为\n",
        "\n",
        "    - 首先使用 SGD 等简单的优化器和 `tf.distribute.OneDeviceStrategy` 等简单的分布策略\n",
        "    - 训练指标\n",
        "    - 评估指标\n",
        "    - 找出对固有随机性的合理容忍度是多少\n",
        "\n",
        "    b. 使用高级优化器/学习率调度器/分布策略检查等价性\n",
        "\n",
        "    c. 使用混合精度时检查等价性\n",
        "\n",
        "5. 附加产品基准"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKakQBI9-FLb"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sopP--i7-LaF"
      },
      "outputs": [],
      "source": [
        "!pip uninstall -y -q tensorflow"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i1ghHyXl-Oqd"
      },
      "outputs": [],
      "source": [
        "# Install tf-nightly as the DeterministicRandomTestTool is only available in\n",<commit_msg>Translate migration_debugging.ipynb via GitLocalize<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEL3NlTTDlSX"
      },
      "source": [
        "##### Copyright 2021 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "FlUw7tSKbtg4"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77z2OchJTk0l"
      },
      "source": [
        "# 调试 TF2 迁移的训练流水线\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/migrate/migration_debugging\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">在 TensorFlow.org 上查看</a>   </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/migrate/migration_debugging.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">在 Google Colab 运行</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate/migration_debugging.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">在 Github 上查看源代码</a>\n",
        "</td>\n",
        "  <td>     <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/migrate/migration_debugging.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">下载笔记本</a>   </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zTwPu-w6M5sz"
      },
      "source": [
        "此笔记本演示了如何在迁移到 TF2 时调试训练流水线。它由以下组件组成：\n",
        "\n",
        "1. 调试训练流水线的建议步骤和代码示例\n",
        "2. 用于调试的工具\n",
        "3. 其他相关资源\n",
        "\n",
        "一个假设是您有用于比较的 TF1.x 代码和已训练模型，并且您希望构建一个 TF2 模型来实现类似的验证准确率。\n",
        "\n",
        "此笔记本**不**涵盖有关训练/推断速度或内存使用的调试性能问题。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fKm9R4CtOAP3"
      },
      "source": [
        "## 调试工作流\n",
        "\n",
        "下面是调试 TF2 训练流水线的一般工作流。请注意，您不需要按顺序执行这些步骤。您也可以使用二分查找方法，在中间步骤中测试模型并缩小调试范围。\n",
        "\n",
        "1. 修复编译和运行时错误\n",
        "\n",
        "2. 单次前向传递验证（在单独的[指南](./validate_correctness.ipynb)中）\n",
        "\n",
        "    a. 在单个 CPU 设备上\n",
        "\n",
        "    - 验证变量是否只创建一次\n",
        "    - 检查变量计数、名称和形状是否匹配\n",
        "    - 重置所有变量，在停用所有随机性的情况下检查数值等价性\n",
        "    - 对齐随机数生成，检查推断中的数值等价性\n",
        "    - （可选）检查检查点已正确加载，TF1.x/TF2 模型生成相同的输出\n",
        "\n",
        "    b. 在单个 GPU/TPU 设备上\n",
        "\n",
        "    c. 采用多设备策略\n",
        "\n",
        "3. 几个步骤的模型训练数值等价性验证（下面提供代码示例）\n",
        "\n",
        "    a. 在单个 CPU 设备上使用少量固定数据进行单次训练步骤验证。具体来说，检查以下组件的数值等价性\n",
        "\n",
        "    - 损失计算\n",
        "    - 指标\n",
        "    - 学习率\n",
        "    - 梯度计算和更新\n",
        "\n",
        "    b. 在训练 3 个或更多步骤后检查统计数据，验证优化器的行为（如动量）在单个 CPU 设备上是否仍然使用固定数据\n",
        "\n",
        "    c. 在单个 GPU/TPU 设备上\n",
        "\n",
        "    d. 使用多设备策略（查看底部 [MultiProcessRunner](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/multi_process_runner.py#L108) 的介绍）\n",
        "\n",
        "4. 对真实数据集的端到端覆盖测试\n",
        "\n",
        "    a. 使用 TensorBoard 检查训练行为\n",
        "\n",
        "    - 首先使用 SGD 等简单的优化器和 `tf.distribute.OneDeviceStrategy` 等简单的分布策略\n",
        "    - 训练指标\n",
        "    - 评估指标\n",
        "    - 找出对固有随机性的合理容忍度是多少\n",
        "\n",
        "    b. 使用高级优化器/学习率调度器/分布策略检查等价性\n",
        "\n",
        "    c. 使用混合精度时检查等价性\n",
        "\n",
        "5. 附加产品基准"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKakQBI9-FLb"
      },
      "source": [
        "## 安装"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sopP--i7-LaF"
      },
      "outputs": [],
      "source": [
        "!pip uninstall -y -q tensorflow"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i1ghHyXl-Oqd"
      },
      "outputs": [],
      "source": [
        "# Install tf-nightly as the DeterministicRandomTestTool is only available in\n",
{'fraction_non_alphanumeric': 0.18117828560408833, 'fraction_numerical': 0.009642271719217048, 'mean_word_length': 1.7446414395342684, 'pattern_counts': {'":': 80, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 20, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3948980', 'n_tokens_mistral': 4647, 'n_tokens_neox': 5006, 'n_words': 620}