<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oxb_tjw13y4G"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EAkh2aBJLg6q"
      },
      "outputs": [],
      "source": [
        "# Copyright 2019 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owAopeOtirc9"
      },
      "source": [
        "# Wiki40B 语言模型\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T-nCyGRri-KO"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://tensorflow.google.cn/hub/tutorials/wiki40b_lm\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\">View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">View on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">Download notebook</a>\n",
        "  </td>\n",
        "  <td><a href=\"https://tfhub.dev/google/collections/wiki40b-lm/1\"><img src=\"https://tensorflow.google.cn/images/hub_logo_32px.png\">查看 TF Hub 模型</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eY9jkGpjf3d"
      },
      "source": [
        "使用 [TensorFlow Hub](https://tfhub.dev) 中的 **Wiki40B 语言模型**生成类似 Wikipedia 的文本。\n",
        "\n",
        "此笔记本介绍如何执行以下操作：\n",
        "\n",
        "- 在 TF-Hub 上加载属于 [Wiki40b-LM 集合](https://tfhub.dev/google/collections/wiki40b-lm/1)的 41 种单语言模型和 2 种多语言模型\n",
        "- 使用这些模型获取指定文本片段的复杂度、每层激活和单词嵌入向量\n",
        "- 按词例从种子文本片段生成文本\n",
        "\n",
        "这些语言模型使用新发布并且经过清理的 [Wiki40B 数据集](https://tensorflow.google.cn/datasets/catalog/wiki40b)（在 TensorFlow 数据集上提供）进行训练。训练设置基于论文 [Wiki-40B: Multilingual Language Model Dataset](https://research.google/pubs/pub49029/)。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wK2YnrEhLjDf"
      },
      "source": [
        "## 设置"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "sv2CmI7BdaML"
      },
      "outputs": [],
      "source": [
        "#@title Installing Dependencies\n",
        "!pip install --quiet tensorflow_text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "8uSkaQ-Vdon2"
      },
      "outputs": [],
      "source": [
        "#@title Imports\n",
        "import numpy as np\n",
        "import tensorflow.compat.v1 as tf\n",<commit_msg>Translate wiki40b_lm.ipynb via GitLocalize<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oxb_tjw13y4G"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EAkh2aBJLg6q"
      },
      "outputs": [],
      "source": [
        "# Copyright 2019 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owAopeOtirc9"
      },
      "source": [
        "# Wiki40B 语言模型\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T-nCyGRri-KO"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://tensorflow.google.cn/hub/tutorials/wiki40b_lm\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\">View on TensorFlow.org</a>   </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">Run in Google Colab</a>   </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">View on GitHub</a>   </td>\n",
        "  <td>     <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/hub/tutorials/wiki40b_lm.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">Download notebook</a>   </td>\n",
        "  <td><a href=\"https://tfhub.dev/google/collections/wiki40b-lm/1\"><img src=\"https://tensorflow.google.cn/images/hub_logo_32px.png\">查看 TF Hub 模型</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eY9jkGpjf3d"
      },
      "source": [
        "使用 [TensorFlow Hub](https://tfhub.dev) 中的 **Wiki40B 语言模型**生成类似 Wikipedia 的文本。\n",
        "\n",
        "此笔记本介绍如何执行以下操作：\n",
        "\n",
        "- 在 TF-Hub 上加载属于 [Wiki40b-LM 集合](https://tfhub.dev/google/collections/wiki40b-lm/1)的 41 种单语言模型和 2 种多语言模型\n",
        "- 使用这些模型获取指定文本片段的复杂度、每层激活和单词嵌入向量\n",
        "- 按词例从种子文本片段生成文本\n",
        "\n",
        "这些语言模型使用新发布并且经过清理的 [Wiki40B 数据集](https://tensorflow.google.cn/datasets/catalog/wiki40b)（在 TensorFlow 数据集上提供）进行训练。训练设置基于论文 [Wiki-40B: Multilingual Language Model Dataset](https://research.google/pubs/pub49029/)。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wK2YnrEhLjDf"
      },
      "source": [
        "## 设置"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "sv2CmI7BdaML"
      },
      "outputs": [],
      "source": [
        "#@title Installing Dependencies\n",
        "!pip install --quiet \"tensorflow-text==2.8.*\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "8uSkaQ-Vdon2"
      },
      "outputs": [],
      "source": [
        "#@title Imports\n",
        "import numpy as np\n",
        "import tensorflow.compat.v1 as tf\n",
{'fraction_non_alphanumeric': 0.2008887876025524, 'fraction_numerical': 0.01845943482224248, 'mean_word_length': 2.1870007262164126, 'pattern_counts': {'":': 82, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 28, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6974777', 'n_tokens_mistral': 3318, 'n_tokens_neox': 3293, 'n_words': 528}