<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DweYe9FcbMK_"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "AVV2e0XKbJeX"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZfSvVcDo6GQ"
      },
      "source": [
        "# 加载文本"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "giK0nMbZFnoR"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/load_data/text\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\"> 在 TensorFlow.org 上查看</a>   </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">在 Google Colab 中运行</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">在 GitHub 上查看源代码</a>\n",
        "</td>\n",
        "  <td>     <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">下载笔记本</a>\n",
        "</td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwlfPb11GH8J"
      },
      "source": [
        "本教程演示了两种加载和预处理文本的方法。\n",
        "\n",
        "- 首先，您将使用 Keras 效用函数和预处理层。这包括用于将数据转换为 `tf.data.Dataset` 的 `tf.keras.utils.text_dataset_from_directory` 和用于数据标准化、词例化和向量化的 `tf.keras.layers.TextVectorization`。如果您是 TensorFlow 新手，则应当从这些开始。\n",
        "- 然后，您将使用 `tf.data.TextLineDataset` 等较低级别的效用函数来加载文本文件，并使用 [TensorFlow Text](https://tensorflow.google.cn/text) API（如 `text.UnicodeScriptTokenizer` 和 `text.case_fold_utf8`）来预处理数据以实现粒度更细的控制。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sa6IKWvADqH7"
      },
      "outputs": [],
      "source": [
        "!pip install \"tensorflow-text==2.8.*\""
      ]<commit_msg>Translate text.ipynb via GitLocalize<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DweYe9FcbMK_"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "AVV2e0XKbJeX"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZfSvVcDo6GQ"
      },
      "source": [
        "# 加载文本"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "giK0nMbZFnoR"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/load_data/text\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\"> 在 TensorFlow.org 上查看</a>   </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">在 Google Colab 中运行</a> </td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">在 GitHub 上查看源代码</a> </td>\n",
        "  <td>     <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/tutorials/load_data/text.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">下载笔记本</a> </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwlfPb11GH8J"
      },
      "source": [
        "本教程演示了两种加载和预处理文本的方法。\n",
        "\n",
        "- 首先，您将使用 Keras 效用函数和预处理层。这包括用于将数据转换为 `tf.data.Dataset` 的 `tf.keras.utils.text_dataset_from_directory` 和用于数据标准化、词例化和向量化的 `tf.keras.layers.TextVectorization`。如果您是 TensorFlow 新手，则应当从这些开始。\n",
        "- 然后，您将使用 `tf.data.TextLineDataset` 等较低级别的效用函数来加载文本文件，并使用 [TensorFlow Text](https://tensorflow.google.cn/text) API（如 `text.UnicodeScriptTokenizer` 和 `text.case_fold_utf8`）来预处理数据以实现粒度更细的控制。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sa6IKWvADqH7"
      },
      "outputs": [],
      "source": [
        "!pip install \"tensorflow-text==2.8.*\""
      ]
{'fraction_non_alphanumeric': 0.18247613181398215, 'fraction_numerical': 0.010779180782260549, 'mean_word_length': 2.372274143302181, 'pattern_counts': {'":': 60, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 20, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6803483', 'n_tokens_mistral': 2489, 'n_tokens_neox': 2519, 'n_words': 387}