<commit_before><commit_msg>Add demo for Fairness Indicators on TF Hub.<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aalPefrUUplk"
      },
      "source": [
        "# Fairness Indicators on TF-Hub Text Embeddings\n",
        "\n",
        "In this colab, you will learn how to use Fairness Indicators. Fairness Indicators a suite of tools built on top of [TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis/get_started) that facilitates evaluation and visualization of fairness metrics on models.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "w4-v6mEnY89E"
      },
      "source": [
        "## Read Me First\n",
        "\n",
        "This colab is presently designed for **Python 2**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "BAUEkqYlzP3W"
      },
      "outputs": [],
      "source": [
        "%tensorflow_version 1.x\n",
        "!pip install tensorflow_model_analysis==0.15\n",
        "!pip install tensorflow_data_validation\n",
        "!pip install --upgrade witwidget"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "PIoOlYOSaE2m"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "print(sys.version)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "u33JXdluZ2lG"
      },
      "source": [
        "# Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "B8dlyTyiTe-9"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import tempfile\n",
        "import apache_beam as beam\n",
        "from datetime import datetime\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow as tf\n",
        "import tensorflow_model_analysis as tfma\n",
        "from tensorflow_model_analysis.addons.fairness.view import widget_view\n",
        "from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators  # must include to generate the post_export_metrics callback."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9ekzb7vVnPCc"
      },
      "source": [
        "# Defining Constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "n4_nXQDykX6W"
      },
      "outputs": [],
      "source": [
        "BASE_DIR = tempfile.gettempdir()\n",
        "\n",
        "# The input and output features of the classifier\n",
        "TEXT_FEATURE = 'comment_text'\n",
        "LABEL = 'toxicity'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TsplOJGqWCf5"
      },
      "source": [
        "# Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4sZag6SFaMIp"
      },
      "source": [
        "In this exercise, you'll work with the [Civil Comments dataset](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification), approximately 2 million public comments made public by the [Civil Comments platform](https://github.com/reaktivstudios/civil-comments) in 2017 for ongoing research. This effort was sponsored by Jigsaw, who have hosted competitions on Kaggle to help classify toxic comments as well as minimize unintended model bias.\n",
        "\n",
        "Each individual text comment in the dataset has a toxicity label, with the label being 1 if the comment is toxic and 0 if the comment is non-toxic. Within the data, a subset of comments are labeled with a variety of identity attributes, including categories for gender, sexual orientation, religion, and race or ethnicity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "kBGV8eFZaClA"
      },
      "outputs": [],
      "source": [
        "train_tf_file = tf.keras.utils.get_file('train.tf', 'https://storage.googleapis.com/civil_comments_dataset/train.tfrecord')\n",
        "validate_tf_file = tf.keras.utils.get_file('validate.tf', 'https://storage.googleapis.com/civil_comments_dataset/validate.tfrecord')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NzcVLKPW7sjn"
      },
      "source": [
        "## Identity Terms\n",
        "\n",
        "You can select the subset of identity groups you are interested in by removing the others from the list below. By default, we will look at all identity terms."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "l88BZ7rFKi4-"
      },
      "outputs": [],
      "source": [
        "IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mfbgerCsEOmN"
      },
      "source": [
        "# Creating a Pipeline to Compare Text Embeddings\n",
        "\n",
        "Let's compare the performance of models built with different text embeddings. First, we need to create a model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5dIo0yXvQdjo"
      },
      "source": [
        "## Input Function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jKYGRHaCSsie"
      },
      "source": [
        "TensorFlow parses features from data using [`FixedLenFeature`](https://www.tensorflow.org/api_docs/python/tf/io/FixedLenFeature) and [`VarLenFeature`](https://www.tensorflow.org/api_docs/python/tf/io/VarLenFeature). So to allow TensorFlow to parse our data, we will need to map out our input feature, output feature, and any slicing features that we will want to analyze via Fairness Indicators."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "bFhj0N12SnWv"
      },
      "outputs": [],
      "source": [
        "FEATURE_MAP = {\n",
        "    # input and output features\n",
        "    LABEL: tf.FixedLenFeature([], tf.float32),\n",
        "    TEXT_FEATURE: tf.FixedLenFeature([], tf.string),\n",
        "\n",
        "    # slicing features\n",
        "    'sexual_orientation': tf.VarLenFeature(tf.string),\n",
        "    'gender': tf.VarLenFeature(tf.string),\n",
        "    'religion': tf.VarLenFeature(tf.string),\n",
        "    'race': tf.VarLenFeature(tf.string),\n",
        "    'disability': tf.VarLenFeature(tf.string)\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "W97S4JhASwe_"
      },
      "source": [
        "Now that we have defined our features and their types, we can create an input function for our model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "3IbTVvhAQgmq"
      },
      "outputs": [],
      "source": [
        "def input_fn(tf_file):\n",
        "  def parse_function(serialized):\n",
        "    parsed_example = tf.io.parse_single_example(\n",
        "        serialized=serialized, features=FEATURE_MAP)\n",
        "    # Adds a weight column to deal with unbalanced classes.\n",
        "    parsed_example['weight'] = tf.add(parsed_example[LABEL], 0.1)\n",
        "    return (parsed_example,\n",
        "            parsed_example[LABEL])\n",
        "  train_dataset = tf.data.TFRecordDataset(\n",
        "      filenames=[tf_file]).map(parse_function).batch(512)\n",
        "  return train_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "23i7M_w_bvQu"
      },
      "source": [
        "## Classifier\n",
        "\n",
        "For each text embedding, we will train a **[DNN Classifier](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier)**.\n",
        "\n",
        "**TF Hub** allows us to insert text embeddings as features to our model via **[`text_embedding_column`](https://www.tensorflow.org/hub/api_docs/python/hub/text_embedding_column)**. The function's signature is **`text_embedding_column(key, module_spec)`**, where...\n",
        "\n",
        "* *`key`* is the name of the DataFrame's text feature (ex: `\"comment_text\"`)\n",
        "* *`module_spec`* is a url path to an text embedding module (ex: `\"https://tfhub.dev/google/nnlm-en-dim128/1\"`)\n",
        "\n",
        "Because each text embedding column is memory-intensive, the Colaboratory environment may crash if all embeddings are loaded at once. To avoid this, we encapsulate the embedding columns inside a pipeline and wait to get the pipeline's results before loading the next embedding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2NJHroV8CrdL"
      },
      "outputs": [],
      "source": [
        "def train_classifier(embedding):\n",
        "  embedded_text_feature_column = hub.text_embedding_column(\n",
        "      key=TEXT_FEATURE, \n",
        "      module_spec=embedding)\n",
        "  model_dir = os.path.join(BASE_DIR, 'train', datetime.now().strftime(\n",
        "    \"%Y%m%d-%H%M%S\"))\n",
        "  classifier = tf.estimator.DNNClassifier(\n",
        "      hidden_units=[500, 100],\n",
        "      weight_column='weight',\n",
        "      feature_columns=[embedded_text_feature_column],\n",
        "      n_classes=2,\n",
        "      optimizer=tf.train.AdagradOptimizer(learning_rate=0.003),\n",
        "      model_dir= model_dir)\n",
        "  classifier.train(input_fn=lambda: input_fn(train_tf_file), steps=1000);\n",
        "  return classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "onSL_CMbWHBB"
      },
      "source": [
        "## Fairness Indicators in TFMA\n",
        "\n",
        "To analyze our model's results with **Fairness Indicators**, we need to add Fairness Indicators as a callback that is returned after model execution.\n",
        "\n",
        "To do this, we use [TFMA]((https://www.tensorflow.org/tfx/model_analysis/get_started))'s **[`EvalSharedModel`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/types/EvalSharedModel)** class. `EvalSharedModel` builds on TFMA's **[`EvalSavedModel`](https://g3doc.corp.google.com/intelligence/lantern/tensorflow_model_analysis/g3doc/faq.md#what-is-an-evalsavedmodel)**, so we first need to convert our `tf.estimator` to an `EvalSavedModel`.\n",
        "\n",
        "EvalSavedModels parse [`tf.Examples`](https://www.tensorflow.org/tutorials/load_data/tfrecord#tfexample) with an **`eval_input_receiver_fn`**, so we also need to create that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GRiumqP-WM-j"
      },
      "outputs": [],
      "source": [
        "def eval_model_with_fairness_indicators(classifier):\n",
        "  eval_saved_model_path = tfma.export.export_eval_savedmodel(\n",
        "      estimator=classifier,\n",
        "      export_dir_base=os.path.join(BASE_DIR, 'tfma_eval_model'),\n",
        "      eval_input_receiver_fn=eval_input_receiver_fn)\n",
        "  fairness_indicator_callback = tfma.post_export_metrics.fairness_indicators(\n",
        "                                    thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],\n",
        "                                    labels_key=LABEL)\n",
        "  return tfma.default_eval_shared_model(\n",
        "        eval_saved_model_path=eval_saved_model_path,\n",
        "        add_metrics_callbacks=[fairness_indicator_callback])\n",
        "  \n",
        "def eval_input_receiver_fn():\n",
        "  \"\"\"Create a tfma.export.EvalInputReceiver to parse input features.\"\"\"\n",
        "  serialized_tf_example = tf.compat.v1.placeholder(\n",
        "      dtype=tf.string, shape=[None], name='input_example_placeholder')\n",
        "  receiver_tensors = {'examples': serialized_tf_example}\n",
        "  features = tf.parse_example(serialized_tf_example, FEATURE_MAP)\n",
        "  features['weight'] = tf.ones_like(features[LABEL])\n",
        "  return tfma.export.EvalInputReceiver(\n",
        "    features=features,\n",
        "    receiver_tensors=receiver_tensors,\n",
        "    labels=features[LABEL])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1HbCbLLbWZLT"
      },
      "source": [
        "## TFMA - Apache Beam\n",
        "\n",
        "Now that we have our evaluation model with a callback to the Fairness Indicators results, we can create a function that computes and returns those results!\n",
        "\n",
        "TFMA is built on the **[Apache Beam](https://beam.apache.org/documentation/programming-guide/)**, data processing framework. TFMA provides the [`ExtractEvaluateAndWriteResults`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/ExtractEvaluateAndWriteResults) API to use as a [`PTransform`](https://beam.apache.org/documentation/programming-guide/#transforms) in Beam pipelines. Check the [Get Started with TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis/get_started) tutorial for more information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ZKh1Ale0KynF"
      },
      "outputs": [],
      "source": [
        "def get_eval_result(embedding, eval_shared_model, eval_result_path):\n",
        "  slice_spec = [tfma.slicer.SingleSliceSpec()]\n",
        "  for identity in IDENTITY_TERMS:\n",
        "    slice_spec.append(tfma.slicer.SingleSliceSpec(columns=[identity]))\n",
        "  with beam.Pipeline() as pipeline:\n",
        "    _ = (\n",
        "        pipeline\n",
        "        | 'ReadFromTFRecord' \u003e\u003e beam.io.ReadFromTFRecord(\n",
        "            file_pattern=validate_tf_file)\n",
        "        | 'ExtractEvaluateAndWriteResults' \u003e\u003e\n",
        "        tfma.ExtractEvaluateAndWriteResults(\n",
        "                  eval_shared_model=eval_shared_model,\n",
        "                  slice_spec=slice_spec,\n",
        "                  compute_confidence_intervals=False,\n",
        "                  output_path=eval_result_path)\n",
        "    )\n",
        "  return tfma.load_eval_result(output_path=eval_result_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wM-faceoCPqg"
      },
      "source": [
        "# Evaluate Embedding\n",
        "\n",
        "Now that we have all the steps in place - training a model on an embedding, converting it to a TFMA format, and computing Fairness Indicators on it - we can make a pipeline to compare Fairness Indicators on different embeddings!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7nSvu4IUCigW"
      },
      "outputs": [],
      "source": [
        "def embedding_eval_result(embedding):\n",
        "\n",
        "  # First, we use the train_classifier() function we created earlier to train a\n",
        "  # basic classifier using our chosen embedding.\n",
        "  print(\"Training classifier for \" + embedding)\n",
        "  classifier = train_classifier(embedding)\n",
        "\n",
        "  # Next, we measure the accuracy of our classifier on our validation set.\n",
        "  train_eval_result = classifier.evaluate(input_fn=lambda: input_fn(validate_tf_file))\n",
        "  print('Validation set accuracy for {}: {accuracy}'.format(embedding, **train_eval_result))\n",
        "\n",
        "  # We then use our eval_model_with_fairness_indicators() function to convert\n",
        "  # the model to a TFMA EvalSharedModel with a callback to Fairness Indicators.\n",
        "  eval_shared_model = eval_model_with_fairness_indicators(classifier)\n",
        "\n",
        "  # We also need to create a unique path to store our results for this\n",
        "  # embedding.\n",
        "  embedding_name = embedding.split('/')[-2]\n",
        "  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)\n",
        "\n",
        "  # Finally, we use our get_eval_result() function to compute and return the\n",
        "  # Fairness Indicators results!\n",
        "  eval_result = get_eval_result(embedding, eval_shared_model, eval_result_path)\n",
        "  return eval_result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jTPqije9Eg5b"
      },
      "source": [
        "# Run TFMA \u0026 Fairness Indicators"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LGXCFtScblYt"
      },
      "source": [
        "## Text Embeddings\n",
        "\n",
        "**[TF-Hub](https://www.tensorflow.org/hub)** provides several **text embeddings**. These embeddings will serve as the feature column for our different models. For this Colab, we use the following embeddings:\n",
        "\n",
        "* [**random-nnlm-en-dim128**](https://tfhub.dev/google/random-nnlm-en-dim128/1): random text embeddings, this serves as a convenient baseline.\n",
        "* [**nnlm-en-dim128**](https://tfhub.dev/google/nnlm-en-dim128/1): a text embedding based on [A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf). \n",
        "* [**universal-sentence-encoder**](https://tfhub.dev/google/universal-sentence-encoder/2): a text embedding based on [Universal Sentence Encoder](https://arxiv.org/pdf/1803.11175.pdf).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8AvInTNt8Gyn"
      },
      "source": [
        "## Fairness Indicators Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jiLg5ikCzFR-"
      },
      "source": [
        "Refer [here](https://github.com/tensorflow/fairness-indicators) for more information on analyzing data with Fairness Indicators. Below are some of the available metrics.\n",
        "\n",
        "* [Negative Rate, False Negative Rate (FNR), and True Negative Rate (TNR)](https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates)\n",
        "* [Positive Rate, False Positive Rate (FPR), and True Positive Rate (TPR)](https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates)\n",
        "* [Accuracy](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Accuracy)\n",
        "* [Precision and Recall](https://en.wikipedia.org/wiki/Precision_and_recall)\n",
        "* [Precision-Recall AUC](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC)\n",
        "* [ROC AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve)\n",
        "\n",
        "Note that the `widget_view.render_fairness_indicator()` cells may need to be run twice for the visualization to be displayed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yEUbZ93y8NCW"
      },
      "source": [
        "#### Random NNLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DkSuox-Pb6Pz"
      },
      "outputs": [],
      "source": [
        "eval_result_random_nnlm = embedding_eval_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "05xUesz6VpAe"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result_random_nnlm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jmKe8Z1b8SBy"
      },
      "source": [
        "##### NNLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "5b8HcTUBckj1"
      },
      "outputs": [],
      "source": [
        "eval_result_nnlm = embedding_eval_result('https://tfhub.dev/google/nnlm-en-dim128/1')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "n6hasLzFVrDN"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result_nnlm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1I4xEDNq8T0X"
      },
      "source": [
        "##### Universal Sentence Encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GrdweWRkck8A"
      },
      "outputs": [],
      "source": [
        "eval_result_use = embedding_eval_result('https://tfhub.dev/google/universal-sentence-encoder/2')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "JBABAkZMVtTK"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result_use)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "O2AnFO6Hcofi"
      },
      "source": [
        "## Exercises\n",
        "1. Pick an identity category, such as religion or sexual orientation, and look at False Positive Rate for the Universal Sentence Encoder. How do different slices compare to each other? How do they compare to the Overall baseline?\n",
        "2. Now pick a different identity category. Compare the results of this category with the previous one. Does the model weigh one category as more \"toxic\" than the other? Does this change with the embedding used?\n",
        "3. Does the model generally tend to overestimate or underestimate the number of toxic comments?\n",
        "4. Look at the graphs for different fairness metrics. Which metrics seem most informative? Which embeddings perform best and worst for that metric?\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Fairness Indicators on TF-Hub Text Embeddings - External",
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 2",
      "name": "python2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
{'fraction_non_alphanumeric': 0.15166956665838097, 'fraction_numerical': 0.0083271190653741, 'mean_word_length': 2.2593842830137727, 'pattern_counts': {'":': 271, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 33, 'lorem ipsum': 0, 'www.': 15, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4139338', 'n_tokens_mistral': 8098, 'n_tokens_neox': 7796, 'n_words': 1621}