<commit_before>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "csv.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "DweYe9FcbMK_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "AVV2e0XKbJeX",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sUtoed20cRJJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Load CSV with tf.data"
      ]
    },
    {
      "metadata": {
        "id": "1ap_W4aQcgNT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/alpha/tutorials/load_data/text\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/load_data/text.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/load_data/text.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "metadata": {
        "id": "fgZ9gjmPfSnK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup"
      ]
    },
    {
      "metadata": {
        "id": "I4dwMQVQMQWD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install tensorflow==2.0.0-alpha0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "baYFZMW_bJHh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from __future__ import absolute_import, division, print_function\n",
        "\n",
        "import requests\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ncf5t6tgL5ZI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "TRAIN_DATA_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\"\n",
        "TEST_DATA_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test\"\n",
        "\n",
        "train_file_path = tf.keras.utils.get_file(\"adults.data\", TRAIN_DATA_URL)\n",
        "test_file_path = tf.keras.utils.get_file(\"adults.test\", TEST_DATA_URL)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4ONE94qulk6S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Make numpy values easier to read.\n",
        "np.set_printoptions(precision=3, suppress=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Wuqj601Qw0Ml",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load data\n",
        "\n",
        "So we know what we're doing, lets look at the top of the CSV file we're working with."
      ]
    },
    {
      "metadata": {
        "id": "54Dv7mCrf9Yw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!head {train_file_path}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZS-bt1LvWn2x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As you can see, the columns in the CSV are not labeled. The labels need to be supplied as a list of strings when creating a dataset from an unlabeled CSV file.\n",
        "\n",
        "If the file you are working with contains the column names in the first line, omit the `column_names` argument from the `make_csv_dataset` function. The constructor will then get the names from the file.\n"
      ]
    },
    {
      "metadata": {
        "id": "v0sLG216MtwT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# CSV columns in the input file.\n",
        "CSV_COLUMNS = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
        "               'marital_status', 'occupation', 'relationship', 'race', 'gender',\n",
        "               'capital_gain', 'capital_loss', 'hours_per_week',\n",
        "               'native_country', 'income_bracket']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gP4YYX9rYqiO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This example is going to use a subset of the available columns (we're omitting `fnlwgt` and `native_country`). To specify which columns to use, pass a list of column names in the `select_columns` argument of the constructor.\n",
        "\n",
        "We can also pass in a set of default values to use for these columns, in case any rows in the original data have empty values."
      ]
    },
    {
      "metadata": {
        "id": "UhOVoXXlYq54",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "drop_columns = ['fnlwgt', 'native_country']\n",
        "USED_COLUMNS = [column for column in CSV_COLUMNS if column not in drop_columns]\n",
        "\n",
        "USED_COLUMN_DEFAULTS = [[0], [''], [''], [0], [''], [''], [''], [''], [''], [0],\n",
        "                        [0], [0], ['']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "67mfwr4v-mN_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We also have to identify what column will serve as the labels for each example, and what those labels are."
      ]
    },
    {
      "metadata": {
        "id": "iXROZm5f3V4E",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "LABELS = ['<=50K', '>50K']\n",
        "LABEL_COLUMN = 'income_bracket'\n",
        "\n",
        "FEATURE_COLUMNS = [column for column in USED_COLUMNS if column != LABEL_COLUMN]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t4N-plO4tDXd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that these constructor argument values are in place,  read the CSV data from the file and create a dataset. The arguments we haven't mentioned are:\n",
        "\n",
        "-  `batch_size` — the number of (example, label) pairs that will be combined into each element of the dataset \n",
        "-  `na_value` — a string to represent NA or NaN values\n",
        "-  `num_epochs` — an int specifying the number of times this dataset is repeated\n",
        "-  `ignore_errors` — if true, malformed rows are discarded\n",
        "\n",
        "(For the full documentation, see `tf.data.experimental.make_csv_dataset`)\n"
      ]
    },
    {
      "metadata": {
        "id": "Co7UJ7gpNADC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_dataset(file_path):\n",
        "  dataset = tf.data.experimental.make_csv_dataset(\n",
        "      file_path,\n",
        "      batch_size=64, # Artificially small to make examples easier to show.\n",
        "      column_names=CSV_COLUMNS,\n",
        "      label_name=LABEL_COLUMN,\n",
        "      select_columns=USED_COLUMNS,\n",
        "      column_defaults=USED_COLUMN_DEFAULTS,\n",
        "      na_value=\"?\",\n",
        "      num_epochs=1,\n",
        "      ignore_errors=True)\n",
        "  return dataset\n",
        "\n",
        "raw_train_data = get_dataset(train_file_path)\n",
        "raw_test_data = get_dataset(test_file_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vHUQFKoQI6G7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Each item in the dataset is a batch, represented as a tuple of (*many examples*, *many labels*). The data from the examples is organized in column-based tensors (rather than, for example, row-based tensors). \n",
        "\n",
        "It might help to see this yourself."
      ]
    },
    {
      "metadata": {
        "id": "qWtFYtwXIeuj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "examples, labels = next(iter(raw_train_data)) # Just the first batch of 64.\n",
        "print(\"EXAMPLES: \\n\", examples, \"\\n\")\n",
        "print(\"LABELS: \\n\", labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9cryz31lxs3e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "metadata": {
        "id": "tSyrkSQwYHKi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Categorical data\n",
        "\n",
        "Some of the columns in the CSV data are categorical columns. That is, the content should be one of a limited set of options.\n",
        "\n",
        "In the CSV, these options are represented as text. This text needs to be converted to integers before the model can be trained. To facilitate that, we need to create a list of categorical columns, along with a list of the options available in each column."
      ]
    },
    {
      "metadata": {
        "id": "mWDniduKMw-C",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "CATEGORIES = {\n",
        "    'education': ['Bachelors', 'Some-college', '11th', 'HS-grad', 'Prof-school',\n",
        "                  'Assoc-acdm', 'Assoc-voc', '9th', '7th-8th', '12th',\n",
        "                  'Masters', '1st-4th', '10th', 'Doctorate', '5th-6th',\n",
        "                  'Preschool'],\n",
        "    'marital_status': ['Married-civ-spouse', 'Divorced', 'Never-married',\n",
        "                       'Separated', 'Widowed', 'Married-spouse-absent',\n",
        "                       'Married-AF-spouse'],\n",
        "    'relationship': ['Wife', 'Own-child', 'Husband', 'Not-in-family',\n",
        "                     'Other-relative', 'Unmarried'],\n",
        "    'workclass': ['Private', 'Self-emp-not-inc', 'Self-emp-inc', 'Federal-gov',\n",
        "                  'Local-gov', 'State-gov', 'Without-pay', 'Never-worked'],\n",
        "    'occupation': ['Tech-support', 'Craft-repair', 'Other-service', 'Sales',\n",
        "                   'Exec-managerial', 'Prof-specialty', 'Handlers-cleaners',\n",
        "                   'Machine-op-inspct', 'Adm-clerical', 'Farming-fishing',\n",
        "                   'Transport-moving', 'Priv-house-serv', 'Protective-serv',\n",
        "                   'Armed-Forces'],\n",
        "    'gender': ['Male', 'Female'],\n",
        "    'race': ['White', 'Asian-Pac-Islander', 'Amer-Indian-Eskimo', 'Other',\n",
        "             'Black'],\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8Ii0YWsoKBVx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Write a function that takes a tensor of categorical values, matches it to a list of value names, and then performs a one-hot encoding."
      ]
    },
    {
      "metadata": {
        "id": "bP02_BflkDbv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def process_categorical_data(data, categories):\n",
        "  \"\"\"Returns a one-hot encoded tensor representing categorical values.\"\"\"\n",
        "  \n",
        "  # Remove leading ' '.\n",
        "  data = tf.strings.regex_replace(data, '^ ', '')\n",
        "  # Remove trailing '.'.\n",
        "  data = tf.strings.regex_replace(data, r'\\.$', '')\n",
        "  \n",
        "  # ONE HOT ENCODE\n",
        "  # Reshape data from 1d (a list) to a 2d (a list of one-element lists)\n",
        "  data = tf.reshape(data, [-1, 1])\n",
        "  # For each element, create a new list of boolean values the length of categories,\n",
        "  # where the truth value is element == category label\n",
        "  data = tf.equal(categories, data)\n",
        "  # Cast booleans to floats.\n",
        "  data = tf.cast(data, tf.float32)\n",
        "  \n",
        "  # The entire encoding can fit on one line:\n",
        "  # data = tf.cast(tf.equal(categories, tf.reshape(data, [-1, 1])), tf.float32)\n",
        "  return data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "To2qbBGGMO1D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To help you visualize this, we'll take a single category-column tensor from the first batch, preprocess it, and show the before and after state."
      ]
    },
    {
      "metadata": {
        "id": "Ds7MOLMkK2Gf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "workclass_tensor = examples['workclass']\n",
        "workclass_tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HdDUSgpoTKfA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "workclass_categories = CATEGORIES['workclass']\n",
        "workclass_categories"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yHQeR47_ObpT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "processed_workclass = process_categorical_data(workclass_tensor, workclass_categories)\n",
        "processed_workclass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ACkc_cCaTuos",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Notice the relationship between the lengths of the two inputs and the shape of the output."
      ]
    },
    {
      "metadata": {
        "id": "gvvXM8m0T00O",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(\"Size of batch: \", len(workclass_tensor.numpy()))\n",
        "print(\"Number of category labels: \", len(workclass_categories))\n",
        "print(\"Shape of one-hot encoded tensor: \", processed_workclass.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9AsbaFmCeJtF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Continuous data"
      ]
    },
    {
      "metadata": {
        "id": "o2maE8d2ijsq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Continuous data needs to be normalized, so that the values fall between 0 and 1. To do that, write a function that multiplies each value by 1 over twice the mean of the column values.\n",
        "\n",
        "The function should also reshape the data into a two dimensional tensor.\n"
      ]
    },
    {
      "metadata": {
        "id": "IwGOy61lkQw-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def process_continuous_data(data, mean):\n",
        "  # Normalize data\n",
        "  data = tf.cast(data, tf.float32) * 1/(2*mean)\n",
        "  return tf.reshape(data, [-1, 1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0Yh8R7BujTAu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To do this calculation, you need the column means. You would obviously need to compute these in real life, but for this example we'll just provide them."
      ]
    },
    {
      "metadata": {
        "id": "-wk-BS1WMzHf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "MEANS = {\n",
        "    'age': 38.64358543876172,\n",
        "    'education_num': 10.078088530363212,\n",
        "    'capital_gain': 1079.0676262233324,\n",
        "    'capital_loss': 87.50231358257237,\n",
        "    'hours_per_week': 40.422382375824085,\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "raZtRlmaj-A5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Again, to see what this function is actually doing, we'll take a single tensor of continuous data and show it before and after processing."
      ]
    },
    {
      "metadata": {
        "id": "G-t_RSBrM2Vm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "age_tensor = examples['age']\n",
        "age_tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "M9lMLaEsjq3K",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "process_continuous_data(age_tensor, MEANS['age'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kPWkC4_1l3IG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Preprocess the data"
      ]
    },
    {
      "metadata": {
        "id": "jIvyqVAXmsN4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now assemble these preprocessing tasks into a single function that can be mapped to each batch in the dataset. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "rMxEHN0SNPkC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def preprocess(features, labels):\n",
        "  \n",
        "  # Process categorial features.\n",
        "  for feature in CATEGORIES.keys():\n",
        "    features[feature] = process_categorical_data(features[feature],\n",
        "                                                 CATEGORIES[feature])\n",
        "\n",
        "  # Process continuous features.\n",
        "  for feature in MEANS.keys():\n",
        "    features[feature] = process_continuous_data(features[feature],\n",
        "                                                MEANS[feature])\n",
        "\n",
        "  # Process the labels. (Labels are also categorical.)\n",
        "  labels = process_categorical_data(labels, LABELS)\n",
        "  \n",
        "  # Assemble features into a single tensor.\n",
        "  features = tf.concat([features[column] for column in FEATURE_COLUMNS], 1)\n",
        "  \n",
        "  return features, labels\n",
        "\n",
        "train_data = raw_train_data.map(preprocess)\n",
        "test_data = raw_test_data.map(preprocess)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IQOWatzRr2aF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "And let's see what a single example looks like."
      ]
    },
    {
      "metadata": {
        "id": "Gc1o9ZpCsGGM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "examples, labels = next(iter(train_data))\n",
        "\n",
        "examples, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aJnOromrse57",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The examples and labels are both two dimensional arrays of 64 items each (the batch size). Each item represents a single row in the original CSV file."
      ]
    },
    {
      "metadata": {
        "id": "DlF_omQqtnOP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Build the model"
      ]
    },
    {
      "metadata": {
        "id": "lQoFh16LxtT_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This example uses the [Keras Functional API](https://www.tensorflow.org/alpha/guide/keras/functional) wrapped in a `get_model` constructor to build up a simple model. "
      ]
    },
    {
      "metadata": {
        "id": "JDM3FIgHNCW3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_model(input_dim, labels_dim, hidden_units=[100], learning_rate=0.01):\n",
        "  \"\"\"Create a Keras model with layers.\n",
        "\n",
        "  Args:\n",
        "    input_dim: (int) The shape of an item in a batch. \n",
        "    labels_dim: (int) The shape of a label.\n",
        "    hidden_units: [int] the layer sizes of the DNN (input layer first)\n",
        "    learning_rate: (float) the learning rate for the optimizer.\n",
        "\n",
        "  Returns:\n",
        "    A Keras model.\n",
        "  \"\"\"\n",
        "\n",
        "  inputs = tf.keras.Input(shape=(input_dim,))\n",
        "  x = inputs\n",
        "\n",
        "  for units in hidden_units:\n",
        "    x = tf.keras.layers.Dense(units, activation=tf.keras.backend.relu)(x)\n",
        "  outputs = tf.keras.layers.Dense(labels_dim, activation='softmax')(x)\n",
        "\n",
        "  model = tf.keras.Model(inputs, outputs)\n",
        "  model.compile(\n",
        "      loss='categorical_crossentropy',\n",
        "      optimizer=tf.keras.optimizers.RMSprop(learning_rate),\n",
        "      metrics=['accuracy'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ce9PRb_LzFpm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The `get_model` constructor needs to know the input and output shapes of your data (not including the batch size)."
      ]
    },
    {
      "metadata": {
        "id": "qX-DU34ZuKJX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_shape, output_shape = train_data.output_shapes\n",
        "\n",
        "input_dimension = input_shape.dims[1] # [0] is the batch size\n",
        "output_dimension = output_shape.dims[1] "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hPdtI2ie0lEZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train, evaluate, and predict"
      ]
    },
    {
      "metadata": {
        "id": "8gvw1RE9zXkD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now the model can be instantiated and trained."
      ]
    },
    {
      "metadata": {
        "id": "Q_nm28IzNDTO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model = get_model(input_dimension, output_dimension)\n",
        "\n",
        "\n",
        "model.fit(train_data, epochs=20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QyDMgBurzqQo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once the model is trained, we can check its accuract on the `test_data` set."
      ]
    },
    {
      "metadata": {
        "id": "eB3R3ViVONOp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.evaluate(test_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sTrn_pD90gdJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In production, you want to actually get the output. Use `tf.keras.Model.predict` to infer labels on a batch or a dataset of batches."
      ]
    },
    {
      "metadata": {
        "id": "IOtzjj-RZqDG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "predictions = model.predict(test_data)\n",
        "\n",
        "print(\"Predictions:\\n\", predictions)\n",
        "print(\"\\nShape:\\n\", predictions.shape)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "opigysDf4_Nz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The two values represent the certainty of the model in assiging either of the two labels."
      ]
    }
  ]
}<commit_msg>probably done, with titanic dataset<commit_after>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "csv.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "DweYe9FcbMK_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "AVV2e0XKbJeX",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sUtoed20cRJJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Load CSV with tf.data"
      ]
    },
    {
      "metadata": {
        "id": "1ap_W4aQcgNT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/alpha/tutorials/load_data/text\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/load_data/text.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/load_data/text.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "metadata": {
        "id": "C-3Xbt0FfGfs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This tutorial provides an example of how to load CSV data from a file into a `tf.data.Dataset`.\n",
        "\n",
        "The data used in this tutorial is from the [UCI Machine Learning Repository Adult Data Set](https://archive.ics.uci.edu/ml/datasets/Adult), which provides details from the 1994 U.S. Census. The task is to predict whether a person makes over \\$50,000/year."
      ]
    },
    {
      "metadata": {
        "id": "fgZ9gjmPfSnK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup"
      ]
    },
    {
      "metadata": {
        "id": "I4dwMQVQMQWD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install tensorflow==2.0.0-alpha0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "baYFZMW_bJHh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from __future__ import absolute_import, division, print_function\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ncf5t6tgL5ZI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "TRAIN_DATA_URL = \"https://storage.googleapis.com/tf-datasets/titanic/train.csv\"\n",
        "TEST_DATA_URL = \"https://storage.googleapis.com/tf-datasets/titanic/eval.csv\"\n",
        "\n",
        "train_file_path = tf.keras.utils.get_file(\"train.csv\", TRAIN_DATA_URL)\n",
        "test_file_path = tf.keras.utils.get_file(\"eval.csv\", TEST_DATA_URL)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4ONE94qulk6S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Make numpy values easier to read.\n",
        "np.set_printoptions(precision=3, suppress=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Wuqj601Qw0Ml",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load data\n",
        "\n",
        "So we know what we're doing, lets look at the top of the CSV file we're working with."
      ]
    },
    {
      "metadata": {
        "id": "54Dv7mCrf9Yw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!head {train_file_path}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZS-bt1LvWn2x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As you can see, the columns in the CSV are labeled. The dataset constructor will pick these labels up automatically.\n",
        "\n",
        "If the file you are working with does not contain the column names in the first line, pass them in a list of strings to  the `column_names` argument in the `make_csv_dataset` function.\n"
      ]
    },
    {
      "metadata": {
        "id": "v0sLG216MtwT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# CSV columns in the input file.\n",
        "with open(train_file_path, 'r') as f:\n",
        "    names_row = f.readline()\n",
        "\n",
        "\n",
        "CSV_COLUMNS = names_row.rstrip('\\n').split(',')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AI9uJHzqQ5m1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "CSV_COLUMNS"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gZfhoX7bR9u4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This example is going to use all the available columns. If you need to omit some columns from the dataset, create a list of just the columns you plan to use, and pass it into the (optional) `select_columns` argument of the constructor.\n",
        "\n",
        "\n",
        "```python\n",
        "\n",
        "drop_columns = ['fare', 'embark_town']\n",
        "columns_to_use = [col for col in CSV_COLUMNS if col not in drop_columns]\n",
        "\n",
        "dataset = tf.data.experimental.make_csv_dataset(\n",
        "  ...,\n",
        "  select_columns = columns_to_use, \n",
        "  ...)\n",
        "\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "67mfwr4v-mN_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We also have to identify which column will serve as the labels for each example, and what those labels are."
      ]
    },
    {
      "metadata": {
        "id": "iXROZm5f3V4E",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "LABELS = [0, 1]\n",
        "LABEL_COLUMN = 'survived'\n",
        "\n",
        "FEATURE_COLUMNS = [column for column in CSV_COLUMNS if column != LABEL_COLUMN]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t4N-plO4tDXd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that these constructor argument values are in place,  read the CSV data from the file and create a dataset. The arguments we haven't mentioned are:\n",
        "\n",
        "-  `batch_size` — the number of (example, label) pairs that will be combined into each element of the dataset \n",
        "-  `na_value` — a string to represent NA or NaN values\n",
        "-  `num_epochs` — an int specifying the number of times this dataset is repeated\n",
        "-  `ignore_errors` — if true, malformed rows are discarded\n",
        "\n",
        "(For the full documentation, see `tf.data.experimental.make_csv_dataset`)\n"
      ]
    },
    {
      "metadata": {
        "id": "Co7UJ7gpNADC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_dataset(file_path):\n",
        "  dataset = tf.data.experimental.make_csv_dataset(\n",
        "      file_path,\n",
        "      batch_size=12, # Artificially small to make examples easier to show.\n",
        "      # column_names=CSV_COLUMNS, # not needed; CSV file has column names\n",
        "      label_name=LABEL_COLUMN,\n",
        "      # select_columns=columns_to_use, # not needed; using all columns\n",
        "      #column_defaults=COLUMN_DEFAULTS,\n",
        "      na_value=\"?\",\n",
        "      num_epochs=1,\n",
        "      ignore_errors=True)\n",
        "  return dataset\n",
        "\n",
        "raw_train_data = get_dataset(train_file_path)\n",
        "raw_test_data = get_dataset(test_file_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vHUQFKoQI6G7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Each item in the dataset is a batch, represented as a tuple of (*many examples*, *many labels*). The data from the examples is organized in column-based tensors (rather than, for example, row-based tensors). \n",
        "\n",
        "It might help to see this yourself."
      ]
    },
    {
      "metadata": {
        "id": "qWtFYtwXIeuj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "examples, labels = next(iter(raw_train_data)) # Just the first batch.\n",
        "print(\"EXAMPLES: \\n\", examples, \"\\n\")\n",
        "print(\"LABELS: \\n\", labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9cryz31lxs3e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "metadata": {
        "id": "tSyrkSQwYHKi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Categorical data\n",
        "\n",
        "Some of the columns in the CSV data are categorical columns. That is, the content should be one of a limited set of options.\n",
        "\n",
        "In the CSV, these options are represented as text. This text needs to be converted to integers before the model can be trained. To facilitate that, we need to create a list of categorical columns, along with a list of the options available in each column."
      ]
    },
    {
      "metadata": {
        "id": "mWDniduKMw-C",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "CATEGORIES = {\n",
        "    'sex': ['male', 'female'],\n",
        "    'class' : ['First', 'Second', 'Third'],\n",
        "    'deck' : ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'],\n",
        "    'embark_town' : ['Cherbourg', 'Southhampton', 'Queenstown'],\n",
        "    'alone' : ['y', 'n']\n",
        "}\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8Ii0YWsoKBVx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Write a function that takes a tensor of categorical values, matches it to a list of value names, and then performs a one-hot encoding."
      ]
    },
    {
      "metadata": {
        "id": "bP02_BflkDbv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def process_categorical_data(data, categories):\n",
        "  \"\"\"Returns a one-hot encoded tensor representing categorical values.\"\"\"\n",
        "  \n",
        "  # Remove leading ' '.\n",
        "  data = tf.strings.regex_replace(data, '^ ', '')\n",
        "  # Remove trailing '.'.\n",
        "  data = tf.strings.regex_replace(data, r'\\.$', '')\n",
        "  \n",
        "  # ONE HOT ENCODE\n",
        "  # Reshape data from 1d (a list) to a 2d (a list of one-element lists)\n",
        "  data = tf.reshape(data, [-1, 1])\n",
        "  # For each element, create a new list of boolean values the length of categories,\n",
        "  # where the truth value is element == category label\n",
        "  data = tf.equal(categories, data)\n",
        "  # Cast booleans to floats.\n",
        "  data = tf.cast(data, tf.float32)\n",
        "  \n",
        "  # The entire encoding can fit on one line:\n",
        "  # data = tf.cast(tf.equal(categories, tf.reshape(data, [-1, 1])), tf.float32)\n",
        "  return data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "To2qbBGGMO1D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To help you visualize this, we'll take a single category-column tensor from the first batch, preprocess it, and show the before and after state."
      ]
    },
    {
      "metadata": {
        "id": "Ds7MOLMkK2Gf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_tensor = examples['class']\n",
        "class_tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HdDUSgpoTKfA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_categories = CATEGORIES['class']\n",
        "class_categories"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yHQeR47_ObpT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "processed_class = process_categorical_data(class_tensor, class_categories)\n",
        "processed_class"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ACkc_cCaTuos",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Notice the relationship between the lengths of the two inputs and the shape of the output."
      ]
    },
    {
      "metadata": {
        "id": "gvvXM8m0T00O",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(\"Size of batch: \", len(class_tensor.numpy()))\n",
        "print(\"Number of category labels: \", len(class_categories))\n",
        "print(\"Shape of one-hot encoded tensor: \", processed_class.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9AsbaFmCeJtF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Continuous data"
      ]
    },
    {
      "metadata": {
        "id": "o2maE8d2ijsq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Continuous data needs to be normalized, so that the values fall between 0 and 1. To do that, write a function that multiplies each value by 1 over twice the mean of the column values.\n",
        "\n",
        "The function should also reshape the data into a two dimensional tensor.\n"
      ]
    },
    {
      "metadata": {
        "id": "IwGOy61lkQw-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def process_continuous_data(data, mean):\n",
        "  # Normalize data\n",
        "  data = tf.cast(data, tf.float32) * 1/(2*mean)\n",
        "  return tf.reshape(data, [-1, 1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0Yh8R7BujTAu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To do this calculation, you need the column means. You would obviously need to compute these in real life, but for this example we'll just provide them."
      ]
    },
    {
      "metadata": {
        "id": "iNE_mTJqegGQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "MEANS = {\n",
        "    'age' : 29.631308,\n",
        "    'n_siblings_spouses' : 0.545455,\n",
        "    'parch' : 0.379585,\n",
        "    'fare' : 34.385399\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "raZtRlmaj-A5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Again, to see what this function is actually doing, we'll take a single tensor of continuous data and show it before and after processing."
      ]
    },
    {
      "metadata": {
        "id": "G-t_RSBrM2Vm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "age_tensor = examples['age']\n",
        "age_tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "M9lMLaEsjq3K",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "process_continuous_data(age_tensor, MEANS['age'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kPWkC4_1l3IG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Preprocess the data"
      ]
    },
    {
      "metadata": {
        "id": "jIvyqVAXmsN4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now assemble these preprocessing tasks into a single function that can be mapped to each batch in the dataset. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "rMxEHN0SNPkC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def preprocess(features, labels):\n",
        "  \n",
        "  # Process categorial features.\n",
        "  for feature in CATEGORIES.keys():\n",
        "    features[feature] = process_categorical_data(features[feature],\n",
        "                                                 CATEGORIES[feature])\n",
        "\n",
        "  # Process continuous features.\n",
        "  for feature in MEANS.keys():\n",
        "    features[feature] = process_continuous_data(features[feature],\n",
        "                                                MEANS[feature])\n",
        "\n",
        "  # Process the labels. (Labels are also categorical.)\n",
        "  #labels = process_categorical_data(labels, LABELS)\n",
        "  \n",
        "  # Assemble features into a single tensor.\n",
        "  features = tf.concat([features[column] for column in FEATURE_COLUMNS], 1)\n",
        "  \n",
        "  return features, labels\n",
        "\n",
        "train_data = raw_train_data.map(preprocess)\n",
        "test_data = raw_test_data.map(preprocess)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IQOWatzRr2aF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "And let's see what a single example looks like."
      ]
    },
    {
      "metadata": {
        "id": "Gc1o9ZpCsGGM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "examples, labels = next(iter(train_data))\n",
        "\n",
        "examples, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aJnOromrse57",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The examples and labels are both two dimensional arrays of 12 items each (the batch size). Each item represents a single row in the original CSV file."
      ]
    },
    {
      "metadata": {
        "id": "DlF_omQqtnOP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Build the model"
      ]
    },
    {
      "metadata": {
        "id": "lQoFh16LxtT_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This example uses the [Keras Functional API](https://www.tensorflow.org/alpha/guide/keras/functional) wrapped in a `get_model` constructor to build up a simple model. "
      ]
    },
    {
      "metadata": {
        "id": "JDM3FIgHNCW3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_model(input_dim, hidden_units=[100], learning_rate=0.01):\n",
        "  \"\"\"Create a Keras model with layers.\n",
        "\n",
        "  Args:\n",
        "    input_dim: (int) The shape of an item in a batch. \n",
        "    labels_dim: (int) The shape of a label.\n",
        "    hidden_units: [int] the layer sizes of the DNN (input layer first)\n",
        "    learning_rate: (float) the learning rate for the optimizer.\n",
        "\n",
        "  Returns:\n",
        "    A Keras model.\n",
        "  \"\"\"\n",
        "\n",
        "  inputs = tf.keras.Input(shape=(input_dim,))\n",
        "  x = inputs\n",
        "\n",
        "  for units in hidden_units:\n",
        "    x = tf.keras.layers.Dense(units, activation=tf.keras.backend.relu)(x)\n",
        "  outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n",
        "\n",
        "  model = tf.keras.Model(inputs, outputs)\n",
        "  model.compile(\n",
        "      loss='binary_crossentropy',\n",
        "      optimizer=tf.keras.optimizers.RMSprop(learning_rate),\n",
        "      metrics=['accuracy'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ce9PRb_LzFpm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The `get_model` constructor needs to know the input and output shapes of your data (not including the batch size)."
      ]
    },
    {
      "metadata": {
        "id": "qX-DU34ZuKJX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_shape, output_shape = train_data.output_shapes\n",
        "\n",
        "input_dimension = input_shape.dims[1] # [0] is the batch size"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hPdtI2ie0lEZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train, evaluate, and predict"
      ]
    },
    {
      "metadata": {
        "id": "8gvw1RE9zXkD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now the model can be instantiated and trained."
      ]
    },
    {
      "metadata": {
        "id": "Q_nm28IzNDTO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model = get_model(input_dimension, output_dimension)\n",
        "\n",
        "\n",
        "model.fit(train_data, epochs=20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QyDMgBurzqQo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once the model is trained, we can check its accuract on the `test_data` set."
      ]
    },
    {
      "metadata": {
        "id": "eB3R3ViVONOp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.evaluate(test_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sTrn_pD90gdJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In production, you'd want to actually get the output. Use `tf.keras.Model.predict` to infer labels on a batch or a dataset of batches."
      ]
    },
    {
      "metadata": {
        "id": "IOtzjj-RZqDG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "predictions = model.predict(test_data)\n",
        "\n",
        "print(\"Predictions:\\n\", predictions)\n",
        "print(\"\\nShape:\\n\", predictions.shape)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "opigysDf4_Nz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The values are the probability of the person represented by each row surviving the sinking of the Titanic.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "y5MKQBe-h-ni",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
{'fraction_non_alphanumeric': 0.1714317917120637, 'fraction_numerical': 0.010275387909982342, 'mean_word_length': 1.8034651640423403, 'pattern_counts': {'":': 797, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 21, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '810419', 'n_tokens_mistral': 18505, 'n_tokens_neox': 17861, 'n_words': 3959}