<commit_before><commit_msg>updated linear reg to work with tf.data<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Dataset API\n",
    "\n",
    "**Learning Objectives**\n",
    "    - Learn how use tf.datma to read data into TF models\n",
    "    - Understand difference between loading data entirely in-memory and loading in batches from disk\n",
    "    - Practice loading a .csv file from disk in batches using the tf.data module\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf-nightly-2.0-preview==2.0.0.dev20190906\r\n"
     ]
    }
   ],
   "source": [
    "# Ensure that we have Tensorflow 1.13.1 installed.\n",
    "!pip freeze | grep tf-nightly-2.0-preview || pip install tf-nightly-2.0-preview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-dev20190908\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression\n",
    "\n",
    "Now let's use low level tensorflow operations to implement linear regression.\n",
    "\n",
    "Later in the course you'll see abstracted ways to do this using high level TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toy Dataset\n",
    "\n",
    "We'll model the following function:\n",
    "\n",
    "\\begin{equation}\n",
    "y= 2x + 10\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_POINTS = 10\n",
    "X = tf.constant(range(N_POINTS), dtype=tf.float32)\n",
    "Y = 2 * X + 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataset(X, Y, epochs=5, batch_size=N_POINTS):\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((X, Y))\n",
    "    dataset = dataset.repeat(epochs).batch(batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = create_dataset(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] y: [10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]\n",
      "x: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] y: [10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]\n",
      "x: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] y: [10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]\n",
      "x: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] y: [10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]\n",
      "x: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] y: [10. 12. 14. 16. 18. 20. 22. 24. 26. 28.]\n"
     ]
    }
   ],
   "source": [
    "for i, (x, y) in enumerate(dataset):\n",
    "    print(\"x:\", x.numpy(), \"y:\", y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss Function\n",
    "\n",
    "Using mean squared error, our loss function is:\n",
    "\\begin{equation}\n",
    "MSE = \\frac{1}{m}\\sum_{i=1}^{m}(\\hat{Y}_i-Y_i)^2\n",
    "\\end{equation}\n",
    "\n",
    "$\\hat{Y}$ represents the vector containing our model's predictions:\n",
    "\\begin{equation}\n",
    "\\hat{Y} = w_0X + w_1\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_mse(X, Y, w0, w1):\n",
    "    Y_hat = w0 * X + w1\n",
    "    errors = (Y_hat - Y)**2\n",
    "    return tf.reduce_mean(errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Function\n",
    "\n",
    "To use gradient descent we need to take the partial derivatives of the loss function with respect to each of the weights. We could manually compute the derivatives, but with Tensorflow's automatic differentiation capabilities we don't have to!\n",
    "\n",
    "During gradient descent we think of the loss as a function of the parameters $w_0$ and $w_1$. Thus, we want to compute the partial derivative with respect to these variables. \n",
    "\n",
    "For that we need to wrap our loss computation within the context of `tf.GradientTape` instance which will reccord gradient information:\n",
    "\n",
    "```python\n",
    "with tf.GradientTape() as tape:\n",
    "    loss = # computation \n",
    "```\n",
    "\n",
    "This will allow us to later compute the gradients of any tensor computed within the `tf.GradientTape` context with respect to instances of `tf.Variable`:\n",
    "\n",
    "```python\n",
    "gradients = tape.gradient(loss, [w0, w1])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We illustrate this procedure with by computing the loss gradients with respect to the model weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradients(X, Y, w0, w1):\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss = loss_mse(X, Y, w0, w1)\n",
    "    return tape.gradient(loss, [w0, w1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "w0 = tf.Variable(0.0)\n",
    "w1 = tf.Variable(0.0)\n",
    "\n",
    "dw0, dw1 = compute_gradients(X, Y, w0, w1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dw0: -204.0\n"
     ]
    }
   ],
   "source": [
    "print(\"dw0:\", dw0.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dw1 -38.0\n"
     ]
    }
   ],
   "source": [
    "print(\"dw1\", dw1.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Loop\n",
    "\n",
    "Here we have a very simple training loop that converges. Note we are ignoring best practices like batching, creating a separate test set, and random weight initialization for the sake of simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STEP 0 - loss: 35.70719528198242, w0: 4.079999923706055, w1: 0.7599999904632568\n",
      "\n",
      "STEP 100 - loss: 2.6017532348632812, w0: 2.4780430793762207, w1: 7.002389907836914\n",
      "\n",
      "STEP 200 - loss: 0.26831889152526855, w0: 2.153517961502075, w1: 9.037351608276367\n",
      "\n",
      "STEP 300 - loss: 0.027671903371810913, w0: 2.0493006706237793, w1: 9.690855979919434\n",
      "\n",
      "STEP 400 - loss: 0.0028539239428937435, w0: 2.0158326625823975, w1: 9.90071964263916\n",
      "\n",
      "STEP 500 - loss: 0.0002943490108009428, w0: 2.005084753036499, w1: 9.96811580657959\n",
      "\n",
      "STEP 600 - loss: 3.0356444767676294e-05, w0: 2.0016329288482666, w1: 9.989760398864746\n",
      "\n",
      "STEP 700 - loss: 3.1322738323069643e-06, w0: 2.0005245208740234, w1: 9.996710777282715\n",
      "\n",
      "STEP 800 - loss: 3.2238213520940917e-07, w0: 2.0001683235168457, w1: 9.998944282531738\n",
      "\n",
      "STEP 900 - loss: 3.369950718479231e-08, w0: 2.000054359436035, w1: 9.999658584594727\n",
      "\n"
     ]
    }
   ],
   "source": [
    "STEPS = 1000\n",
    "BATCH_SIZE = 10\n",
    "LEARNING_RATE = .02\n",
    "MSG = \"STEP {step} - loss: {loss}, w0: {w0}, w1: {w1}\\n\"\n",
    "\n",
    "\n",
    "w0 = tf.Variable(0.0)\n",
    "w1 = tf.Variable(0.0)\n",
    "\n",
    "dataset = create_dataset(X, Y, epochs=EPOCHS, batch_size=BATCH_SIZE)\n",
    "\n",
    "for step, (X, Y) in enumerate(dataset):\n",
    "\n",
    "    dw0, dw1 = compute_gradients(X, Y, w0, w1)\n",
    "    w0.assign_sub(dw0 * LEARNING_RATE)\n",
    "    w1.assign_sub(dw1 * LEARNING_RATE)\n",
    "\n",
    "    if step % 100 == 0:\n",
    "        loss = loss_mse(X, Y, w0, w1)\n",
    "        print(MSG.format(step=step, loss=loss, w0=w0.numpy(), w1=w1.numpy()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try modelling a non-linear function such as: $y=xe^{-x^2}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.constant(np.linspace(0, 2, 1000), dtype=tf.float32)\n",
    "Y = X * tf.exp(-X**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1461c3828>]"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "plt.plot(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_features(X, Y):\n",
    "    f1 = tf.ones_like(X)  # Bias.\n",
    "    f2 = X\n",
    "    f3 = tf.square(X)\n",
    "    f4 = tf.sqrt(X)\n",
    "    f5 = tf.exp(X)\n",
    "    return X, Y\n",
    "    return tf.stack([f1, f2, f3, f4, f5], axis=1), Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataset(X, Y, epochs=5, batch_size=10):\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((X, Y))\n",
    "    dataset = dataset.map(make_features)\n",
    "    dataset = dataset.repeat(epochs).batch(batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <function make_features at 0x146d37488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <function make_features at 0x146d37488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
      "STEP 0 X: [0.06006006 0.06206206 0.06406406 0.06606606 0.06806806 0.07007007\n",
      " 0.07207207 0.07407407 0.07607608 0.07807808] Y: [0.0598438  0.06182348 0.06380167 0.06577833 0.06775342 0.06972688\n",
      " 0.07169867 0.07366875 0.07563705 0.07760355]\n",
      "STEP 1 X: [0.06006006 0.06206206 0.06406406 0.06606606 0.06806806 0.07007007\n",
      " 0.07207207 0.07407407 0.07607608 0.07807808] Y: [0.0598438  0.06182348 0.06380167 0.06577833 0.06775342 0.06972688\n",
      " 0.07169867 0.07366875 0.07563705 0.07760355]\n"
     ]
    }
   ],
   "source": [
    "dataset = create_dataset(X, Y)\n",
    "\n",
    "for step, (X, Y) in enumerate(dataset):\n",
    "    if step > 1: break\n",
    "    print(\"STEP\", step, \"X:\", X.numpy(), \"Y:\", Y.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, W):\n",
    "    return tf.squeeze(X @ W, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_mse(X, Y, W):\n",
    "    Y_hat = predict(X, W)\n",
    "    errors = (Y_hat - Y)**2\n",
    "    return tf.reduce_mean(errors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradients(X, Y, W):\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss = loss_mse(Xf, Y, W)\n",
    "    return tape.gradient(loss, W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STEP: 2000 MSE: 0.002376794582232833\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x13f634a90>"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "STEPS = 2000\n",
    "LEARNING_RATE = .02\n",
    "\n",
    "\n",
    "Xf = make_features(X)\n",
    "n_weights = Xf.shape[1]\n",
    "\n",
    "W = tf.Variable(np.zeros((n_weights, 1)), dtype=tf.float32)\n",
    "\n",
    "# For plotting\n",
    "steps, losses = [], []\n",
    "plt.figure()\n",
    "\n",
    "\n",
    "for step in range(1, STEPS + 1):\n",
    "\n",
    "    dW = compute_gradients(X, Y, W)\n",
    "    W.assign_sub(dW * LEARNING_RATE)\n",
    "\n",
    "    if step % 100 == 0:\n",
    "        loss = loss_mse(Xf, Y, W)\n",
    "        steps.append(step)\n",
    "        losses.append(loss)\n",
    "        plt.clf()\n",
    "        plt.plot(steps, losses)\n",
    "\n",
    "\n",
    "print(\"STEP: {} MSE: {}\".format(STEPS, loss_mse(Xf, Y, W)))\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(X, Y, label='actual')\n",
    "plt.plot(X, predict(Xf, W), label='predicted')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
{'fraction_non_alphanumeric': 0.19916986353059557, 'fraction_numerical': 0.08854789006980693, 'mean_word_length': 2.3414582895566296, 'pattern_counts': {'":': 210, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1693198', 'n_tokens_mistral': 7210, 'n_tokens_neox': 6225, 'n_words': 1429}