<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction #\n",
    "\n",
    "*stochastic gradient descent*\n",
    "\n",
    "In deep learning, SGD isn't just a technical distraction. It's something that you have a great deal of control over, and your skill in controlling it will determine your success in real deep learning problems.\n",
    "\n",
    "The reason we need to know about SGD \n",
    "\n",
    "# The Loss Surface #\n",
    "\n",
    "Let's look at the training animation from Lesson 1 again:\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/rFI1tIk.gif\" width=\"1600\" alt=\"Fitting a line batch by batch. The loss decreases and the weights approach their true values.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Training a neural network with Stochastic Gradient Descent.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "Recall from Lesson 1 that to train a network, we use samples from the training data called *batches* (or *minibatches*) to make small shifts to the weights in the direction of their true values. This process is what we called *stochastic gradient descent*.\n",
    "\n",
    "To help us understand the SGD procedure we're going to plot something called a **loss surface**. The loss surface will show us exactly how the loss on the dataset depends on the weights we give the model. Let's see the training again, but this time with the loss surface.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/.gif\" width=\"800\" alt=\"The weights traverse the loss surface in the direction of their true values as the training progresses.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>The Loss Surface of the linear model\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "How are we to understand this loss surface? A point on the loss surface is a weight and a bias $(w, b)$. To get the loss value, plot the corresponding line and measure the error between the line and the points. That's the value of the contour.\n",
    "\n",
    "Do you see how the path the weights trace out during training corresponds to the fitted line? And look at the values on the contour lines. These are the loss values. Do you see how the path goes towards the minimum loss value as the line gets a better fit?\n",
    "\n",
    "The path the weights trace out is the **optimization path**.\n",
    "\n",
    "<!-- blockquote: contour plots -->\n",
    "\n",
    "# What's in a Name? #\n",
    "\n",
    "Let's use this loss surface to understand SGD.\n",
    "\n",
    "The **gradient** is simply a vector that tells us what direction the weights need to go. More precisely, it will tell us in what direction the loss is changing *fastest*. On the loss surface, this is where the contour lines are closest together. We call our process gradient **descent** because we use the gradient to *descend* the loss surface toward a minimum.\n",
    "\n",
    "<!-- FIGURE: the gradient along contour lines -->\n",
    "\n",
    "Now, the word **stochastic** simply means \"determined by chance.\" So in SGD, this just refers to our use of randomly sampled minibatches during training. If instead of random minibatches, we used the whole dataset at once, we would then have just ordinary gradient descent.\n",
    "\n",
    "And now you know what SGD means!\n",
    "\n",
    "# Batch Size and Learning Rate #\n",
    "\n",
    "When training a neural network, there are two parameters you need to choose that will have a large effect on the outcome: the *batch size* and the *learning rate*. These two parameters are what largely determine the behavior of the optimization path.\n",
    "\n",
    "The batch size, as you know, is how large of a sample each minibatch is. During SGD, we don't actually find the exact gradient vector. TensorFlow instead will *estimate* the gradient using a minibatch. If you've had some statistics, you probably know that the size of a sample determines how much noise there is in an estimate. So, size of the minibatch determines how much noise there is in the gradient.\n",
    "\n",
    "<!-- FIGURE: gradient noise and batch size -->\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example - Red Wine Quality #\n",
    "\n",
    "Show bad batch size/learning rate. Then show how to improve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate Schedules ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion #\n",
    "\n",
    "# Your Turn #"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>Revise tutorial 3<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Loss Function #\n",
    "\n",
    "The **loss function** measures the disparity between the true value of the target and the value the model predicts. During training, the model will use the loss function as a guide for finding the correct values of its weights. It tells the network its objective, \"where it's supposed to go.\"\n",
    "\n",
    "Different problems call for different loss functions. or this course, we'll use the **mean absolute error** or **MAE**. For each prediction `y_pred`, MAE measures the disparity from the true target `y_true` using `abs(y_true - y_pred)`.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/.png\" width=\"300\" alt=\"The graph of the MAE function, a 'V'.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>MAE takes the absolute value of their difference.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "The total MAE loss on a dataset would be the mean of all those absolute differences. In the illustration, it would be the average length of all the red bars.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/.png\" width=\"300\" alt=\"The graph of the MAE function, a 'V'.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the Network #\n",
    "\n",
    "When we first create a neural network, all of its weights are set randomly -- it doesn't \"know\" anything yet. Training the network means finding the right values for the weights, the values that minimize the loss.\n",
    "\n",
    "The way we train a network is through an iterative process called **stochastic gradient descent**. One *step* of training goes like this:\n",
    "1. Sample some training data and run it through the network to make predictions.\n",
    "2. Measure the loss between the predictions and the true values.\n",
    "3. Finally, adjust the weights in a direction that makes the loss smaller.\n",
    "\n",
    "Then just do this over and over until the loss is as small as you like (or until it won't decrease any further.)\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/rFI1tIk.gif\" width=\"1600\" alt=\"Fitting a line batch by batch. The loss decreases and the weights approach their true values.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Training a neural network with Stochastic Gradient Descent.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "Each iteration's sample of training data is called a **minibatch** (or often just \"batch\"), while a complete round of the training data is called an **epoch**. The number of epochs you train for is how many times the network will see each training example.\n",
    "\n",
    "The animation shows the linear model from Lesson 1 being trained with SGD. The pale red dots depict the entire training set, while the solid red dots are the minibatches. Every time SGD sees a new minibatch, it will shift the weights (`w` the slope and `b` the y-intercept) toward their correct values on that batch. Batch after batch, the line eventually converges to its best fit. You can see that the loss gets smaller as the weights get closer to their true values.\n",
    "\n",
    "Notice that the line only makes a small shift in the direction of each batch (instead of moving all the way). The size of these shifts is determined by the **learning rate**. A smaller learning rate means the network needs to see more minibatches before its weights converge to their best values.\n",
    "\n",
    "The learning rate and the size of the minibatches are the two parameters that have the largest effect on how the SGD training proceeds. Their interaction is often subtle and the right choice for these parameters isn't always obvious. We'll explore these effects in the exercises.\n",
    "\n",
    "Fortunately, for most work, it won't be necessary to do an extensive hyperparameter search to get satisfactory results. There have been a number of modifications to the original SGD algorithm that are easier to use. The variant that we'll use most is called **Adam**. It's thought to perform well in most situations and typically doesn't require hyperparamter tuning. It's a great general-purpose algorithm.\n",
    "\n",
    "After defining a model, you can add a loss function and optimizer with the `compile` method:\n",
    "\n",
    "```\n",
    "model.compile(\n",
    "    optimizer=\"adam\",\n",
    "    loss=\"mse\",\n",
    ")\n",
    "```\n",
    "\n",
    "Notice that we are able to specify these with just a string. You can also access these directly through the Keras API -- if you wanted to tune paramters, for instance -- but for us, the defaults will work fine.\n",
    "\n",
    "<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n",
    "    <strong>What's In a Name?</strong><br>\n",
    "The <strong>gradient</strong> is a vector that tells us in what direction the weights need to go. More precisely, it tells us how to change the weights to make the loss change <em>fastest</em>. We call our process gradient <strong>descent</strong> because it uses the gradient to <em>descend</em> the loss curve towards a minimum. <strong>Stochastic</strong> means \"determined by chance.\" Our training is <em>stochastic</em> because the minibatches are <em>random samples</em> from the dataset. And that's why it's called SGD!\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example - Red Wine Quality #\n",
    "\n",
    "**TODO - discussion**\n",
    "\n",
    "Now let's see it in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "#$HIDE_INPUT$\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "\n",
    "red_wine = pd.read_csv('../input/dl-course-data/dl-course-data/red-wine.csv')\n",
    "\n",
    "# Create training and validation splits\n",
    "df_train = red_wine.sample(frac=0.7, random_state=0)\n",
    "df_valid = red_wine.drop(df_train.index)\n",
    "display(df_train.head(4))\n",
    "\n",
    "# Scale to [0, 1]\n",
    "max_ = df_train.max(axis=0)\n",
    "min_ = df_train.min(axis=0)\n",
    "df_train = (df_train - min_) / (max_ - min_)\n",
    "df_valid = (df_valid - min_) / (max_ - min_)\n",
    "\n",
    "# Split features and target\n",
    "X_train = df_train.drop('quality', axis=1)\n",
    "X_valid = df_valid.drop('quality', axis=1)\n",
    "y_train = df_train['quality']\n",
    "y_valid = df_valid['quality']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many inputs? We're looking at columns (not including the target `quality`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "print (X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now make the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation='relu', input_shape=[11]),\n",
    "    layers.Dense(512, activation='relu'),\n",
    "    layers.Dense(512, activation='relu'),\n",
    "    layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include the loss and optimization algorithm in `compile`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='mae',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you've defined the network architecture and compiled it with an optimizer and loss function, you're ready to start training.\n",
    "\n",
    "A few things to note\n",
    "- validation data\n",
    "- batch size\n",
    "- epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "    X_train, y_train,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    batch_size=256,\n",
    "    epochs=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the \"learning curves\" in the `History` object. We'll learn more about these in the next lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "history_df = pd.DataFrame(history.history)\n",
    "history_df.loc[:, ['loss', 'val_loss']].plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion #"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.15307182751006623, 'fraction_numerical': 0.005390310429925964, 'mean_word_length': 3.083532219570406, 'pattern_counts': {'":': 117, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6948923', 'n_tokens_mistral': 4927, 'n_tokens_neox': 4763, 'n_words': 1800}