<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--TITLE: Deep Classifiers -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction #\n",
    "\n",
    "So far in this course, we've learned about how neural networks can solve regression problems. Now we're going to apply neural networks to another common machine learning problem: classification. Most everything we've learned up until now still applies. The main difference is in the loss function we use and in what kind of outputs we want the final layer to produce.\n",
    "\n",
    "# Binary Classification #\n",
    "\n",
    "Classification into one of two classes is a common machine learning problem. You might want to predict whether or not a customer is likely to make a purchase, whether or not a credit card transaction was fraudulent, whether deep space signals show evidence of a new planet, or a medical test evidence of a disease. These are all **binary classification** problems.\n",
    "\n",
    "In your raw data, the classes might be represented by strings like `\"Yes\"` and `\"No\"`, or `\"Dog\"` and `\"Cat\"`. Before using this data we'll assign a **class label**: one class will be `0` and the other will be `1`. Assigning numeric labels puts the data in a form a neural network can use.\n",
    "\n",
    "# Accuracy and Cross-Entropy #\n",
    "\n",
    "**Accuracy** is one of the many metrics in use for measuring success on a classification problem. Accuracy is the ratio of correct predictions to total predictions: `accuracy = number_correct / total`. A model that always predicted correctly would have an accuracy score of `1.0`. All else being equal, accuracy is a reasonable metric to use whenever the classes in the dataset occur with about the same frequency.\n",
    "\n",
    "The problem with accuracy (and most other classification metrics) is that it can't be used as a loss function. SGD needs a loss function that changes smoothly, but accuracy, being a ratio of counts, changes in \"jumps\". So, we have to choose a substitute to act as the loss function. This substitute is the *cross-entropy* function.\n",
    "\n",
    "Now, recall that the loss function defines the *objective* of the network during training. With regression, our goal was to minimize the distance between the expected outcome and the predicted outcome. We chose MAE to measure this distance.\n",
    "\n",
    "For classification, what we want instead is a distance between *probabilities*, and this is what cross-entropy provides. **Cross-entropy** is a sort of measure for the distance from one probability distribution to another.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/DwVV9bR.png\" width=\"400\" alt=\"Graphs of accuracy and cross-entropy.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Cross-entropy penalizes incorrect probability predictions.</center></figcaption>\n",<commit_msg>Remove spurious md cell<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction #\n",
    "\n",
    "So far in this course, we've learned about how neural networks can solve regression problems. Now we're going to apply neural networks to another common machine learning problem: classification. Most everything we've learned up until now still applies. The main difference is in the loss function we use and in what kind of outputs we want the final layer to produce.\n",
    "\n",
    "# Binary Classification #\n",
    "\n",
    "Classification into one of two classes is a common machine learning problem. You might want to predict whether or not a customer is likely to make a purchase, whether or not a credit card transaction was fraudulent, whether deep space signals show evidence of a new planet, or a medical test evidence of a disease. These are all **binary classification** problems.\n",
    "\n",
    "In your raw data, the classes might be represented by strings like `\"Yes\"` and `\"No\"`, or `\"Dog\"` and `\"Cat\"`. Before using this data we'll assign a **class label**: one class will be `0` and the other will be `1`. Assigning numeric labels puts the data in a form a neural network can use.\n",
    "\n",
    "# Accuracy and Cross-Entropy #\n",
    "\n",
    "**Accuracy** is one of the many metrics in use for measuring success on a classification problem. Accuracy is the ratio of correct predictions to total predictions: `accuracy = number_correct / total`. A model that always predicted correctly would have an accuracy score of `1.0`. All else being equal, accuracy is a reasonable metric to use whenever the classes in the dataset occur with about the same frequency.\n",
    "\n",
    "The problem with accuracy (and most other classification metrics) is that it can't be used as a loss function. SGD needs a loss function that changes smoothly, but accuracy, being a ratio of counts, changes in \"jumps\". So, we have to choose a substitute to act as the loss function. This substitute is the *cross-entropy* function.\n",
    "\n",
    "Now, recall that the loss function defines the *objective* of the network during training. With regression, our goal was to minimize the distance between the expected outcome and the predicted outcome. We chose MAE to measure this distance.\n",
    "\n",
    "For classification, what we want instead is a distance between *probabilities*, and this is what cross-entropy provides. **Cross-entropy** is a sort of measure for the distance from one probability distribution to another.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/DwVV9bR.png\" width=\"400\" alt=\"Graphs of accuracy and cross-entropy.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Cross-entropy penalizes incorrect probability predictions.</center></figcaption>\n",
{'fraction_non_alphanumeric': 0.10264125710464728, 'fraction_numerical': 0.003009027081243731, 'mean_word_length': 3.7597454256165475, 'pattern_counts': {'":': 11, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3096979', 'n_tokens_mistral': 1540, 'n_tokens_neox': 1506, 'n_words': 831}