<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "description: Automatic Evaluation with Pre-implemented Metrics\n",
    "---\n",
    "\n",
    "# Evaluation\n",
    "\n",
    "Welcome to the \"_Evaluation_\" tutorial of the \"_From Zero to Hero_\" series. In this part we will present the functionalities offered by the `evaluation` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/ContinualAI/avalanche.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📈 The Evaluation Module\n",
    "\n",
    "\n",
    "\n",
    "The `evaluation` module is quite straightforward: it offers all the basic functionalities to evaluate and keep track of a continual learning experiment.\n",
    "\n",
    "This is mostly done through the **Metrics**: a set of classes which implement the main continual learning metrics computation like A_ccuracy_, F_orgetting_, M_emory Usage_, R_unning Times_, etc. At the moment, in _Avalanche_ we offer a number of pre-implemented metrics you can use for your own experiments. We made sure to include all the major accuracy-based metrics but also the ones related to computation and memory.\n",
    "\n",
    "Each metric comes with a standalone class and a set of plugin classes aimed at emitting metric values on specific moments during training and evaluation.\n",
    "\n",
    "#### Standalone metric\n",
    "\n",
    "As an example, the standalone `Accuracy` class can be used to monitor the average accuracy over a stream of `<input,target>` pairs. The class provides an `update` method to update the current average accuracy, a `result` method to print the current average accuracy and a `reset` method to set the current average accuracy to zero. The call to `result`does not change the metric state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Accuracy:  0.0\n",
      "Average Accuracy:  0.5\n",
      "Average Accuracy:  0.75\n",
      "After reset:  0.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from avalanche.evaluation.metrics import Accuracy\n",
    "\n",
    "# create an instance of the standalone Accuracy metric\n",
    "# initial accuracy is 0\n",
    "acc_metric = Accuracy()\n",
    "print(\"Initial Accuracy: \", acc_metric.result()) #  output 0\n",
    "\n",
    "# two consecutive metric updates\n",
    "real_y = torch.tensor([1, 2]).long()\n",
    "predicted_y = torch.tensor([1, 0]).float()\n",
    "acc_metric.update(real_y, predicted_y)\n",
    "acc = acc_metric.result()\n",
    "print(\"Average Accuracy: \", acc) # output 0.5\n",
    "predicted_y = torch.tensor([1,2]).float()\n",
    "acc_metric.update(real_y, predicted_y)\n",
    "acc = acc_metric.result()\n",
    "print(\"Average Accuracy: \", acc) # output 0.75\n",
    "\n",
    "# reset accuracy to 0\n",
    "acc_metric.reset()\n",
    "print(\"After reset: \", acc_metric.result()) # output 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plugin metric\n",
    "\n",
    "If you want to integrate the available metrics automatically in the training and evaluation flow, you can use plugion metrics, like `EpochAccuracy` which logs the accuracy after each training epoch, or `ExperienceAccuracy` which logs the accuracy after each evaluation experience. Each of these metrics emits a **curve** composed by its values at different points in time \\(e.g. on different training epochs\\).  In order to simplify the use of these metrics, we provided utility functions with which you can create different plugin metrics in one shot. The results of these functions can be passed as parameters directly to the `EvaluationPlugin`\\(see below\\).\n",
    "\n",
    "{% hint style=\"info\" %}\n",
    "We recommend to use the helper functions when creating plugin metrics.\n",
    "{% endhint %}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from avalanche.evaluation.metrics import accuracy_metrics, \\\n",<commit_msg>Tiny Typo<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "description: Automatic Evaluation with Pre-implemented Metrics\n",
    "---\n",
    "\n",
    "# Evaluation\n",
    "\n",
    "Welcome to the \"_Evaluation_\" tutorial of the \"_From Zero to Hero_\" series. In this part we will present the functionalities offered by the `evaluation` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/ContinualAI/avalanche.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📈 The Evaluation Module\n",
    "\n",
    "\n",
    "\n",
    "The `evaluation` module is quite straightforward: it offers all the basic functionalities to evaluate and keep track of a continual learning experiment.\n",
    "\n",
    "This is mostly done through the **Metrics**: a set of classes which implement the main continual learning metrics computation like A_ccuracy_, F_orgetting_, M_emory Usage_, R_unning Times_, etc. At the moment, in _Avalanche_ we offer a number of pre-implemented metrics you can use for your own experiments. We made sure to include all the major accuracy-based metrics but also the ones related to computation and memory.\n",
    "\n",
    "Each metric comes with a standalone class and a set of plugin classes aimed at emitting metric values on specific moments during training and evaluation.\n",
    "\n",
    "#### Standalone metric\n",
    "\n",
    "As an example, the standalone `Accuracy` class can be used to monitor the average accuracy over a stream of `<input,target>` pairs. The class provides an `update` method to update the current average accuracy, a `result` method to print the current average accuracy and a `reset` method to set the current average accuracy to zero. The call to `result`does not change the metric state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Accuracy:  0.0\n",
      "Average Accuracy:  0.5\n",
      "Average Accuracy:  0.75\n",
      "After reset:  0.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from avalanche.evaluation.metrics import Accuracy\n",
    "\n",
    "# create an instance of the standalone Accuracy metric\n",
    "# initial accuracy is 0\n",
    "acc_metric = Accuracy()\n",
    "print(\"Initial Accuracy: \", acc_metric.result()) #  output 0\n",
    "\n",
    "# two consecutive metric updates\n",
    "real_y = torch.tensor([1, 2]).long()\n",
    "predicted_y = torch.tensor([1, 0]).float()\n",
    "acc_metric.update(real_y, predicted_y)\n",
    "acc = acc_metric.result()\n",
    "print(\"Average Accuracy: \", acc) # output 0.5\n",
    "predicted_y = torch.tensor([1,2]).float()\n",
    "acc_metric.update(real_y, predicted_y)\n",
    "acc = acc_metric.result()\n",
    "print(\"Average Accuracy: \", acc) # output 0.75\n",
    "\n",
    "# reset accuracy to 0\n",
    "acc_metric.reset()\n",
    "print(\"After reset: \", acc_metric.result()) # output 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plugin metric\n",
    "\n",
    "If you want to integrate the available metrics automatically in the training and evaluation flow, you can use plugin metrics, like `EpochAccuracy` which logs the accuracy after each training epoch, or `ExperienceAccuracy` which logs the accuracy after each evaluation experience. Each of these metrics emits a **curve** composed by its values at different points in time \\(e.g. on different training epochs\\).  In order to simplify the use of these metrics, we provided utility functions with which you can create different plugin metrics in one shot. The results of these functions can be passed as parameters directly to the `EvaluationPlugin`\\(see below\\).\n",
    "\n",
    "{% hint style=\"info\" %}\n",
    "We recommend to use the helper functions when creating plugin metrics.\n",
    "{% endhint %}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from avalanche.evaluation.metrics import accuracy_metrics, \\\n",
{'fraction_non_alphanumeric': 0.15064705231722156, 'fraction_numerical': 0.005751576153080412, 'mean_word_length': 3.0008849557522126, 'pattern_counts': {'":': 72, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5213920', 'n_tokens_mistral': 2913, 'n_tokens_neox': 2802, 'n_words': 966}