<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Perceptron Multicouche"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principales impl\u00e9mentations en Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Scikit-learn: http://scikit-learn.org/stable/modules/neural_networks_supervised.html\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes du livre Dunod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\newcommand{\\sigin}{x}$\n",
    "$\\sigin$: **TODO**\n",
    "\n",
    "$\\newcommand{\\pot}{p}$\n",
    "$\\pot$: *Potentiel (d'activation ?)* du neurone **TODO**\n",
    "\n",
    "$\\newcommand{\\wij}{w_{ij}}$\n",
    "$\\wij$: Poids de la connexion entre le neurone $j$ et le neurone $i$\n",
    "\n",
    "$\\newcommand{\\activthres}{\\theta}$\n",
    "$\\activthres$: *Seuil d'activation* du neurone\n",
    "\n",
    "$\\newcommand{\\activfunc}{f}$\n",
    "$\\activfunc$: *Fonction d'activation*\n",
    "\n",
    "$\\newcommand{\\errfunc}{E}$\n",
    "$\\errfunc$: *Fonction objectif* ou *fonction d'erreur*\n",
    "\n",
    "$\\newcommand{\\learnrate}{\\epsilon}$\n",
    "$\\learnrate$: *Pas d'apprentissage* ou *Taux d'apprentissage*\n",
    "\n",
    "$\\newcommand{\\learnit}{n}$\n",
    "$\\learnit$: **TODO**\n",
    "\n",
    "$\\newcommand{\\it}{t}$\n",
    "$\\it$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigincur}{x_i}$\n",
    "$\\sigincur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigout}{y}$\n",
    "$\\sigout$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutcur}{y_i}$\n",
    "$\\sigoutcur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutprev}{y_j}$\n",
    "$\\sigoutprev$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutdes}{d_i}$\n",
    "$\\sigoutdes$: **TODO**\n",
    "\n",
    "$\\newcommand{\\weights}{\\boldsymbol{W}}$\n",
    "$\\weights$: **TODO**\n",
    "\n",
    "$\\newcommand{\\weight}{w}$\n",
    "$\\weight$: **TODO**\n",
    "\n",
    "$\\newcommand{\\errsig}{\\Delta}$\n",
    "$\\errsig$: **TODO**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 0), label=r\"$w_{ij}$\", label_position=0.5, fontsize=14)\n",
    "\n",
    "nnfig.draw_synapse(ax, (10, 0), (12, 0))\n",
    "\n",
    "nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)\n",
    "plt.text(x=0, y=7.5, s=r\"$j$\", fontsize=14)\n",
    "plt.text(x=10, y=1.5, s=r\"$i$\", fontsize=14)\n",
    "plt.text(x=0, y=0, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=-2.5, y=0, s=r\"$y_j$\", fontsize=14)\n",
    "plt.text(x=13, y=0, s=r\"$y_i$\", fontsize=14)\n",
    "plt.text(x=9.2, y=-1.8, s=r\"$x_i$\", fontsize=14)\n",
    "\n",
    "nnfig.draw_neuron(ax, (10, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sigincur = \\sum_j \\wij \\sigoutprev\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\sigoutcur = \\activfunc(\\sigincur)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\weights = \\begin{pmatrix}\n",
    "    \\weight_{11} & \\cdots & \\weight_{1m} \\\\\n",
    "    \\vdots       & \\ddots & \\vdots       \\\\\n",
    "    \\weight_{n1} & \\cdots & \\weight_{nm}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Divers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le PMC peut approximer n'importe quelle fonction **continue** avec une pr\u00e9cision arbitraire suivant le nombre de neurones pr\u00e9sents sur la couche cach\u00e9e."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialisation des poids: g\u00e9n\u00e9ralement des petites valeurs al\u00e9atoires"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: quelle diff\u00e9rence entre:\n",
    "* r\u00e9seau boucl\u00e9\n",
    "* r\u00e9seau r\u00e9curent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fonction objectif (ou fonction d'erreur)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonction objectif: $\\errfunc \\left( \\weights(\\learnit) \\right)$\n",
    "\n",
    "$\\learnit$: it\u00e9ration courante de l'apprentissage $(1, 2, ...)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typiquement, la fonction objectif (fonction d'erreur) est la somme du carr\u00e9 de l'erreur de chaque neurone de sortie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errfunc = \\frac12 \\sum_{i \\in \\Omega} \\left[ \\sigoutcur - \\sigoutdes \\right]^2\n",
    "$$\n",
    "\n",
    "$\\Omega$: l'ensemble des neurones de sortie\n",
    "\n",
    "Le $\\frac12$, c'est juste pour simplifier les calculs de la d\u00e9riv\u00e9e ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 0))\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, -4))\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 4))\n",
    "\n",
    "nnfig.draw_synapse(ax, (10, -4), (12, -4))\n",
    "nnfig.draw_synapse(ax, (10, 0), (12, 0))\n",
    "nnfig.draw_synapse(ax, (10, 4), (12, 4))\n",
    "\n",
    "nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)\n",
    "\n",
    "nnfig.draw_neuron(ax, (10, -4), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (10, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (10, 4), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "plt.text(x=0, y=7.5, s=r\"$j$\", fontsize=14)\n",
    "plt.text(x=10, y=7.5, s=r\"$i$\", fontsize=14)\n",
    "\n",
    "plt.text(x=0, y=0, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=9.7, y=-6.1, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=9.7, y=5.8, s=r\"$\\vdots$\", fontsize=14)\n",
    "\n",
    "plt.text(x=12.5, y=4,  s=r\"$y_{i1}$\", fontsize=14)\n",
    "plt.text(x=12.5, y=0,  s=r\"$y_{i2}$\", fontsize=14)\n",
    "plt.text(x=12.5, y=-4, s=r\"$y_{i3}$\", fontsize=14)\n",
    "\n",
    "plt.text(x=16, y=4,  s=r\"$E_1 = y_{i1} - d_{i1}$\", fontsize=14)\n",
    "plt.text(x=16, y=0,  s=r\"$E_2 = y_{i2} - d_{i2}$\", fontsize=14)\n",
    "plt.text(x=16, y=-4, s=r\"$E_3 = y_{i3} - d_{i3}$\", fontsize=14)\n",
    "\n",
    "plt.text(x=16, y=-8, s=r\"$E = 1/2 (E_1^2 + E_2^2 + E_3^2 + \\dots)$\", fontsize=14)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apprentissage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise \u00e0 jours des poids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\weights(\\learnit + 1) = \\weights(\\learnit) \\underbrace{- \\learnrate \\nabla \\errfunc \\left( \\weights(\\learnit) \\right)}\n",
    "$$\n",
    "\n",
    "$- \\learnrate \\nabla \\errfunc \\left( \\weights(\\learnit) \\right)$: descend dans la direction oppos\u00e9e au gradient (plus forte pente)\n",
    "\n",
    "avec $\\nabla \\errfunc \\left( \\weights(\\learnit) \\right)$: gradient de la fonction objectif au point $\\weights$\n",
    "\n",
    "$\\learnrate > 0$: pas (ou taux) d'apprentissage\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\delta_{\\wij} & = \\wij(\\learnit + 1) - \\wij(\\learnit) \\\\\n",
    "              & = - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\Leftrightarrow \\wij(\\learnit + 1) = \\wij(\\learnit) - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chaque pr\u00e9sentation de l'ensemble des exemples = un *cycle* (ou une *\u00e9poque*) d'apprentissage\n",
    "\n",
    "Crit\u00e8re d'arr\u00eat de l'apprentissage: quand la valeur de la fonction objectif se stabilise (ou que le probl\u00e8me est r\u00e9solu avec la pr\u00e9cision souhait\u00e9e)\n",
    "\n",
    "- \"g\u00e9n\u00e9ralement il n'y a qu'un seul minimum local\" (preuve ???)\n",
    "- \"dans le cas contraire, le plus simple est de recommencer plusieurs fois l'apprentissage avec des poids initiaux diff\u00e9rents et de conserver la meilleure matrice $\\weights$ (celle qui minimise $\\errfunc$)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(4, 4))\n",
    "\n",
    "x = np.arange(10, 30, 0.1)\n",
    "y = (x - 20)**2 + 2\n",
    "\n",
    "ax.set_xlabel(r\"Poids $W$\", fontsize=14)\n",
    "ax.set_ylabel(r\"Fonction objectif $E$\", fontsize=14)\n",
    "\n",
    "# See http://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes.tick_params\n",
    "ax.tick_params(axis='both',       # changes apply to the x and y axis\n",
    "               which='both',      # both major and minor ticks are affected\n",
    "               bottom='on',       # ticks along the bottom edge are on\n",
    "               top='off',         # ticks along the top edge are off\n",
    "               left='on',         # ticks along the left edge are on\n",
    "               right='off',       # ticks along the right edge are off\n",
    "               labelbottom='off', # labels along the bottom edge are off\n",
    "               labelleft='off')   # labels along the lefleft are off\n",
    "\n",
    "ax.set_xlim(left=10, right=25)\n",
    "ax.set_ylim(bottom=0, top=5)\n",
    "\n",
    "ax.plot(x, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Apprentissage incr\u00e9mentiel* (ou *partiel*) (ang. *incremental learning*):\n",
    "on ajuste les poids $\\weights$ apr\u00e8s la pr\u00e9sentation d'un seul exemple\n",
    "(\"ce n'est pas une v\u00e9ritable descente de gradient\").\n",
    "C'est mieux pour \u00e9viter les minimums locaux, surtout si les exemples sont\n",
    "m\u00e9lang\u00e9s au d\u00e9but de chaque it\u00e9ration\n",
    "\n",
    "*Apprentissage diff\u00e9r\u00e9* (ang. *batch learning*):\n",
    "**TODO...**\n",
    "Est-ce que la fonction objectif $\\errfunc$ est une fonction multivari\u00e9e\n",
    "ou est-ce une aggr\u00e9gation des erreurs de chaque exemple ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### R\u00e9tropropagation du gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*R\u00e9tropropagation du gradient*:\n",
    "une m\u00e9thode pour calculer efficacement le gradient de la fonction objectif $\\errfunc$\n",
    "\n",
    "Principe:\n",
    "on modifie les poids \u00e0 l'aide des *signaux d'erreur* $\\errsig_i$.\n",
    "\n",
    "$$\n",
    "\\wij(\\learnit + 1) = \\wij(\\learnit) \\underbrace{- \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij(n)}}_{\\delta_{ij}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\delta_{ij} & = - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij(n)} \\\\\n",
    "            & = - \\learnrate \\errsig_i \\sigoutprev\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "- Dans le cas de *l'apprentissage diff\u00e9r\u00e9* (*batch*), on calcule pour chaque exemple l'erreur correspondante. Leur contribution individuelle aux modifications des poids sont additionn\u00e9es\n",
    "- L'apprentissage suppervis\u00e9 fonctionne mieux avec des neurones de sortie lin\u00e9aires (fonction d'activation $\\activfunc$ = fonction identit\u00e9e) \"car les signaux d'erreurs se transmettent mieux\".\n",
    "- Des donn\u00e9es d'entr\u00e9e binaires doivent \u00eatre choisies dans $\\{-1,1\\}$ plut\u00f4t que $\\{0,1\\}$ car un signal nul ne contribu pas \u00e0 l'apprentissage.\n",
    "\n",
    "Voc:\n",
    "- *erreur marginale*: **TODO**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Signaux d'erreur $\\errsig_i$ pour les neurones de sortie $(i \\in \\Omega)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errsig_i = \\activfunc'(\\sigincur)[\\sigoutcur - \\sigoutdes]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Signaux d'erreur $\\errsig_i$ pour les neurones cach\u00e9s $(i \\not\\in \\Omega)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errsig_i = \\activfunc'(\\sigincur) \\sum_k \\weight_{ki} \\errsig_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fonctions d'activation : fonctions sigmoides (en forme de \"S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La fonction sigmo\u00efde (en forme de \"S\") est d\u00e9finie par :\n",
    "\n",
    "$$f(x) = \\frac{1}{1 + e^{-x}}$$\n",
    "\n",
    "pour tout r\u00e9el $x$.\n",
    "\n",
    "On peut la g\u00e9n\u00e9raliser \u00e0 toute fonction dont l'expression est :\n",
    "\n",
    "$$f(x) = \\frac{1}{1 + e^{-\\lambda x}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x, _lambda=1.):\n",
    "    y = 1. / (1. + np.exp(-_lambda * x))\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(-5, 5, 300)\n",
    "\n",
    "y1 = sigmoid(x, 1.)\n",
    "y2 = sigmoid(x, 5.)\n",
    "y3 = sigmoid(x, 0.5)\n",
    "\n",
    "plt.plot(x, y1, label=r\"$\\lambda=1$\")\n",
    "plt.plot(x, y2, label=r\"$\\lambda=5$\")\n",
    "plt.plot(x, y3, label=r\"$\\lambda=0.5$\")\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.legend()\n",
    "\n",
    "plt.title(\"Fonction sigmo\u00efde\")\n",
    "plt.axis([-5, 5, -0.5, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonction d\u00e9riv\u00e9e :\n",
    "\n",
    "$$\n",
    "f'(x) = \\frac{\\lambda e^{-\\lambda x}}{(1+e^{-\\lambda x})^{2}}\n",
    "$$\n",
    "\n",
    "qui peut aussi \u00eatre d\u00e9fini par\n",
    "\n",
    "$$\n",
    "\\frac{\\mathrm{d} y}{\\mathrm{d} x} = \\lambda y (1-y)\n",
    "$$\n",
    "\n",
    "o\u00f9 $y$ varie de 0 \u00e0 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def d_sigmoid(x, _lambda=1.):\n",
    "    e = np.exp(-_lambda * x)\n",
    "    y = _lambda * e / np.power(1 + e, 2)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(-5, 5, 300)\n",
    "\n",
    "y1 = d_sigmoid(x, 1.)\n",
    "y2 = d_sigmoid(x, 5.)\n",
    "y3 = d_sigmoid(x, 0.5)\n",
    "\n",
    "plt.plot(x, y1, label=r\"$\\lambda=1$\")\n",
    "plt.plot(x, y2, label=r\"$\\lambda=5$\")\n",
    "plt.plot(x, y3, label=r\"$\\lambda=0.5$\")\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.legend()\n",
    "\n",
    "plt.title(\"Fonction d\u00e9riv\u00e9e de la sigmo\u00efde\")\n",
    "plt.axis([-5, 5, -0.5, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tangente hyperbolique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def tanh(x):\n",
    "    y = np.tanh(x)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-5, 5, 300)\n",
    "y = tanh(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.title(\"Fonction tangente hyperbolique\")\n",
    "plt.axis([-5, 5, -2, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "D\u00e9riv\u00e9e :\n",
    "\n",
    "$$\n",
    "\\tanh '= \\frac{1}{\\cosh^{2}} = 1-\\tanh^{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def d_tanh(x):\n",
    "    y = 1. - np.power(np.tanh(x), 2)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-5, 5, 300)\n",
    "y = d_tanh(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.title(\"Fonction d\u00e9riv\u00e9e de la tangente hyperbolique\")\n",
    "plt.axis([-5, 5, -2, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fonction logistique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonctions ayant pour expression\n",
    "\n",
    "$$\n",
    "f(t) = K \\frac{1}{1+ae^{-rt}}\n",
    "$$\n",
    "\n",
    "o\u00f9 $K$ et $r$ sont des r\u00e9els positifs et $a$ un r\u00e9el quelconque.\n",
    "\n",
    "Les fonctions sigmo\u00efdes sont un cas particulier de fonctions logistique avec $a > 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the activation function and its derivative\n",
    "activation_function = tanh\n",
    "d_activation_function = d_tanh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def init_weights(num_input_cells, num_output_cells, num_cell_per_hidden_layer, num_hidden_layers=1):\n",
    "    \"\"\"\n",
    "    The returned `weights` object is a list of weight matrices,\n",
    "    where weight matrix at index $i$ represents the weights between\n",
    "    layer $i$ and layer $i+1$.\n",
    "    \n",
    "    Numpy array shapes for e.g. num_input_cells=2, num_output_cells=2,\n",
    "    num_cell_per_hidden_layer=3 (without taking account bias):\n",
    "    - in:        (2,)\n",
    "    - in+bias:   (3,)\n",
    "    - w[0]:      (3,3)\n",
    "    - w[0]+bias: (3,4)\n",
    "    - w[1]:      (3,2)\n",
    "    - w[1]+bias: (4,2)\n",
    "    - out:       (2,)\n",
    "    \"\"\"\n",
    "    \n",
    "    # TODO:\n",
    "    # - faut-il que wij soit positif ?\n",
    "    # - loi normale plus appropri\u00e9e que loi uniforme ?\n",
    "    # - quel sigma conseill\u00e9 ?\n",
    "    \n",
    "    W = []\n",
    "    \n",
    "    # Weights between the input layer and the first hidden layer\n",
    "    W.append(np.random.uniform(low=0., high=1., size=(num_input_cells + 1, num_cell_per_hidden_layer + 1)))\n",
    "    \n",
    "    # Weights between hidden layers (if there are more than one hidden layer)\n",
    "    for layer in range(num_hidden_layers - 1):\n",
    "        W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_cell_per_hidden_layer + 1)))\n",
    "    \n",
    "    # Weights between the last hidden layer and the output layer\n",
    "    W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_output_cells)))\n",
    "    \n",
    "    return W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def evaluate_network(weights, input_signal):      # TODO: find a better name\n",
    "    \n",
    "    # Add the bias on the input layer\n",
    "    input_signal = np.concatenate([input_signal, [-1]])\n",
    "    \n",
    "    assert input_signal.ndim == 1\n",
    "    assert input_signal.shape[0] == weights[0].shape[0]\n",
    "    \n",
    "    # Compute the output of the first hidden layer\n",
    "    p = np.dot(input_signal, weights[0])\n",
    "    output_hidden_layer = activation_function(p)\n",
    "    \n",
    "    # Compute the output of the intermediate hidden layers\n",
    "    # TODO: check this\n",
    "    num_layers = len(weights)\n",
    "    for n in range(num_layers - 2):\n",
    "        p = np.dot(output_hidden_layer, weights[n + 1])\n",
    "        output_hidden_layer = activation_function(p)\n",
    "    \n",
    "    # Compute the output of the output layer\n",
    "    p = np.dot(output_hidden_layer, weights[-1])\n",
    "    output_signal = activation_function(p)\n",
    "    \n",
    "    return output_signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_gradient():\n",
    "    # TODO\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights = init_weights(num_input_cells=2, num_output_cells=2, num_cell_per_hidden_layer=3, num_hidden_layers=1)\n",
    "print(weights)\n",
    "#print(weights[0].shape)\n",
    "#print(weights[1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input_signal = np.array([.1, .2])\n",
    "input_signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "evaluate_network(weights, input_signal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes de la documentation sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- *features* : les donn\u00e9es d'entr\u00e9e du r\u00e9seau (i.e. les entr\u00e9es de la 1ere couche du r\u00e9seau)\n",
    "  - \"nombre de features\" =  taille du vecteur d'entr\u00e9es\n",
    "- *loss function*: fonction objectif (ou fonction d'erreur)\n",
    "- *fitting*: processus d'apprentissage (training)\n",
    "- *sample*: exemple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les biais sont stock\u00e9s dans une liste de vecteurs plut\u00f4t qu'une liste de scalaires... pourquoi ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Avantages des PMC:\n",
    "- capables d'apprendre des mod\u00e8les non lin\u00e9aires\n",
    "- capables d'apprendre des mod\u00e8les en temps r\u00e9el (apprentissage *on-line*)\n",
    "\n",
    "Inconvenients des PMC:\n",
    "- les PMC avec une ou plusieurs couches cach\u00e9es ont une fonction objectif non-convexe avec des minimas locaux. Par cons\u00e9quent, le r\u00e9sultat du processus d'apprentissage peut varier d'une execution \u00e0 l'autre suivant la valeur des poids initiaux et l'obtention d'un r\u00e9seau optimal n'est pas garanti\n",
    "- pour obtenir un r\u00e9sultat satisfaisant, il est souvant n\u00e9cessaire de r\u00e9gler (plus ou moins empiriquement) de nombreux meta-param\u00e8tres (nombres de couches cach\u00e9es, nombre de neurones sur les couches cach\u00e9es, nombres d'it\u00e9rations, ...)\n",
    "- une  mauvaise normalisation des donn\u00e9es d'entr\u00e9e a un impact tr\u00e8s n\u00e9gatif sur la qualit\u00e9 du r\u00e9sultat (\"mal conditionn\u00e9\" ???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Cross-Entropy Loss Function*: ...\n",
    "\n",
    "*Softmax*: ...\n",
    "\n",
    "*Multi-label classification*: ... mod\u00e8le de classifieur qui permet a un exemple d'appartenir \u00e0 plusieurs classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes du livre de <NAME>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}<commit_msg>Explain in detail Delta_i equation.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Perceptron Multicouche"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principales implémentations en Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Scikit-learn: http://scikit-learn.org/stable/modules/neural_networks_supervised.html\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes du livre Dunod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\newcommand{\\sigin}{x}$\n",
    "$\\sigin$: **TODO**\n",
    "\n",
    "$\\newcommand{\\pot}{x}$\n",
    "$\\pot$: *Potentiel (d'activation ?)* du neurone **TODO**\n",
    "\n",
    "$\\newcommand{\\wij}{w_{ij}}$\n",
    "$\\wij$: Poids de la connexion entre le neurone $j$ et le neurone $i$\n",
    "\n",
    "$\\newcommand{\\activthres}{\\theta}$\n",
    "$\\activthres$: *Seuil d'activation* du neurone\n",
    "\n",
    "$\\newcommand{\\activfunc}{f}$\n",
    "$\\activfunc_i$: *Fonction d'activation* du neurone $i$\n",
    "\n",
    "$\\newcommand{\\errfunc}{E}$\n",
    "$\\errfunc$: *Fonction objectif* ou *fonction d'erreur*\n",
    "\n",
    "$\\newcommand{\\learnrate}{\\epsilon}$\n",
    "$\\learnrate$: *Pas d'apprentissage* ou *Taux d'apprentissage*\n",
    "\n",
    "$\\newcommand{\\learnit}{n}$\n",
    "$\\learnit$: **TODO**\n",
    "\n",
    "$\\newcommand{\\it}{t}$\n",
    "$\\it$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigincur}{x_i}$\n",
    "$\\sigincur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigout}{y}$\n",
    "$\\sigout$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutcur}{y_i}$\n",
    "$\\sigoutcur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutprev}{y_j}$\n",
    "$\\sigoutprev$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutdes}{d}$\n",
    "$\\sigoutdescur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\sigoutdescur}{d_i}$\n",
    "$\\sigoutdescur$: **TODO**\n",
    "\n",
    "$\\newcommand{\\weights}{\\boldsymbol{W}}$\n",
    "$\\weights$: **TODO**\n",
    "\n",
    "$\\newcommand{\\weight}{w}$\n",
    "$\\weight$: **TODO**\n",
    "\n",
    "$\\newcommand{\\errsig}{\\Delta}$\n",
    "$\\errsig_i$: *Signal d'erreur* du neurone $i$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 0), label=r\"$w_{ij}$\", label_position=0.5, fontsize=14)\n",
    "\n",
    "nnfig.draw_synapse(ax, (10, 0), (12, 0))\n",
    "\n",
    "nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)\n",
    "plt.text(x=0, y=7.5, s=r\"$j$\", fontsize=14)\n",
    "plt.text(x=10, y=1.5, s=r\"$i$\", fontsize=14)\n",
    "plt.text(x=0, y=0, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=-2.5, y=0, s=r\"$y_j$\", fontsize=14)\n",
    "plt.text(x=13, y=0, s=r\"$y_i$\", fontsize=14)\n",
    "plt.text(x=9.2, y=-1.8, s=r\"$x_i$\", fontsize=14)\n",
    "\n",
    "nnfig.draw_neuron(ax, (10, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sigincur = \\sum_j \\wij \\sigoutprev\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\sigoutcur = \\activfunc(\\sigincur)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\weights = \\begin{pmatrix}\n",
    "    \\weight_{11} & \\cdots & \\weight_{1m} \\\\\n",
    "    \\vdots       & \\ddots & \\vdots       \\\\\n",
    "    \\weight_{n1} & \\cdots & \\weight_{nm}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Divers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le PMC peut approximer n'importe quelle fonction **continue** avec une précision arbitraire suivant le nombre de neurones présents sur la couche cachée."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialisation des poids: généralement des petites valeurs aléatoires"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: quelle différence entre:\n",
    "* réseau bouclé\n",
    "* réseau récurent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fonction objectif (ou fonction d'erreur)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonction objectif: $\\errfunc \\left( \\weights(\\learnit) \\right)$\n",
    "\n",
    "$\\learnit$: itération courante de l'apprentissage $(1, 2, ...)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typiquement, la fonction objectif (fonction d'erreur) est la somme du carré de l'erreur de chaque neurone de sortie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errfunc = \\frac12 \\sum_{i \\in \\Omega} \\left[ \\sigoutcur - \\sigoutdescur \\right]^2\n",
    "$$\n",
    "\n",
    "$\\Omega$: l'ensemble des neurones de sortie\n",
    "\n",
    "Le $\\frac12$, c'est juste pour simplifier les calculs de la dérivée ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 0))\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, -4))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, -4))\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -6), (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 4))\n",
    "nnfig.draw_synapse(ax, (0, 6),  (10, 4))\n",
    "\n",
    "nnfig.draw_synapse(ax, (10, -4), (12, -4))\n",
    "nnfig.draw_synapse(ax, (10, 0), (12, 0))\n",
    "nnfig.draw_synapse(ax, (10, 4), (12, 4))\n",
    "\n",
    "nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)\n",
    "\n",
    "nnfig.draw_neuron(ax, (10, -4), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (10, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (10, 4), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "plt.text(x=0, y=7.5, s=r\"$j$\", fontsize=14)\n",
    "plt.text(x=10, y=7.5, s=r\"$i$\", fontsize=14)\n",
    "\n",
    "plt.text(x=0, y=0, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=9.7, y=-6.1, s=r\"$\\vdots$\", fontsize=14)\n",
    "plt.text(x=9.7, y=5.8, s=r\"$\\vdots$\", fontsize=14)\n",
    "\n",
    "plt.text(x=12.5, y=4,  s=r\"$y_{i1}$\", fontsize=14)\n",
    "plt.text(x=12.5, y=0,  s=r\"$y_{i2}$\", fontsize=14)\n",
    "plt.text(x=12.5, y=-4, s=r\"$y_{i3}$\", fontsize=14)\n",
    "\n",
    "plt.text(x=16, y=4,  s=r\"$E_1 = y_{i1} - d_{i1}$\", fontsize=14)\n",
    "plt.text(x=16, y=0,  s=r\"$E_2 = y_{i2} - d_{i2}$\", fontsize=14)\n",
    "plt.text(x=16, y=-4, s=r\"$E_3 = y_{i3} - d_{i3}$\", fontsize=14)\n",
    "\n",
    "plt.text(x=16, y=-8, s=r\"$E = 1/2 (E_1^2 + E_2^2 + E_3^2 + \\dots)$\", fontsize=14)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apprentissage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise à jours des poids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\weights(\\learnit + 1) = \\weights(\\learnit) \\underbrace{- \\learnrate \\nabla \\errfunc \\left( \\weights(\\learnit) \\right)}\n",
    "$$\n",
    "\n",
    "$- \\learnrate \\nabla \\errfunc \\left( \\weights(\\learnit) \\right)$: descend dans la direction opposée au gradient (plus forte pente)\n",
    "\n",
    "avec $\\nabla \\errfunc \\left( \\weights(\\learnit) \\right)$: gradient de la fonction objectif au point $\\weights$\n",
    "\n",
    "$\\learnrate > 0$: pas (ou taux) d'apprentissage\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\delta_{\\wij} & = \\wij(\\learnit + 1) - \\wij(\\learnit) \\\\\n",
    "              & = - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\Leftrightarrow \\wij(\\learnit + 1) = \\wij(\\learnit) - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chaque présentation de l'ensemble des exemples = un *cycle* (ou une *époque*) d'apprentissage\n",
    "\n",
    "Critère d'arrêt de l'apprentissage: quand la valeur de la fonction objectif se stabilise (ou que le problème est résolu avec la précision souhaitée)\n",
    "\n",
    "- \"généralement il n'y a qu'un seul minimum local\" (preuve ???)\n",
    "- \"dans le cas contraire, le plus simple est de recommencer plusieurs fois l'apprentissage avec des poids initiaux différents et de conserver la meilleure matrice $\\weights$ (celle qui minimise $\\errfunc$)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(4, 4))\n",
    "\n",
    "x = np.arange(10, 30, 0.1)\n",
    "y = (x - 20)**2 + 2\n",
    "\n",
    "ax.set_xlabel(r\"Poids $W$\", fontsize=14)\n",
    "ax.set_ylabel(r\"Fonction objectif $E$\", fontsize=14)\n",
    "\n",
    "# See http://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes.tick_params\n",
    "ax.tick_params(axis='both',       # changes apply to the x and y axis\n",
    "               which='both',      # both major and minor ticks are affected\n",
    "               bottom='on',       # ticks along the bottom edge are on\n",
    "               top='off',         # ticks along the top edge are off\n",
    "               left='on',         # ticks along the left edge are on\n",
    "               right='off',       # ticks along the right edge are off\n",
    "               labelbottom='off', # labels along the bottom edge are off\n",
    "               labelleft='off')   # labels along the lefleft are off\n",
    "\n",
    "ax.set_xlim(left=10, right=25)\n",
    "ax.set_ylim(bottom=0, top=5)\n",
    "\n",
    "ax.plot(x, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Apprentissage incrémentiel* (ou *partiel*) (ang. *incremental learning*):\n",
    "on ajuste les poids $\\weights$ après la présentation d'un seul exemple\n",
    "(\"ce n'est pas une véritable descente de gradient\").\n",
    "C'est mieux pour éviter les minimums locaux, surtout si les exemples sont\n",
    "mélangés au début de chaque itération\n",
    "\n",
    "*Apprentissage différé* (ang. *batch learning*):\n",
    "**TODO...**\n",
    "Est-ce que la fonction objectif $\\errfunc$ est une fonction multivariée\n",
    "ou est-ce une aggrégation des erreurs de chaque exemple ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rétropropagation du gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Rétropropagation du gradient*:\n",
    "une méthode pour calculer efficacement le gradient de la fonction objectif $\\errfunc$\n",
    "\n",
    "Principe:\n",
    "on modifie les poids à l'aide des *signaux d'erreur* $\\errsig_i$.\n",
    "\n",
    "$$\n",
    "\\wij(\\learnit + 1) = \\wij(\\learnit) \\underbrace{- \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij(n)}}_{\\delta_{ij}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\delta_{ij} & = - \\learnrate \\frac{\\partial \\errfunc}{\\partial \\wij(n)} \\\\\n",
    "            & = - \\learnrate \\errsig_i \\sigoutprev\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "- Dans le cas de *l'apprentissage différé* (*batch*), on calcule pour chaque exemple l'erreur correspondante. Leur contribution individuelle aux modifications des poids sont additionnées\n",
    "- L'apprentissage suppervisé fonctionne mieux avec des neurones de sortie linéaires (fonction d'activation $\\activfunc$ = fonction identitée) \"car les signaux d'erreurs se transmettent mieux\".\n",
    "- Des données d'entrée binaires doivent être choisies dans $\\{-1,1\\}$ plutôt que $\\{0,1\\}$ car un signal nul ne contribu pas à l'apprentissage.\n",
    "\n",
    "Voc:\n",
    "- *erreur marginale*: **TODO**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Signaux d'erreur $\\errsig_i$ pour les neurones de sortie $(i \\in \\Omega)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errsig_i = \\activfunc'(\\sigincur)[\\sigoutcur - \\sigoutdescur]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Signaux d'erreur $\\errsig_i$ pour les neurones cachés $(i \\not\\in \\Omega)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\errsig_i = \\activfunc'(\\sigincur) \\sum_k \\weight_{ki} \\errsig_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "nnfig.draw_synapse(ax, (0, -2), (10, 0))\n",
    "nnfig.draw_synapse(ax, (0, 2),  (10, 0), label=r\"$w_{ki}$\", label_position=0.5, fontsize=14)\n",
    "\n",
    "nnfig.draw_synapse(ax, (10, 0), (12, 0))\n",
    "\n",
    "nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)\n",
    "\n",
    "plt.text(x=0, y=3.5, s=r\"$i$\", fontsize=14)\n",
    "plt.text(x=10, y=3.5, s=r\"$k$\", fontsize=14)\n",
    "plt.text(x=0, y=-0.2, s=r\"$\\vdots$\", fontsize=14)\n",
    "\n",
    "nnfig.draw_neuron(ax, (10, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plus de détail : calcul de $\\errsig_i$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dans l'exemple suivant on ne s'intéresse qu'aux poids $\\weight_1$, $\\weight_2$, $\\weight_3$, $\\weight_4$ et $\\weight_5$ pour simplifier la demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import nnfigs\n",
    "\n",
    "# https://github.com/jeremiedecock/neural-network-figures.git\n",
    "import nnfigs.core as nnfig\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = nnfig.init_figure(size_x=8, size_y=4)\n",
    "\n",
    "HSPACE = 6\n",
    "VSPACE = 4\n",
    "\n",
    "# Synapse #####################################\n",
    "\n",
    "# Layer 1-2\n",
    "nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE,  VSPACE), label=r\"$w_1$\", label_position=0.4)\n",
    "nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE,  VSPACE), color=\"lightgray\")\n",
    "\n",
    "nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, -VSPACE), color=\"lightgray\")\n",
    "nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, -VSPACE), color=\"lightgray\")\n",
    "\n",
    "# Layer 2-3\n",
    "nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE,  VSPACE), label=r\"$w_2$\", label_position=0.4)\n",
    "nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE,  VSPACE), color=\"lightgray\")\n",
    "\n",
    "nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE, -VSPACE), label=r\"$w_3$\", label_position=0.4)\n",
    "nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE, -VSPACE), color=\"lightgray\")\n",
    "\n",
    "# Layer 3-4\n",
    "nnfig.draw_synapse(ax, (2*HSPACE,  VSPACE), (3*HSPACE, 0), label=r\"$w_4$\", label_position=0.4)\n",
    "nnfig.draw_synapse(ax, (2*HSPACE, -VSPACE), (3*HSPACE, 0), label=r\"$w_5$\", label_position=0.4, label_offset_y=-0.8)\n",
    "\n",
    "# Neuron ######################################\n",
    "\n",
    "# Layer 1 (input)\n",
    "nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)\n",
    "nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True, line_color=\"lightgray\")\n",
    "\n",
    "# Layer 2\n",
    "nnfig.draw_neuron(ax, (HSPACE,  VSPACE), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (HSPACE, -VSPACE), 1, ag_func=\"sum\", tr_func=\"sigmoid\", line_color=\"lightgray\")\n",
    "\n",
    "# Layer 3\n",
    "nnfig.draw_neuron(ax, (2*HSPACE,  VSPACE), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "nnfig.draw_neuron(ax, (2*HSPACE, -VSPACE), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "# Layer 4\n",
    "nnfig.draw_neuron(ax, (3*HSPACE, 0), 1, ag_func=\"sum\", tr_func=\"sigmoid\")\n",
    "\n",
    "# Text ########################################\n",
    "\n",
    "# Layer 1 (input)\n",
    "plt.text(x=0.5, y=VSPACE+1, s=r\"$y_i$\", fontsize=12)\n",
    "\n",
    "# Layer 2\n",
    "plt.text(x=HSPACE-1.25, y=VSPACE+1.5, s=r\"$x_1$\", fontsize=12)\n",
    "plt.text(x=HSPACE+0.4,  y=VSPACE+1.5, s=r\"$y_1$\", fontsize=12)\n",
    "\n",
    "# Layer 3\n",
    "plt.text(x=2*HSPACE-1.25, y=VSPACE+1.5, s=r\"$x_2$\", fontsize=12)\n",
    "plt.text(x=2*HSPACE+0.4,  y=VSPACE+1.5, s=r\"$y_2$\", fontsize=12)\n",
    "\n",
    "plt.text(x=2*HSPACE-1.25, y=-VSPACE-1.8, s=r\"$x_3$\", fontsize=12)\n",
    "plt.text(x=2*HSPACE+0.4,  y=-VSPACE-1.8, s=r\"$y_3$\", fontsize=12)\n",
    "\n",
    "# Layer 4\n",
    "plt.text(x=3*HSPACE-1.25, y=1.5, s=r\"$x_o$\", fontsize=12)\n",
    "plt.text(x=3*HSPACE+0.4,  y=1.5, s=r\"$y_o$\", fontsize=12)\n",
    "\n",
    "plt.text(x=3*HSPACE+2,  y=-0.3, s=r\"$E = (y_o - d_o)^2/2$\", fontsize=12)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Attention**: $\\weight_1$ influe $\\pot_2$ **et** $\\pot_3$ en plus de $\\pot_1$ et $\\pot_o$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calcul de $\\frac{\\partial \\errfunc}{\\partial \\weight_4}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\errfunc}{\\partial \\weight_4} =\n",
    "\\frac{\\partial \\pot_o}{\\partial \\weight_4}\n",
    "\\underbrace{\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    "}_{\\errsig_o}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "avec:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\pot_o}{\\partial \\weight_4}   &= \\sigout_2 \\\\\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}   &= \\activfunc'(\\pot_o) \\\\\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o} &= \\sigout_o - \\sigoutdes_o \\\\\n",
    "\\errsig_o &=\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o} \\\\\n",
    "&= \\activfunc'(\\pot_o) [\\sigout_o - \\sigoutdes_o]\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calcul de $\\frac{\\partial \\errfunc}{\\partial \\weight_5}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\errfunc}{\\partial \\weight_5} =\n",
    "\\frac{\\partial \\pot_o}{\\partial \\weight_5}\n",
    "\\underbrace{\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    "}_{\\errsig_o}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "avec:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\pot_o}{\\partial \\weight_5}   &= \\sigout_3 \\\\\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}   &= \\activfunc'(\\pot_o) \\\\\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o} &= \\sigout_o - \\sigoutdes_o \\\\\n",
    "\\errsig_o &=\n",
    "\\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "\\frac{\\partial \\errfunc}{\\partial \\sigout_o} \\\\\n",
    "&= \\activfunc'(\\pot_o) [\\sigout_o - \\sigoutdes_o]\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calcul de $\\frac{\\partial \\errfunc}{\\partial \\weight_2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\errfunc}{\\partial \\weight_2} =\n",
    "\\frac{\\partial \\pot_2}{\\partial \\weight_2}\n",
    "%\n",
    "\\underbrace{\n",
    " \\frac{\\partial \\sigout_2}{\\partial \\pot_2}\n",
    " \\frac{\\partial \\pot_o}{\\partial \\sigout_2}\n",
    " \\underbrace{\n",
    "  \\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "  \\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    " }_{\\errsig_o}\n",
    "}_{\\errsig_2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "avec:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\pot_2}{\\partial \\weight_2} &= \\sigout_1 \\\\\n",
    "\\frac{\\partial \\sigout_2}{\\partial \\pot_2} &= \\activfunc'(\\pot_2) \\\\\n",
    "\\frac{\\partial \\pot_o}{\\partial \\sigout_2} &= \\weight_4 \\\\\n",
    "\\errsig_2 &=\n",
    "\\frac{\\partial \\sigout_2}{\\partial \\pot_2}\n",
    "\\frac{\\partial \\pot_o}{\\partial \\sigout_2}\n",
    "\\errsig_o \\\\\n",
    "&= \\activfunc'(\\pot_2) \\weight_4 \\errsig_o\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calcul de $\\frac{\\partial \\errfunc}{\\partial \\weight_3}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\errfunc}{\\partial \\weight_3} =\n",
    "\\frac{\\partial \\pot_3}{\\partial \\weight_3}\n",
    "%\n",
    "\\underbrace{\n",
    " \\frac{\\partial \\sigout_3}{\\partial \\pot_3}\n",
    " \\frac{\\partial \\pot_o}{\\partial \\sigout_3}\n",
    " \\underbrace{\n",
    "  \\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "  \\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    " }_{\\errsig_o}\n",
    "}_{\\errsig_3}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "avec:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\pot_3}{\\partial \\weight_3} &= \\sigout_1 \\\\\n",
    "\\frac{\\partial \\sigout_3}{\\partial \\pot_3} &= \\activfunc'(\\pot_3) \\\\\n",
    "\\frac{\\partial \\pot_o}{\\partial \\sigout_3} &= \\weight_5 \\\\\n",
    "\\errsig_3 &= \n",
    "\\frac{\\partial \\sigout_3}{\\partial \\pot_3}\n",
    "\\frac{\\partial \\pot_o}{\\partial \\sigout_3}\n",
    "\\errsig_o \\\\\n",
    "&= \\activfunc'(\\pot_3) \\weight_5 \\errsig_o\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calcul de $\\frac{\\partial \\errfunc}{\\partial \\weight_1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\errfunc}{\\partial \\weight_1} =\n",
    "\\frac{\\partial \\pot_1}{\\partial \\weight_1}\n",
    "%\n",
    "\\underbrace{\n",
    " \\frac{\\partial \\sigout_1}{\\partial \\pot_1}\n",
    " \\left(\n",
    "  \\frac{\\partial \\pot_2}{\\partial \\sigout_1} % err?\n",
    "  \\underbrace{\n",
    "   \\frac{\\partial \\sigout_2}{\\partial \\pot_2}\n",
    "   \\frac{\\partial \\pot_o}{\\partial \\sigout_2}\n",
    "   \\underbrace{\n",
    "    \\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "    \\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    "   }_{\\errsig_o}\n",
    "  }_{\\errsig_2}\n",
    "  +\n",
    "  \\frac{\\partial \\pot_3}{\\partial \\sigout_1} % err?\n",
    "  \\underbrace{\n",
    "   \\frac{\\partial \\sigout_3}{\\partial \\pot_3}\n",
    "   \\frac{\\partial \\pot_o}{\\partial \\sigout_3}\n",
    "   \\underbrace{\n",
    "    \\frac{\\partial \\sigout_o}{\\partial \\pot_o}\n",
    "    \\frac{\\partial \\errfunc}{\\partial \\sigout_o}\n",
    "   }_{\\errsig_o}\n",
    "  }_{\\errsig_3}\n",
    " \\right)\n",
    "}_{\\errsig_1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "avec:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\pot_1}{\\partial \\weight_1} &= \\sigout_i \\\\\n",
    "\\frac{\\partial \\sigout_1}{\\partial \\pot_1} &= \\activfunc'(\\pot_1) \\\\\n",
    "\\frac{\\partial \\pot_2}{\\partial \\sigout_1} &= \\weight_2 \\\\\n",
    "\\frac{\\partial \\pot_3}{\\partial \\sigout_1} &= \\weight_3 \\\\\n",
    "\\errsig_1 &=\n",
    "\\frac{\\partial \\sigout_1}{\\partial \\pot_1}\n",
    "\\left(\n",
    "\\frac{\\partial \\pot_2}{\\partial \\sigout_1}\n",
    "\\errsig_2\n",
    "+\n",
    "\\frac{\\partial \\pot_3}{\\partial \\sigout_1}\n",
    "\\errsig_3\n",
    "\\right) \\\\\n",
    "&= \n",
    "\\activfunc'(\\pot_1) \\left( \\weight_2 \\errsig_2 + \\weight_3 \\errsig_3 \\right)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fonctions d'activation : fonctions sigmoides (en forme de \"S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La fonction sigmoïde (en forme de \"S\") est définie par :\n",
    "\n",
    "$$f(x) = \\frac{1}{1 + e^{-x}}$$\n",
    "\n",
    "pour tout réel $x$.\n",
    "\n",
    "On peut la généraliser à toute fonction dont l'expression est :\n",
    "\n",
    "$$f(x) = \\frac{1}{1 + e^{-\\lambda x}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x, _lambda=1.):\n",
    "    y = 1. / (1. + np.exp(-_lambda * x))\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(-5, 5, 300)\n",
    "\n",
    "y1 = sigmoid(x, 1.)\n",
    "y2 = sigmoid(x, 5.)\n",
    "y3 = sigmoid(x, 0.5)\n",
    "\n",
    "plt.plot(x, y1, label=r\"$\\lambda=1$\")\n",
    "plt.plot(x, y2, label=r\"$\\lambda=5$\")\n",
    "plt.plot(x, y3, label=r\"$\\lambda=0.5$\")\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.legend()\n",
    "\n",
    "plt.title(\"Fonction sigmoïde\")\n",
    "plt.axis([-5, 5, -0.5, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonction dérivée :\n",
    "\n",
    "$$\n",
    "f'(x) = \\frac{\\lambda e^{-\\lambda x}}{(1+e^{-\\lambda x})^{2}}\n",
    "$$\n",
    "\n",
    "qui peut aussi être défini par\n",
    "\n",
    "$$\n",
    "\\frac{\\mathrm{d} y}{\\mathrm{d} x} = \\lambda y (1-y)\n",
    "$$\n",
    "\n",
    "où $y$ varie de 0 à 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def d_sigmoid(x, _lambda=1.):\n",
    "    e = np.exp(-_lambda * x)\n",
    "    y = _lambda * e / np.power(1 + e, 2)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(-5, 5, 300)\n",
    "\n",
    "y1 = d_sigmoid(x, 1.)\n",
    "y2 = d_sigmoid(x, 5.)\n",
    "y3 = d_sigmoid(x, 0.5)\n",
    "\n",
    "plt.plot(x, y1, label=r\"$\\lambda=1$\")\n",
    "plt.plot(x, y2, label=r\"$\\lambda=5$\")\n",
    "plt.plot(x, y3, label=r\"$\\lambda=0.5$\")\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.legend()\n",
    "\n",
    "plt.title(\"Fonction dérivée de la sigmoïde\")\n",
    "plt.axis([-5, 5, -0.5, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tangente hyperbolique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def tanh(x):\n",
    "    y = np.tanh(x)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-5, 5, 300)\n",
    "y = tanh(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.title(\"Fonction tangente hyperbolique\")\n",
    "plt.axis([-5, 5, -2, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dérivée :\n",
    "\n",
    "$$\n",
    "\\tanh '= \\frac{1}{\\cosh^{2}} = 1-\\tanh^{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def d_tanh(x):\n",
    "    y = 1. - np.power(np.tanh(x), 2)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-5, 5, 300)\n",
    "y = d_tanh(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "\n",
    "plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')\n",
    "plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')\n",
    "\n",
    "plt.title(\"Fonction dérivée de la tangente hyperbolique\")\n",
    "plt.axis([-5, 5, -2, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fonction logistique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonctions ayant pour expression\n",
    "\n",
    "$$\n",
    "f(t) = K \\frac{1}{1+ae^{-rt}}\n",
    "$$\n",
    "\n",
    "où $K$ et $r$ sont des réels positifs et $a$ un réel quelconque.\n",
    "\n",
    "Les fonctions sigmoïdes sont un cas particulier de fonctions logistique avec $a > 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the activation function and its derivative\n",
    "activation_function = tanh\n",
    "d_activation_function = d_tanh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def init_weights(num_input_cells, num_output_cells, num_cell_per_hidden_layer, num_hidden_layers=1):\n",
    "    \"\"\"\n",
    "    The returned `weights` object is a list of weight matrices,\n",
    "    where weight matrix at index $i$ represents the weights between\n",
    "    layer $i$ and layer $i+1$.\n",
    "    \n",
    "    Numpy array shapes for e.g. num_input_cells=2, num_output_cells=2,\n",
    "    num_cell_per_hidden_layer=3 (without taking account bias):\n",
    "    - in:        (2,)\n",
    "    - in+bias:   (3,)\n",
    "    - w[0]:      (3,3)\n",
    "    - w[0]+bias: (3,4)\n",
    "    - w[1]:      (3,2)\n",
    "    - w[1]+bias: (4,2)\n",
    "    - out:       (2,)\n",
    "    \"\"\"\n",
    "    \n",
    "    # TODO:\n",
    "    # - faut-il que wij soit positif ?\n",
    "    # - loi normale plus appropriée que loi uniforme ?\n",
    "    # - quel sigma conseillé ?\n",
    "    \n",
    "    W = []\n",
    "    \n",
    "    # Weights between the input layer and the first hidden layer\n",
    "    W.append(np.random.uniform(low=0., high=1., size=(num_input_cells + 1, num_cell_per_hidden_layer + 1)))\n",
    "    \n",
    "    # Weights between hidden layers (if there are more than one hidden layer)\n",
    "    for layer in range(num_hidden_layers - 1):\n",
    "        W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_cell_per_hidden_layer + 1)))\n",
    "    \n",
    "    # Weights between the last hidden layer and the output layer\n",
    "    W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_output_cells)))\n",
    "    \n",
    "    return W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_network(weights, input_signal):      # TODO: find a better name\n",
    "    \n",
    "    # Add the bias on the input layer\n",
    "    input_signal = np.concatenate([input_signal, [-1]])\n",
    "    \n",
    "    assert input_signal.ndim == 1\n",
    "    assert input_signal.shape[0] == weights[0].shape[0]\n",
    "    \n",
    "    # Compute the output of the first hidden layer\n",
    "    p = np.dot(input_signal, weights[0])\n",
    "    output_hidden_layer = activation_function(p)\n",
    "    \n",
    "    # Compute the output of the intermediate hidden layers\n",
    "    # TODO: check this\n",
    "    num_layers = len(weights)\n",
    "    for n in range(num_layers - 2):\n",
    "        p = np.dot(output_hidden_layer, weights[n + 1])\n",
    "        output_hidden_layer = activation_function(p)\n",
    "    \n",
    "    # Compute the output of the output layer\n",
    "    p = np.dot(output_hidden_layer, weights[-1])\n",
    "    output_signal = activation_function(p)\n",
    "    \n",
    "    return output_signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_gradient():\n",
    "    # TODO\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights = init_weights(num_input_cells=2, num_output_cells=2, num_cell_per_hidden_layer=3, num_hidden_layers=1)\n",
    "print(weights)\n",
    "#print(weights[0].shape)\n",
    "#print(weights[1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_signal = np.array([.1, .2])\n",
    "input_signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "evaluate_network(weights, input_signal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes de la documentation sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- *features* : les données d'entrée du réseau (i.e. les entrées de la 1ere couche du réseau)\n",
    "  - \"nombre de features\" =  taille du vecteur d'entrées\n",
    "- *loss function*: fonction objectif (ou fonction d'erreur)\n",
    "- *fitting*: processus d'apprentissage (training)\n",
    "- *sample*: exemple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les biais sont stockés dans une liste de vecteurs plutôt qu'une liste de scalaires... pourquoi ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Avantages des PMC:\n",
    "- capables d'apprendre des modèles non linéaires\n",
    "- capables d'apprendre des modèles en temps réel (apprentissage *on-line*)\n",
    "\n",
    "Inconvenients des PMC:\n",
    "- les PMC avec une ou plusieurs couches cachées ont une fonction objectif non-convexe avec des minimas locaux. Par conséquent, le résultat du processus d'apprentissage peut varier d'une execution à l'autre suivant la valeur des poids initiaux et l'obtention d'un réseau optimal n'est pas garanti\n",
    "- pour obtenir un résultat satisfaisant, il est souvant nécessaire de régler (plus ou moins empiriquement) de nombreux meta-paramètres (nombres de couches cachées, nombre de neurones sur les couches cachées, nombres d'itérations, ...)\n",
    "- une  mauvaise normalisation des données d'entrée a un impact très négatif sur la qualité du résultat (\"mal conditionné\" ???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Cross-Entropy Loss Function*: ...\n",
    "\n",
    "*Softmax*: ...\n",
    "\n",
    "*Multi-label classification*: ... modèle de classifieur qui permet a un exemple d'appartenir à plusieurs classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes du livre de <NAME>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
{'fraction_non_alphanumeric': 0.2511293941660434, 'fraction_numerical': 0.02432311144353029, 'mean_word_length': 2.4270261957246118, 'pattern_counts': {'":': 597, '<': 5, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '907250', 'n_tokens_mistral': 28799, 'n_tokens_neox': 27939, 'n_words': 5164}