<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning a scikit-learn estimator with `skopt`\n",
    "\n",
    "<NAME>, July 2016 <br />\n",
    "<NAME>, August 2016\n",
    "\n",
    "If you are looking for a `GridSearchCV` replacement checkout [the `BayesSearchCV` example](sklearn-gridsearchcv-replacement.ipynb) instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem statement\n",
    "\n",
    "Tuning the hyper-parameters of a machine learning model is often carried out using an exhaustive exploration of (a subset of) the space all hyper-parameter configurations (e.g., using `sklearn.model_selection.GridSearchCV`), which often results in a very time consuming operation. \n",
    "\n",
    "In this notebook, we illustrate how to couple `gp_minimize` with sklearn's estimators to tune hyper-parameters using sequential model-based optimisation, hopefully resulting in equivalent or better solutions, but within less evaluations. \n",
    "\n",
    "Note: scikit-optimize provides a dedicated interface for estimator tuning via `BayesSearchCV` class which has a similar interface to those of `GridSearchCV`. This class uses functions of skopt to perform hyperparameter search efficiently. For example usage of this class, see [the `BayesSearchCV` example](sklearn-gridsearchcv-replacement.ipynb) example notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To tune the hyper-parameters of our model we need to define a model, decide which parameters to optimize, and define the objective function we want to minimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "boston = load_boston()\n",
    "X, y = boston.data, boston.target\n",
    "n_features = X.shape[1]\n",
    "\n",
    "# gradient boosted trees tend to do well on problems like this\n",
    "reg = GradientBoostingRegressor(n_estimators=50, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to define the bounds of the dimensions of the search space we want to explore and pick the objective. In this case the cross-validation mean absolute error of a gradient boosting regressor over the Boston dataset, as a function of its hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import space as sk_space\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "\n",
    "# The list of hyper-parameters we want to optimize. For each one we define the bounds,\n",
    "# the corresponding scikit-learn parameter name, as well as how to sample values\n",
    "# from that dimension (`'log-uniform'` for the learning rate)\n",
    "space  = [sk_space.Integer(1, 5, name='max_depth'),\n",
    "          sk_space.Real(10**-5, 10**0, \"log-uniform\", name='learning_rate'),\n",
    "          sk_space.Integer(1, n_features, name='max_features'),\n",
    "          sk_space.Integer(2, 100, name='min_samples_split'),\n",
    "          sk_space.Integer(1, 100, name='min_samples_leaf')]\n",
    "\n",
    "# this decorator allows your objective function to receive a the parameters as\n",
    "# keyword arguments. This is particularly convenient when you want to set scikit-learn\n",
    "# estimator parameters\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=5, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_absolute_error\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize all the things!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these two pieces, we are now ready for sequential model-based optimisation. Here we use gaussian process-based optimisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Best score=2.8941'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=50, random_state=0)\n",
    "\n",
    "\"Best score=%.4f\" % res_gp.fun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters:\n",
      "- max_depth=2\n",
      "- learning_rate=0.157830\n",
      "- max_features=13\n",
      "- min_samples_split=2\n",
      "- min_samples_leaf=1\n"
     ]
    }
   ],
   "source": [
    "print(\"\"\"Best parameters:\n",
    "- max_depth=%d\n",
    "- learning_rate=%.6f\n",
    "- max_features=%d\n",
    "- min_samples_split=%d\n",
    "- min_samples_leaf=%d\"\"\" % (res_gp.x[0], res_gp.x[1], \n",
    "                            res_gp.x[2], res_gp.x[3], \n",
    "                            res_gp.x[4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convergence plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10d1545c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from skopt.plots import plot_convergence\n",
    "\n",
    "plot_convergence(res_gp);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
<commit_msg>Refactor imports for less noise<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning a scikit-learn estimator with `skopt`\n",
    "\n",
    "<NAME>, July 2016 <br />\n",
    "<NAME>, August 2016\n",
    "\n",
    "If you are looking for a `GridSearchCV` replacement checkout [the `BayesSearchCV` example](sklearn-gridsearchcv-replacement.ipynb) instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem statement\n",
    "\n",
    "Tuning the hyper-parameters of a machine learning model is often carried out using an exhaustive exploration of (a subset of) the space all hyper-parameter configurations (e.g., using `sklearn.model_selection.GridSearchCV`), which often results in a very time consuming operation. \n",
    "\n",
    "In this notebook, we illustrate how to couple `gp_minimize` with sklearn's estimators to tune hyper-parameters using sequential model-based optimisation, hopefully resulting in equivalent or better solutions, but within less evaluations. \n",
    "\n",
    "Note: scikit-optimize provides a dedicated interface for estimator tuning via `BayesSearchCV` class which has a similar interface to those of `GridSearchCV`. This class uses functions of skopt to perform hyperparameter search efficiently. For example usage of this class, see [the `BayesSearchCV` example](sklearn-gridsearchcv-replacement.ipynb) example notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To tune the hyper-parameters of our model we need to define a model, decide which parameters to optimize, and define the objective function we want to minimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "boston = load_boston()\n",
    "X, y = boston.data, boston.target\n",
    "n_features = X.shape[1]\n",
    "\n",
    "# gradient boosted trees tend to do well on problems like this\n",
    "reg = GradientBoostingRegressor(n_estimators=50, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to define the bounds of the dimensions of the search space we want to explore and pick the objective. In this case the cross-validation mean absolute error of a gradient boosting regressor over the Boston dataset, as a function of its hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "\n",
    "# The list of hyper-parameters we want to optimize. For each one we define the bounds,\n",
    "# the corresponding scikit-learn parameter name, as well as how to sample values\n",
    "# from that dimension (`'log-uniform'` for the learning rate)\n",
    "space  = [Integer(1, 5, name='max_depth'),\n",
    "          Real(10**-5, 10**0, \"log-uniform\", name='learning_rate'),\n",
    "          Integer(1, n_features, name='max_features'),\n",
    "          Integer(2, 100, name='min_samples_split'),\n",
    "          Integer(1, 100, name='min_samples_leaf')]\n",
    "\n",
    "# this decorator allows your objective function to receive a the parameters as\n",
    "# keyword arguments. This is particularly convenient when you want to set scikit-learn\n",
    "# estimator parameters\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=5, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_absolute_error\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize all the things!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these two pieces, we are now ready for sequential model-based optimisation. Here we use gaussian process-based optimisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Best score=2.8941'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=50, random_state=0)\n",
    "\n",
    "\"Best score=%.4f\" % res_gp.fun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters:\n",
      "- max_depth=2\n",
      "- learning_rate=0.157830\n",
      "- max_features=13\n",
      "- min_samples_split=2\n",
      "- min_samples_leaf=1\n"
     ]
    }
   ],
   "source": [
    "print(\"\"\"Best parameters:\n",
    "- max_depth=%d\n",
    "- learning_rate=%.6f\n",
    "- max_features=%d\n",
    "- min_samples_split=%d\n",
    "- min_samples_leaf=%d\"\"\" % (res_gp.x[0], res_gp.x[1], \n",
    "                            res_gp.x[2], res_gp.x[3], \n",
    "                            res_gp.x[4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convergence plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10d063518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from skopt.plots import plot_convergence\n",
    "\n",
    "plot_convergence(res_gp);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}

{'fraction_non_alphanumeric': 0.16911060349274187, 'fraction_numerical': 0.012600397907302337, 'mean_word_length': 2.74090407938258, 'pattern_counts': {'":': 180, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4728182', 'n_tokens_mistral': 4788, 'n_tokens_neox': 4626, 'n_words': 1147}