<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to select features for machine learning\n",
    "\n",
    "Which features do you use? All of them? Some of them?\n",
    "    \n",
    "__Remember__: Our goal is to find the smallest set of the available features such that the fitted model will reach its maximal predictive value. \n",
    "    \n",
    "### Why?     \n",
    " - Less complexity = reduced bias    \n",
    " - Lower dimensional space = less computation time    \n",
    " - Fewer variables = better interpretability    \n",
    " \n",
    "### How to pick?    \n",
    " - Domain expertise    \n",
    " - Regularization techniques    \n",
    " - Automated methods    \n",
    " - Dimensionality reduction    \n",
    "\n",
    "#### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import zipfile\n",
    "import requests\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fetch the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "OCCUPANCY = ('http://bit.ly/ddl-occupancy-dataset', 'occupancy.zip')\n",
    "CREDIT    = ('http://bit.ly/ddl-credit-dataset', 'credit.xls')\n",
    "CONCRETE  = ('http://bit.ly/ddl-concrete-data', 'concrete.xls')\n",
    "\n",
    "def download_data(url, name, path='data'):\n",
    "    if not os.path.exists(path):\n",
    "        os.mkdir(path)\n",
    "\n",
    "    response = requests.get(url)\n",
    "    with open(os.path.join(path, name), 'w') as f:\n",
    "        f.write(response.content)\n",
    "\n",
    "\n",
    "def download_all(path='data'):\n",
    "    for href, name in (OCCUPANCY, CREDIT, CONCRETE):\n",
    "        download_data(href, name, path)\n",
    "\n",
    "    # Extract the occupancy zip data\n",
    "    z = zipfile.ZipFile(os.path.join(path, 'occupancy.zip'))\n",
    "    z.extractall(os.path.join(path, 'occupancy'))\n",
    "\n",
    "path='data'\n",
    "download_all(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load into dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the room occupancy dataset\n",
    "occupancy = os.path.join('data','occupancy_data','datatraining.txt')\n",
    "occupancy = pd.read_csv(occupancy, sep=',')\n",
    "occupancy.columns = [\n",
    "    'date', 'temp', 'humid', 'light', 'co2', 'hratio', 'occupied'\n",
    "]\n",
    "\n",
    "# Load the credit card default dataset\n",
    "credit = os.path.join('data','credit.xls')\n",
    "credit = pd.read_excel(credit, header=1)\n",
    "credit.columns = [\n",
    "    'id', 'limit', 'sex', 'edu', 'married', 'age', 'apr_delay', 'may_delay',\n",
    "    'jun_delay', 'jul_delay', 'aug_delay', 'sep_delay', 'apr_bill', 'may_bill',\n",
    "    'jun_bill', 'jul_bill', 'aug_bill', 'sep_bill', 'apr_pay', 'may_pay', 'jun_pay',\n",
    "    'jul_pay', 'aug_pay', 'sep_pay', 'default'\n",
    "]\n",
    "\n",
    "# Load the concrete compression data set\n",
    "concrete   = pd.read_excel(os.path.join('data','concrete.xls'))\n",
    "concrete.columns = [\n",
    "    'cement', 'slag', 'ash', 'water', 'splast',\n",
    "    'coarse', 'fine', 'age', 'strength'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization techniques\n",
    "\n",
    "### LASSO  (L1 Regularization) \n",
    "LASSO forces weak features to have zeroes as coefficients, effectively dropping the least predictive features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ridge Regression (L2 Regularization) \n",
    "Ridge assigns every feature a weight, but spreads the coefficient values out more equally, shrinking but still maintaining less predictive features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ElasticNet\n",
    "ElasticNet is a linear combination of L1 and L2 regularization, meaning it combines Ridge and LASSO and essentially splits the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automated methods    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensionality reduction"
   ]
  },
  {<commit_msg>fleshed out feature selection demo notebook<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to select features for machine learning\n",
    "\n",
    "Which features do you use? All of them? Some of them?\n",
    "    \n",
    "__Remember__: Our goal is to find the smallest set of the available features such that the fitted model will reach its maximal predictive value. \n",
    "    \n",
    "### Why?     \n",
    " - Less complexity = reduced bias    \n",
    " - Lower dimensional space = less computation time    \n",
    " - Fewer variables = better interpretability    \n",
    " \n",
    "### How to pick?    \n",
    " - Domain expertise    \n",
    " - Regularization techniques    \n",
    " - Transformer methods    \n",
    " - Dimensionality reduction    \n",
    "\n",
    "#### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import zipfile\n",
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.linear_model import Ridge, Lasso, ElasticNet\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fetch the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "OCCUPANCY = ('http://bit.ly/ddl-occupancy-dataset', 'occupancy.zip')\n",
    "CREDIT    = ('http://bit.ly/ddl-credit-dataset', 'credit.xls')\n",
    "CONCRETE  = ('http://bit.ly/ddl-concrete-data', 'concrete.xls')\n",
    "\n",
    "def download_data(url, name, path='data'):\n",
    "    if not os.path.exists(path):\n",
    "        os.mkdir(path)\n",
    "\n",
    "    response = requests.get(url)\n",
    "    with open(os.path.join(path, name), 'w') as f:\n",
    "        f.write(response.content)\n",
    "\n",
    "\n",
    "def download_all(path='data'):\n",
    "    for href, name in (OCCUPANCY, CREDIT, CONCRETE):\n",
    "        download_data(href, name, path)\n",
    "\n",
    "    # Extract the occupancy zip data\n",
    "    z = zipfile.ZipFile(os.path.join(path, 'occupancy.zip'))\n",
    "    z.extractall(os.path.join(path, 'occupancy'))\n",
    "\n",
    "path='data'\n",
    "download_all(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the first dataset into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load the room occupancy dataset into a dataframe\n",
    "occupancy = os.path.join('data','occupancy','datatraining.txt')\n",
    "occupancy = pd.read_csv(occupancy, sep=',')\n",
    "occupancy.columns = [\n",
    "    'date', 'temp', 'humid', 'light', 'co2', 'hratio', 'occupied'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Separate dataframe into features and targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = occupancy[['temp', 'humid', 'light', 'co2', 'hratio']]\n",
    "labels   = occupancy['occupied']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['temp', 'humid', 'light', 'co2', 'hratio']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization techniques\n",
    "\n",
    "### LASSO  (L1 Regularization) \n",
    "LASSO forces weak features to have zeroes as coefficients, effectively dropping the least predictive features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('temp', -0.0), ('humid', 0.0), ('light', 0.0016040390303715352), ('co2', 0.0002566541934433863), ('hratio', 0.0)]\n"
     ]
    }
   ],
   "source": [
    "model = Lasso()\n",
    "model.fit(features, labels)\n",
    "print(list(zip(features, model.coef_.tolist())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ridge Regression (L2 Regularization) \n",
    "Ridge assigns every feature a weight, but spreads the coefficient values out more equally, shrinking but still maintaining less predictive features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('temp', -0.06273198200151343), ('humid', -0.0024965287829992844), ('light', 0.001762524835587705), ('co2', 0.00033448323910362167), ('hratio', 0.004924465342949892)]\n"
     ]
    }
   ],
   "source": [
    "model = Ridge()\n",
    "model.fit(features, labels)\n",
    "print(list(zip(features, model.coef_.tolist())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ElasticNet\n",
    "ElasticNet is a linear combination of L1 and L2 regularization, meaning it combines Ridge and LASSO and essentially splits the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('temp', -0.0), ('humid', 0.0), ('light', 0.0016289577972607736), ('co2', 0.00025551009978554416), ('hratio', -0.0)]\n"
     ]
    }
   ],
   "source": [
    "model = ElasticNet(l1_ratio=0.10)\n",
    "model.fit(features, labels)\n",
    "print(list(zip(features, model.coef_.tolist())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer methods    \n",
    "\n",
    "### `SelectFromModel()` \n",
    "Scikit-Learn has a meta-transformer method for selecting features based on importance weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['light', 'co2']\n"
     ]
    }
   ],
   "source": [
    "model = Lasso()\n",
    "sfm = SelectFromModel(model)\n",
    "sfm.fit(features, labels)\n",
    "print(list(features[sfm.get_support(indices=True)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['temp']\n"
     ]
    }
   ],
   "source": [
    "model = Ridge()\n",
    "sfm = SelectFromModel(model)\n",
    "sfm.fit(features, labels)\n",
    "print(list(features[sfm.get_support(indices=True)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['light']\n"
     ]
    }
   ],
   "source": [
    "model = ElasticNet()\n",
    "sfm = SelectFromModel(model)\n",
    "sfm.fit(features, labels)\n",
    "print(list(features[sfm.get_support(indices=True)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality reduction\n",
    "\n",
    "### Principal component analysis (PCA)\n",
    "Linear dimensionality reduction using Singular Value Decomposition (SVD) of the data and keeping only the most significant singular vectors to project the data into a lower dimensional space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-239.75415116 -222.70021675]\n",
      " [-234.83734556 -229.07352309]\n",
      " [-232.82363564 -226.1680712 ]\n",
      " ..., \n",
      " [-312.01464185 -194.23928716]\n",
      " [-331.53968882 -184.46791921]\n",
      " [-338.40133606 -196.6888825 ]]\n"
     ]
    }
   ],
   "source": [
    "pca = PCA(n_components=2)\n",
    "new_features = pca.fit(features).transform(features)\n",
    "print(new_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear discriminant analysis (LDA)\n",
    "A classifier with a linear decision boundary, generated by fitting class conditional densities to the data and using Bayes’ rule. The model fits a Gaussian density to each class, assuming that all classes share the same covariance matrix. Can be used to reduce the dimensionality of the input by projecting it to the most discriminative directions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.44457859]\n",
      " [ 3.4827798 ]\n",
      " [ 3.43530287]\n",
      " ..., \n",
      " [ 4.25852644]\n",
      " [ 4.29565825]\n",
      " [ 4.4748546 ]]\n"
     ]
    }
   ],
   "source": [
    "lda = LDA(n_components=2)\n",
    "new_features = lda.fit(features, labels).transform(features)\n",
    "print(new_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn more about feature selection tools within Scikit-Learn, check out http://scikit-learn.org/stable/modules/feature_selection.html.\n",
    "\n",
    "## Exercises\n",
    "Try out the above techniques yourself with the Credit Card Default and Concrete Strength datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load the credit card default dataset into a dataframe\n",
    "credit = os.path.join('data','credit.xls')\n",
    "credit = pd.read_excel(credit, header=1)\n",
    "credit.columns = [\n",
    "    'id', 'limit', 'sex', 'edu', 'married', 'age', 'apr_delay', 'may_delay',\n",
    "    'jun_delay', 'jul_delay', 'aug_delay', 'sep_delay', 'apr_bill', 'may_bill',\n",
    "    'jun_bill', 'jul_bill', 'aug_bill', 'sep_bill', 'apr_pay', 'may_pay', 'jun_pay',\n",
    "    'jul_pay', 'aug_pay', 'sep_pay', 'default'\n",
    "]\n",
    "\n",
    "# Separate dataframe into features and targets\n",
    "cred_features = credit[[\n",
    "    'limit', 'sex', 'edu', 'married', 'age', 'apr_delay', 'may_delay',\n",
    "    'jun_delay', 'jul_delay', 'aug_delay', 'sep_delay', 'apr_bill', 'may_bill',\n",
    "    'jun_bill', 'jul_bill', 'aug_bill', 'sep_bill', 'apr_pay', 'may_pay',\n",
    "    'jun_pay', 'jul_pay', 'aug_pay', 'sep_pay'\n",
    "]]\n",
    "cred_labels = credit['default']\n",
    "\n",
    "\n",
    "# Load the concrete compression dataset into a dataframe\n",
    "concrete = pd.read_excel(os.path.join('data','concrete.xls'))\n",
    "concrete.columns = [\n",
    "    'cement', 'slag', 'ash', 'water', 'splast',\n",
    "    'coarse', 'fine', 'age', 'strength'\n",
    "]\n",
    "\n",
    "# Separate dataframe into features and targets\n",
    "conc_features = concrete[[\n",
    "    'cement', 'slag', 'ash', 'water', 'splast', 'coarse', 'fine', 'age'\n",
    "]]\n",
    "conc_labels = concrete['strength']"
   ]
  },
  {
{'fraction_non_alphanumeric': 0.20496821285613373, 'fraction_numerical': 0.024487873793265835, 'mean_word_length': 2.636344178082192, 'pattern_counts': {'":': 214, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3603591', 'n_tokens_mistral': 6574, 'n_tokens_neox': 6159, 'n_words': 1301}