<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Categorical Encodings\n",
    "\n",
    "In this tutorial I'll show you methods for representing categorical variables as numerical features. This will typically improve the performance of your models. In general these encodings will be learned from the data itself. The two most basic encodings are one-hot encoding which you should be familiar with and label encoding which you saw in the first tutorial. Here I'll cover count encoding, target encoding (and variations), and learning encodings with singular value decomposition.\n",
    "\n",
    "First I'm going to load in the data and rebuild the baseline model from the first tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "ks = pd.read_csv('../input/kickstarter-projects/ks-projects-201801.csv',\n",
    "                 parse_dates=['deadline', 'launched'])\n",
    "\n",
    "# Drop live projects\n",
    "ks = ks.query('state != \"live\"')\n",
    "\n",
    "# Add outcome column, \"successful\" == 1, others are 0\n",
    "ks = ks.assign(outcome=(ks['state'] == 'successful').astype(int))\n",
    "\n",
    "# Timestamp features\n",
    "ks = ks.assign(hour=ks.launched.dt.hour,\n",
    "               day=ks.launched.dt.day,\n",
    "               month=ks.launched.dt.month,\n",
    "               year=ks.launched.dt.year)\n",
    "\n",
    "# Label encoding\n",
    "cat_features = ['category', 'currency', 'country']\n",
    "encoder = LabelEncoder()\n",
    "encoded = ks[cat_features].apply(encoder.fit_transform)\n",
    "\n",
    "data_cols = ['goal', 'hour', 'day', 'month', 'year', 'outcome']\n",
    "baseline_data = ks[data_cols].join(encoded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining some functions that will help us test our encodings\n",
    "import lightgbm as lgb\n",
    "from sklearn import metrics\n",
    "\n",
    "def get_data_splits(dataframe, valid_fraction=0.1):\n",
    "    valid_fraction = 0.1\n",
    "    valid_size = int(len(dataframe) * valid_fraction)\n",
    "\n",
    "    train = dataframe[:-valid_size * 2]\n",
    "    # valid size == test size, last two sections of the data\n",
    "    valid = dataframe[-valid_size * 2:-valid_size]\n",
    "    test = dataframe[-valid_size:]\n",
    "    \n",
    "    return train, valid, test\n",
    "\n",
    "def train_model(train, valid):\n",
    "    feature_cols = train.columns.drop('outcome')\n",
    "\n",
    "    dtrain = lgb.Dataset(train[feature_cols], label=train['outcome'])\n",
    "    dvalid = lgb.Dataset(valid[feature_cols], label=valid['outcome'])\n",
    "\n",
    "    param = {'num_leaves': 64, 'objective': 'binary', \n",
    "             'metric': 'auc', 'seed': 7}\n",
    "    print(\"Training model!\")\n",
    "    bst = lgb.train(param, dtrain, num_boost_round=1000, valid_sets=[dvalid], \n",
    "                    early_stopping_rounds=10, verbose_eval=False)\n",
    "\n",
    "    valid_pred = bst.predict(valid[feature_cols])\n",
    "    valid_score = metrics.roc_auc_score(valid['outcome'], valid_pred)\n",
    "    print(f\"Validation AUC score: {valid_score:.4f}\")\n",
    "    return bst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7467\n"
     ]
    }
   ],
   "source": [
    "# Training a model on the baseline data\n",
    "train, valid, _ = get_data_splits(baseline_data)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Count Encoding\n",
    "\n",
    "Count encoding replaces each categorical value with the number of times it appears in the dataset. For example, if the value \"GB\" occured 10 times in the country feature, then \"GB\" would be replaced with 10 for each occurence.\n",
    "\n",
    "I'll be using the [`categorical-encodings` package](https://github.com/scikit-learn-contrib/categorical-encoding) for most of the work here. However, the current release does not include a count encoder. It should be in the next release, but for now I'll write my own class for count encoding in the style of a scikit-learn transformer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CountEncoder:\n",
    "    def __init__(self):\n",
    "        self.mapping = {}\n",
    "        \n",
    "    def fit(self, df):\n",
    "        \"\"\" Calculates count encodings for each column in a dataframe. \"\"\"\n",
    "        for col in df.columns:\n",
    "            self.mapping[col] = df.groupby(col).count().iloc[:, 0]\n",
    "        \n",
    "    def transform(self, df):\n",
    "        \"\"\" Applies learned encodings to a dataframe. Returned datafrom has the same\n",
    "            indices and columns as original dataframe. \n",
    "        \"\"\"\n",
    "        out_df = df.copy()\n",
    "        for col, encoding in self.mapping.items():\n",
    "            out_df[col] = df[col].map(self.mapping[col]).fillna(0)\n",
    "        \n",
    "        return out_df\n",
    "    \n",
    "    def fit_transform(self, df):\n",
    "        self.fit(df)\n",
    "        return self.transform(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that defined we can apply it to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "count_enc = CountEncoder()\n",
    "count_encoded = count_enc.fit_transform(ks[cat_features])\n",
    "\n",
    "data = baseline_data.join(count_encoded.add_suffix(\"_count\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7486\n"
     ]
    }
   ],
   "source": [
    "# Training a model on the baseline data\n",
    "train, valid, test = get_data_splits(data)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding the count encoding features increase the validation score from 0.7467 to 0.7486, a slight improvement!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Target Encoding\n",
    "\n",
    "Target encoding replaces a categorical value with the target probability given that value. For example, given the country value \"CA\", you'd calculate the average outcome for all the rows with `country == 'CA'`, around 0.28. This is often blended with the target probability over the entire dataset to reduce the variance of values with few occurences.\n",
    "\n",
    "One thing to note here is that we are using the targets to create new features. This means if we include the validation or test data in the target encodings, information from these datasets will end up in the model (often called leakage). For this reason, I'll learn the target encodings from the training dataset only and apply it to the other datasets.\n",
    "\n",
    "The `category_encoders` package provides `TargetEncoder` for this. It works like scikit-learn transformers with `.fit` and `.transform` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "import category_encoders as ce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "\n",
    "# Create the encoder itself\n",
    "target_enc = ce.TargetEncoder(cols=cat_features)\n",
    "\n",
    "train, valid, _ = get_data_splits(data)\n",
    "\n",
    "# Fit the encoder using the categorical features and target\n",
    "target_enc.fit(train[cat_features], train['outcome'])\n",
    "\n",
    "# Transform the features, rename the columns with _target suffix, and join to dataframe\n",
    "train = train.join(target_enc.transform(train[cat_features]).add_suffix('_target'))\n",
    "valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>goal</th>\n",
       "      <th>hour</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>outcome</th>\n",
       "      <th>category</th>\n",
       "      <th>currency</th>\n",
       "      <th>country</th>\n",
       "      <th>category_count</th>\n",
       "      <th>currency_count</th>\n",
       "      <th>country_count</th>\n",
       "      <th>category_target</th>\n",
       "      <th>currency_target</th>\n",
       "      <th>country_target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>2015</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>1362</td>\n",
       "      <td>33853</td>\n",
       "      <td>33393</td>\n",
       "      <td>0.360190</td>\n",
       "      <td>0.357122</td>\n",
       "      <td>0.361636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30000.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>2017</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>5174</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>0.384615</td>\n",
       "      <td>0.373392</td>\n",
       "      <td>0.376631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>2013</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>5174</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>0.384615</td>\n",
       "      <td>0.373392</td>\n",
       "      <td>0.376631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>2012</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>15647</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>0.412655</td>\n",
       "      <td>0.373392</td>\n",
       "      <td>0.376631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19500.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>2015</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>10054</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>0.302625</td>\n",
       "      <td>0.373392</td>\n",
       "      <td>0.376631</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      goal  hour  day  month  year  outcome  category  currency  country  \\\n",
       "0   1000.0    12   11      8  2015        0       108         5        9   \n",
       "1  30000.0     4    2      9  2017        0        93        13       22   \n",
       "2  45000.0     0   12      1  2013        0        93        13       22   \n",
       "3   5000.0     3   17      3  2012        0        90        13       22   \n",
       "4  19500.0     8    4      7  2015        0        55        13       22   \n",
       "\n",
       "   category_count  currency_count  country_count  category_target  \\\n",
       "0            1362           33853          33393         0.360190   \n",
       "1            5174          293624         290887         0.384615   \n",
       "2            5174          293624         290887         0.384615   \n",
       "3           15647          293624         290887         0.412655   \n",
       "4           10054          293624         290887         0.302625   \n",
       "\n",
       "   currency_target  country_target  \n",
       "0         0.357122        0.361636  \n",
       "1         0.373392        0.376631  \n",
       "2         0.373392        0.376631  \n",
       "3         0.373392        0.376631  \n",
       "4         0.373392        0.376631  "
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7491\n"
     ]
    }
   ],
   "source": [
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation score is higher again, from 0.7467 to 0.7491."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Leave-One-Out Encoding\n",
    "\n",
    "Leave-One-Out encoding is the same as target encoding except that it leaves out the target for each row. That is, for each row it calculates the target probability over all the other rows with the same value. This can help the model generalize since it's leaving out information from the current record. This is implemented with `LeaveOneOutEncoder`, similar to `TargetEncoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "target_enc = ce.LeaveOneOutEncoder(cols=cat_features)\n",
    "\n",
    "train, valid, _ = get_data_splits(data)\n",
    "target_enc.fit(train[cat_features], train['outcome'])\n",
    "\n",
    "train = train.join(target_enc.transform(train[cat_features]).add_suffix('_loo'))\n",
    "valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_loo'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7491\n"
     ]
    }
   ],
   "source": [
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us the same improvement as target encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CatBoost Encoding\n",
    "\n",
    "Finally, we'll look at CatBoost encoding. This is similar to leave-one-out encoding and target encoding in that it's based on the target probablity for a given value. However with CatBoost, for each row, the target probability is calculated only from the rows before it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "target_enc = ce.CatBoostEncoder(cols=cat_features)\n",
    "\n",
    "train, valid, _ = get_data_splits(data)\n",
    "target_enc.fit(train[cat_features], train['outcome'])\n",
    "\n",
    "train = train.join(target_enc.transform(train[cat_features]).add_suffix('_cb'))\n",
    "valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_cb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7492\n"
     ]
    }
   ],
   "source": [
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This does slightly better than target and LOO encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encoding with Singular Value Decomposition\n",
    "\n",
    "Now we'll use singular value decomposition to learn encodings from pairs of categorical features. The idea here is we'll construct a matrix of co-occurences for each pair of categorical features. Each row corresponds to a value in feature A, while each column corresponds to a value in feature B. Each element is the count of rows where the value in A appears together with the value in B.\n",
    "\n",
    "We can use singular value decomposition (SVD) to find two smaller matrices that equal the count matrix when multiplied.\n",
    "\n",
    "<center><img src=\"https://i.imgur.com/mnnsBKJ.png\" width=600px></center>\n",
    "\n",
    "We can choose how many components we'll find with SVD, how long each factor vector will be. In general, longer vectors will contain more information at the cost of more memory/computation. To get the encodings for feature A, you multiply the count matrix by the small matrix for feature B.\n",
    "\n",
    "I'll show you how you can do this for one pair of features using scikit-learn's `TruncatedSVD` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "# Use 3 components in the latent vectors\n",
    "svd = TruncatedSVD(n_components=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we can use `.groupby` to count up co-occurences for any pair of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country  category\n",
       "0        0            3\n",
       "         1            1\n",
       "         2            3\n",
       "         3            1\n",
       "         5            3\n",
       "         6            1\n",
       "         7           10\n",
       "         8           22\n",
       "         9            1\n",
       "         10           6\n",
       "Name: outcome, dtype: int64"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train, valid, _ = get_data_splits(data)\n",
    "\n",
    "# Create a sparse matrix with cooccurence counts\n",
    "pair_counts = train.groupby(['country', 'category'])['outcome'].count()\n",
    "pair_counts.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a series with a two-level index. We want to convert this into a matrix with `country` on one axis and `category` on the other. To do this, we can use `.unstack`. By default it'll put `NaN`s where data doesn't exist, but we can tell it to fill those spots with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>category</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>149</th>\n",
       "      <th>150</th>\n",
       "      <th>151</th>\n",
       "      <th>152</th>\n",
       "      <th>153</th>\n",
       "      <th>154</th>\n",
       "      <th>155</th>\n",
       "      <th>156</th>\n",
       "      <th>157</th>\n",
       "      <th>158</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>17</td>\n",
       "      <td>94</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>34</td>\n",
       "      <td>17</td>\n",
       "      <td>199</td>\n",
       "      <td>218</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>22</td>\n",
       "      <td>4</td>\n",
       "      <td>185</td>\n",
       "      <td>9</td>\n",
       "      <td>69</td>\n",
       "      <td>15</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33</td>\n",
       "      <td>34</td>\n",
       "      <td>160</td>\n",
       "      <td>32</td>\n",
       "      <td>16</td>\n",
       "      <td>80</td>\n",
       "      <td>42</td>\n",
       "      <td>275</td>\n",
       "      <td>298</td>\n",
       "      <td>32</td>\n",
       "      <td>...</td>\n",
       "      <td>68</td>\n",
       "      <td>5</td>\n",
       "      <td>254</td>\n",
       "      <td>52</td>\n",
       "      <td>133</td>\n",
       "      <td>36</td>\n",
       "      <td>9</td>\n",
       "      <td>49</td>\n",
       "      <td>44</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>17</td>\n",
       "      <td>29</td>\n",
       "      <td>33</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>93</td>\n",
       "      <td>125</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>109</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>41</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>39</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>49</td>\n",
       "      <td>85</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>46</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>117</td>\n",
       "      <td>13</td>\n",
       "      <td>...</td>\n",
       "      <td>27</td>\n",
       "      <td>2</td>\n",
       "      <td>107</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>52</td>\n",
       "      <td>70</td>\n",
       "      <td>272</td>\n",
       "      <td>78</td>\n",
       "      <td>29</td>\n",
       "      <td>180</td>\n",
       "      <td>71</td>\n",
       "      <td>568</td>\n",
       "      <td>502</td>\n",
       "      <td>55</td>\n",
       "      <td>...</td>\n",
       "      <td>76</td>\n",
       "      <td>9</td>\n",
       "      <td>483</td>\n",
       "      <td>25</td>\n",
       "      <td>273</td>\n",
       "      <td>64</td>\n",
       "      <td>12</td>\n",
       "      <td>99</td>\n",
       "      <td>42</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>22</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>74</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>48</td>\n",
       "      <td>111</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>62</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>20</td>\n",
       "      <td>9</td>\n",
       "      <td>33</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>64</td>\n",
       "      <td>107</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>28</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "      <td>42</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>33</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>31</td>\n",
       "      <td>43</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>27</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>330</td>\n",
       "      <td>477</td>\n",
       "      <td>1575</td>\n",
       "      <td>421</td>\n",
       "      <td>123</td>\n",
       "      <td>1651</td>\n",
       "      <td>470</td>\n",
       "      <td>4094</td>\n",
       "      <td>3081</td>\n",
       "      <td>421</td>\n",
       "      <td>...</td>\n",
       "      <td>651</td>\n",
       "      <td>43</td>\n",
       "      <td>2466</td>\n",
       "      <td>413</td>\n",
       "      <td>4020</td>\n",
       "      <td>746</td>\n",
       "      <td>92</td>\n",
       "      <td>1406</td>\n",
       "      <td>500</td>\n",
       "      <td>174</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23 rows × 159 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "category  0    1     2    3    4     5    6     7     8    9    ...  149  150  \\\n",
       "country                                                         ...             \n",
       "0           3    1     3    1    0     3    1    10    22    1  ...    4    0   \n",
       "1          13   17    94   17   10    34   17   199   218   20  ...   22    4   \n",
       "2           2    2     6    2    1     4    1    14    18    1  ...    1    0   \n",
       "3          33   34   160   32   16    80   42   275   298   32  ...   68    5   \n",
       "4           0    2    19    0    0     4    0    13    34    0  ...    5    0   \n",
       "5          17   29    33    6    6     9    7    93   125    9  ...   22    1   \n",
       "6           3    1    13    1    0     2    0    17    41    4  ...    4    0   \n",
       "7          10   12    39    4    1     8    6    49    85    5  ...   12    0   \n",
       "8          14    8    46    3    1    15    0    50   117   13  ...   27    2   \n",
       "9          52   70   272   78   29   180   71   568   502   55  ...   76    9   \n",
       "10         10    0    10    0    0     0    1    10     7    1  ...    6    0   \n",
       "11          3    1     6    2    2     5    4     5    22    3  ...    5    1   \n",
       "12         20   10    74    4    4     6    2    48   111   12  ...   15    1   \n",
       "13          0    0     2    0    0     0    0     0     1    0  ...    0    0   \n",
       "14          2    1     0    0    0     0    0     1     2    0  ...    0    0   \n",
       "15          3   12     9    1    1    15    6    16    62    4  ...    7    5   \n",
       "16          1    0     3    0    0     0    0     1     0    0  ...    0    0   \n",
       "17         20    9    33    9    5     7    0    64   107    8  ...   18    0   \n",
       "18          1    4    12    3    2     3    1     9    28    4  ...    1    0   \n",
       "19          5    1    14    2    2     8    1    23    42    3  ...    0    0   \n",
       "20          2   11    33    5    2     9    7    31    43    6  ...    7    3   \n",
       "21          2    0    27    0    0     1    1    26    10    0  ...    4    0   \n",
       "22        330  477  1575  421  123  1651  470  4094  3081  421  ...  651   43   \n",
       "\n",
       "category   151  152   153  154  155   156  157  158  \n",
       "country                                              \n",
       "0           14    1     4    6    0     2    1    0  \n",
       "1          185    9    69   15    3    13   18   13  \n",
       "2           15    2     1    4    0     3    2    0  \n",
       "3          254   52   133   36    9    49   44   24  \n",
       "4           27    0     2    4    0     2    1    0  \n",
       "5          109    0    15   11    1    12    8    7  \n",
       "6           36    1     6    7    1     2    6    1  \n",
       "7           44    2    10    3    1     7    4    2  \n",
       "8          107    4    16    6    1    14    4    8  \n",
       "9          483   25   273   64   12    99   42   68  \n",
       "10           3    1     0    0    0     0    0    0  \n",
       "11          24    3     2    8    0    11    0    8  \n",
       "12         100    2    11    3    0    11    8    1  \n",
       "13           0    0     0    0    0     0    0    0  \n",
       "14           2    0     0    0    0     0    0    0  \n",
       "15          29    4    11    4    2    17   11    0  \n",
       "16           0    0     0    0    0     0    0    0  \n",
       "17          65    0    16    7    1    12    7    3  \n",
       "18          23    1     8    2    0     6    3    0  \n",
       "19          20    1    23    4    0     2    4    1  \n",
       "20          27    2     7    2    0     2    4    0  \n",
       "21           2    0     0    1    0     1    1    0  \n",
       "22        2466  413  4020  746   92  1406  500  174  \n",
       "\n",
       "[23 rows x 159 columns]"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair_matrix = pair_counts.unstack(fill_value=0)\n",
    "pair_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59.365276</td>\n",
       "      <td>-17.716868</td>\n",
       "      <td>28.732793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>803.302921</td>\n",
       "      <td>-230.297291</td>\n",
       "      <td>311.076301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>56.609274</td>\n",
       "      <td>-16.332383</td>\n",
       "      <td>37.153160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1516.634213</td>\n",
       "      <td>-389.099596</td>\n",
       "      <td>498.836358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>78.918379</td>\n",
       "      <td>-31.018666</td>\n",
       "      <td>42.184847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>419.671057</td>\n",
       "      <td>-141.159794</td>\n",
       "      <td>240.848483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>108.802633</td>\n",
       "      <td>-30.550797</td>\n",
       "      <td>45.430678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>226.403703</td>\n",
       "      <td>-100.570748</td>\n",
       "      <td>150.670188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>293.403001</td>\n",
       "      <td>-102.305092</td>\n",
       "      <td>190.624686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3517.365433</td>\n",
       "      <td>-583.252591</td>\n",
       "      <td>1020.856020</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0           1            2\n",
       "0    59.365276  -17.716868    28.732793\n",
       "1   803.302921 -230.297291   311.076301\n",
       "2    56.609274  -16.332383    37.153160\n",
       "3  1516.634213 -389.099596   498.836358\n",
       "4    78.918379  -31.018666    42.184847\n",
       "5   419.671057 -141.159794   240.848483\n",
       "6   108.802633  -30.550797    45.430678\n",
       "7   226.403703 -100.570748   150.670188\n",
       "8   293.403001 -102.305092   190.624686\n",
       "9  3517.365433 -583.252591  1020.856020"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd_encoding = pd.DataFrame(svd.fit_transform(pair_matrix))\n",
    "svd_encoding.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a mapping of the values in the country feature, the index of the dataframe, to our encoded vectors. Next, we need to replace the values in our data with these vectors. We can do this using the `.reindex` method. This method takes the values in the country column and creates a new dataframe from from `svd_encoding` using those values as the index. Then we need to set the index back to the original index. Note that I learned the encodings from the training data, but I'm applying them to the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3517.365433</td>\n",
       "      <td>-583.252591</td>\n",
       "      <td>1020.856020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0           1            2\n",
       "0   3517.365433 -583.252591  1020.856020\n",
       "1  33124.671369   51.453250  -159.740957\n",
       "2  33124.671369   51.453250  -159.740957\n",
       "3  33124.671369   51.453250  -159.740957\n",
       "4  33124.671369   51.453250  -159.740957\n",
       "5  33124.671369   51.453250  -159.740957\n",
       "6  33124.671369   51.453250  -159.740957\n",
       "7  33124.671369   51.453250  -159.740957\n",
       "8  33124.671369   51.453250  -159.740957\n",
       "9  33124.671369   51.453250  -159.740957"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded = svd_encoding.reindex(data['country']).set_index(data.index)\n",
    "encoded.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>goal</th>\n",
       "      <th>hour</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>outcome</th>\n",
       "      <th>category</th>\n",
       "      <th>currency</th>\n",
       "      <th>country</th>\n",
       "      <th>category_count</th>\n",
       "      <th>currency_count</th>\n",
       "      <th>country_count</th>\n",
       "      <th>country_category_svd_0</th>\n",
       "      <th>country_category_svd_1</th>\n",
       "      <th>country_category_svd_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>2015</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>1362</td>\n",
       "      <td>33853</td>\n",
       "      <td>33393</td>\n",
       "      <td>3517.365433</td>\n",
       "      <td>-583.252591</td>\n",
       "      <td>1020.856020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30000.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>2017</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>5174</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>2013</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>5174</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>2012</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>15647</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19500.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>2015</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>10054</td>\n",
       "      <td>293624</td>\n",
       "      <td>290887</td>\n",
       "      <td>33124.671369</td>\n",
       "      <td>51.453250</td>\n",
       "      <td>-159.740957</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      goal  hour  day  month  year  outcome  category  currency  country  \\\n",
       "0   1000.0    12   11      8  2015        0       108         5        9   \n",
       "1  30000.0     4    2      9  2017        0        93        13       22   \n",
       "2  45000.0     0   12      1  2013        0        93        13       22   \n",
       "3   5000.0     3   17      3  2012        0        90        13       22   \n",
       "4  19500.0     8    4      7  2015        0        55        13       22   \n",
       "\n",
       "   category_count  currency_count  country_count  country_category_svd_0  \\\n",
       "0            1362           33853          33393             3517.365433   \n",
       "1            5174          293624         290887            33124.671369   \n",
       "2            5174          293624         290887            33124.671369   \n",
       "3           15647          293624         290887            33124.671369   \n",
       "4           10054          293624         290887            33124.671369   \n",
       "\n",
       "   country_category_svd_1  country_category_svd_2  \n",
       "0             -583.252591             1020.856020  \n",
       "1               51.453250             -159.740957  \n",
       "2               51.453250             -159.740957  \n",
       "3               51.453250             -159.740957  \n",
       "4               51.453250             -159.740957  "
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Join encoded feature to the dataframe, with info in the column names\n",
    "data_svd = data.join(encoded.add_prefix(\"country_category_svd_\"))\n",
    "data_svd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.7495\n"
     ]
    }
   ],
   "source": [
    "train, valid, _ = get_data_splits(data_svd)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the best score yet, 0.7495 compared to baseline of 0.7467. In practice you'd create these encodings for each pair of categorical variables, likely improving the score even more.\n",
    "\n",
    "Next up, you'll get hands-on practice at encoding categorical features."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}<commit_msg>resolve review comments for tutorial 2<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Categorical Encodings\n",
    "\n",
    "In this tutorial I'll show you methods for representing categorical variables as numerical features. This will typically improve the performance of your models. In general these encodings will be learned from the data itself. The two most basic encodings are one-hot encoding which you should be familiar with and label encoding which you saw in the first tutorial. Here I'll cover count encoding, target encoding (and variations), and learning encodings with singular value decomposition.\n",
    "\n",
    "First I'm going to load in the data and rebuild the baseline model from the first tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "ks = pd.read_csv('../input/kickstarter-projects/ks-projects-201801.csv',\n",
    "                 parse_dates=['deadline', 'launched'])\n",
    "\n",
    "# Drop live projects\n",
    "ks = ks.query('state != \"live\"')\n",
    "\n",
    "# Add outcome column, \"successful\" == 1, others are 0\n",
    "ks = ks.assign(outcome=(ks['state'] == 'successful').astype(int))\n",
    "\n",
    "# Timestamp features\n",
    "ks = ks.assign(hour=ks.launched.dt.hour,\n",
    "               day=ks.launched.dt.day,\n",
    "               month=ks.launched.dt.month,\n",
    "               year=ks.launched.dt.year)\n",
    "\n",
    "# Label encoding\n",
    "cat_features = ['category', 'currency', 'country']\n",
    "encoder = LabelEncoder()\n",
    "encoded = ks[cat_features].apply(encoder.fit_transform)\n",
    "\n",
    "data_cols = ['goal', 'hour', 'day', 'month', 'year', 'outcome']\n",
    "baseline_data = ks[data_cols].join(encoded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining some functions that will help us test our encodings\n",
    "import lightgbm as lgb\n",
    "from sklearn import metrics\n",
    "\n",
    "def get_data_splits(dataframe, valid_fraction=0.1):\n",
    "    valid_fraction = 0.1\n",
    "    valid_size = int(len(dataframe) * valid_fraction)\n",
    "\n",
    "    train = dataframe[:-valid_size * 2]\n",
    "    # valid size == test size, last two sections of the data\n",
    "    valid = dataframe[-valid_size * 2:-valid_size]\n",
    "    test = dataframe[-valid_size:]\n",
    "    \n",
    "    return train, valid, test\n",
    "\n",
    "def train_model(train, valid):\n",
    "    feature_cols = train.columns.drop('outcome')\n",
    "\n",
    "    dtrain = lgb.Dataset(train[feature_cols], label=train['outcome'])\n",
    "    dvalid = lgb.Dataset(valid[feature_cols], label=valid['outcome'])\n",
    "\n",
    "    param = {'num_leaves': 64, 'objective': 'binary', \n",
    "             'metric': 'auc', 'seed': 7}\n",
    "    print(\"Training model!\")\n",
    "    bst = lgb.train(param, dtrain, num_boost_round=1000, valid_sets=[dvalid], \n",
    "                    early_stopping_rounds=10, verbose_eval=False)\n",
    "\n",
    "    valid_pred = bst.predict(valid[feature_cols])\n",
    "    valid_score = metrics.roc_auc_score(valid['outcome'], valid_pred)\n",
    "    print(f\"Validation AUC score: {valid_score:.4f}\")\n",
    "    return bst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training a model on the baseline data\n",
    "train, valid, _ = get_data_splits(baseline_data)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Count Encoding\n",
    "\n",
    "Count encoding replaces each categorical value with the number of times it appears in the dataset. For example, if the value \"GB\" occured 10 times in the country feature, then \"GB\" would be replaced with 10 for each occurence.\n",
    "\n",
    "I'll be using the [`categorical-encodings` package](https://github.com/scikit-learn-contrib/categorical-encoding) for most of the work here. However, the current release does not include a count encoder. It should be in the next release, but for now I'll write my own class for count encoding in the style of a scikit-learn transformer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CountEncoder:\n",
    "    def __init__(self):\n",
    "        self.mapping = {}\n",
    "        \n",
    "    def fit(self, df):\n",
    "        \"\"\" Calculates count encodings for each column in a dataframe. \"\"\"\n",
    "        for col in df.columns:\n",
    "            self.mapping[col] = df.groupby(col).count().iloc[:, 0]\n",
    "        \n",
    "    def transform(self, df):\n",
    "        \"\"\" Applies learned encodings to a dataframe. Returned datafrom has the same\n",
    "            indices and columns as original dataframe. \n",
    "        \"\"\"\n",
    "        out_df = df.copy()\n",
    "        for col, encoding in self.mapping.items():\n",
    "            out_df[col] = df[col].map(self.mapping[col]).fillna(0)\n",
    "        \n",
    "        return out_df\n",
    "    \n",
    "    def fit_transform(self, df):\n",
    "        self.fit(df)\n",
    "        return self.transform(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that defined we can apply it to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "count_enc = CountEncoder()\n",
    "count_encoded = count_enc.fit_transform(ks[cat_features])\n",
    "\n",
    "data = baseline_data.join(count_encoded.add_suffix(\"_count\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training a model on the baseline data\n",
    "train, valid, test = get_data_splits(data)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding the count encoding features increase the validation score from 0.7467 to 0.7486, a slight improvement!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Target Encoding\n",
    "\n",
    "Target encoding replaces a categorical value with the average value of the target for that value of the feature. For example, given the country value \"CA\", you'd calculate the average outcome for all the rows with `country == 'CA'`, around 0.28. This is often blended with the target probability over the entire dataset to reduce the variance of values with few occurences.\n",
    "\n",
    "One thing to note here is that we are using the targets to create new features. This means if we include the validation or test data in the target encodings, information from these datasets will end up in the model (often called leakage). For this reason, I'll learn the target encodings from the training dataset only and apply it to the other datasets.\n",
    "\n",
    "The `category_encoders` package provides `TargetEncoder` for this. It works like scikit-learn transformers with `.fit` and `.transform` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import category_encoders as ce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "\n",
    "# Create the encoder itself\n",
    "target_enc = ce.TargetEncoder(cols=cat_features)\n",
    "\n",
    "train, valid, _ = get_data_splits(data)\n",
    "\n",
    "# Fit the encoder using the categorical features and target\n",
    "target_enc.fit(train[cat_features], train['outcome'])\n",
    "\n",
    "# Transform the features, rename the columns with _target suffix, and join to dataframe\n",
    "train = train.join(target_enc.transform(train[cat_features]).add_suffix('_target'))\n",
    "valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_target'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation score is higher again, from 0.7467 to 0.7491."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CatBoost Encoding\n",
    "\n",
    "Finally, we'll look at CatBoost encoding. This is similar to target encoding in that it's based on the target probablity for a given value. However with CatBoost, for each row, the target probability is calculated only from the rows before it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['category', 'currency', 'country']\n",
    "target_enc = ce.CatBoostEncoder(cols=cat_features)\n",
    "\n",
    "train, valid, _ = get_data_splits(data)\n",
    "target_enc.fit(train[cat_features], train['outcome'])\n",
    "\n",
    "train = train.join(target_enc.transform(train[cat_features]).add_suffix('_cb'))\n",
    "valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_cb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This does slightly better than target encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encoding with Singular Value Decomposition\n",
    "\n",
    "Now we'll use singular value decomposition (SVD) to learn encodings from pairs of categorical features. SVD encoding is more complex than the other encodings you've learned, but it can also be the most effective. The idea here is we'll construct a matrix of co-occurences for each pair of categorical features. Each row corresponds to a value in feature A, while each column corresponds to a value in feature B. Each element is the count of rows where the value in A appears together with the value in B.\n",
    "\n",
    "We can use singular value decomposition to find two smaller matrices that equal the count matrix when multiplied.\n",
    "\n",
    "<center><img src=\"https://i.imgur.com/mnnsBKJ.png\" width=600px></center>\n",
    "\n",
    "You can choose how long each factor vector will be. Longer vectors will contain more information at the cost of more memory/computation. To get the encodings for feature A, you multiply the count matrix by the small matrix for feature B.\n",
    "\n",
    "I'll show you how you can do this for one pair of features using scikit-learn's `TruncatedSVD` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "# Use 3 components in the latent vectors\n",
    "svd = TruncatedSVD(n_components=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we can use `.groupby` to count up co-occurences for any pair of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, valid, _ = get_data_splits(data)\n",
    "\n",
    "# Create a sparse matrix with cooccurence counts\n",
    "pair_counts = train.groupby(['country', 'category'])['outcome'].count()\n",
    "pair_counts.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a series with a two-level index. We want to convert this into a matrix with `country` on one axis and `category` on the other. To do this, we can use `.unstack`. By default it'll put `NaN`s where data doesn't exist, but we can tell it to fill those spots with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pair_matrix = pair_counts.unstack(fill_value=0)\n",
    "pair_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd_encoding = pd.DataFrame(svd.fit_transform(pair_matrix))\n",
    "svd_encoding.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a mapping of the values in the country feature, the index of the dataframe, to our encoded vectors. Next, we need to replace the values in our data with these vectors. We can do this using the `.reindex` method. This method takes the values in the country column and creates a new dataframe from from `svd_encoding` using those values as the index. Then we need to set the index back to the original index. Note that I learned the encodings from the training data, but I'm applying them to the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded = svd_encoding.reindex(data['country']).set_index(data.index)\n",
    "encoded.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join encoded feature to the dataframe, with info in the column names\n",
    "data_svd = data.join(encoded.add_prefix(\"country_category_svd_\"))\n",
    "data_svd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, valid, _ = get_data_splits(data_svd)\n",
    "bst = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the best score yet, 0.7495 compared to baseline of 0.7467. In practice you'd create these encodings for each pair of categorical variables, likely improving the score even more.\n",
    "\n",
    "Next up, you'll get hands-on practice at encoding categorical features."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
{'fraction_non_alphanumeric': 0.21033447978185504, 'fraction_numerical': 0.05815371545239041, 'mean_word_length': 1.3792912333322467, 'pattern_counts': {'":': 373, '<': 1839, '<?xml version=': 0, '>': 1839, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 80, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1877445', 'n_tokens_mistral': 29429, 'n_tokens_neox': 27253, 'n_words': 5150}