<commit_before><commit_msg>Notes from DataCamp Kaggle Competition course, some links<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "37a64738",
   "metadata": {},
   "source": [
    "# Journal 2022-08-23"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "83d71d9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "627cc074",
   "metadata": {},
   "source": [
    "# Links\n",
    "* [Symmetry and Topological phases (twitter thread)](https://twitter.com/MBarkeshli/status/1561417563619426309)\n",
    "  * [2208.07367 Codimension-2 defects and higher symmetries in (3+1)D topological phases\n",
    "](https://arxiv.org/abs/2208.07367)\n",
    "* [Stable Diffusion - Explained (twitter thread)](https://twitter.com/ai__pub/status/1561362542487695360)\n",
    "  * [2112.10752 High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930f0ba0",
   "metadata": {},
   "source": [
    "# DataCamp - Winning a Kaggle Competition\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead74b01",
   "metadata": {},
   "source": [
    "Capturing some of the code used to evaluate exercises as it's a nice template for evaulating performance of new features.  Will have to check SciKit Learn doc, this probably exists somewhere as a pipeline step for a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c44592e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_kfold_rmse(train, features=None, target=None):\n",
    "    #  Validation code from https://campus.datacamp.com/courses/winning-a-kaggle-competition-in-python/feature-engineering-6e5f9c7c-cf6c-4f79-b95c-fcf856dd17d7?ex=2\n",
    "    #  Copied to have an example of simplified validation loop\n",
    "    mse_scores = []\n",
    "\n",
    "    for train_index, test_index in kf.split(train):\n",
    "        train = train.fillna(0)\n",
    "        feats = features or [x for x in train.columns if x not in ['Id', 'SalePrice', 'RoofStyle', 'CentralAir']]\n",
    "        \n",
    "        fold_train, fold_test = train.loc[train_index], train.loc[test_index]\n",
    "\n",
    "        # Fit the data and make predictions\n",
    "        # Create a Random Forest object\n",
    "        rf = RandomForestRegressor(n_estimators=10, min_samples_split=10, random_state=123)\n",
    "\n",
    "        # Train a model\n",
    "        target = target or 'SalePrice'\n",
    "        rf.fit(X=fold_train[feats], y=fold_train[target])\n",
    "\n",
    "        # Get predictions for the test set\n",
    "        pred = rf.predict(fold_test[feats])\n",
    "    \n",
    "        fold_score = mean_squared_error(fold_test[target], pred)\n",
    "        mse_scores.append(np.sqrt(fold_score))\n",
    "        \n",
    "    return round(np.mean(mse_scores) + np.std(mse_scores), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c27d12d",
   "metadata": {},
   "source": [
    "Target Encoding was something I learned from the course: for categorical variables you can use the training set group mean of each category as an encoding of the categorical feature.  Seems similar to an embedding?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b7b26d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_mean_target_encoding(train, test, target, categorical, alpha=5):\n",
    "    # From https://campus.datacamp.com/courses/winning-a-kaggle-competition-in-python/feature-engineering-6e5f9c7c-cf6c-4f79-b95c-fcf856dd17d7?ex=8\n",
    "    # Calculate global mean on the train data\n",
    "    global_mean = train[target].mean()\n",
    "    \n",
    "    # Group by the categorical feature and calculate its properties\n",
    "    train_groups = train.groupby(categorical)\n",
    "    category_sum = train_groups[target].sum()\n",
    "    category_size = train_groups.size()\n",
    "    \n",
    "    # Calculate smoothed mean target statistics\n",
    "    train_statistics = (category_sum + global_mean * alpha) / (category_size + alpha)\n",
    "    \n",
    "    # Apply statistics to the test data and fill new categories\n",
    "    test_feature = test[categorical].map(train_statistics).fillna(global_mean)\n",
    "    return test_feature.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee8ed18",
   "metadata": {},
   "source": [
    "For the training encoding we need to use cross validation and take the mean value of the other folds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c60e733",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_mean_target_encoding(train, target, categorical, alpha=5):\n",
    "    # Create 5-fold cross-validation\n",
    "    kf = KFold(n_splits=5, random_state=123, shuffle=True)\n",
    "    train_feature = pd.Series(index=train.index)\n",
    "    \n",
    "    # For each folds split\n",
    "    for train_index, test_index in kf.split(train):\n",
    "        cv_train, cv_test = train.iloc[train_index], train.iloc[test_index]\n",
    "      \n",
    "        # Calculate out-of-fold statistics and apply to cv_test\n",
    "        cv_test_feature = test_mean_target_encoding(cv_train, cv_test, target, categorical, alpha)\n",
    "        \n",
    "        # Save new feature for this particular fold\n",
    "        train_feature.iloc[test_index] = cv_test_feature       \n",
    "    return train_feature.values"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
{'fraction_non_alphanumeric': 0.16428219042920572, 'fraction_numerical': 0.032724880776188126, 'mean_word_length': 2.82035175879397, 'pattern_counts': {'":': 66, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7533183', 'n_tokens_mistral': 2324, 'n_tokens_neox': 2170, 'n_words': 455}