<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using custom containers with AI Platform Training\n",
    "\n",
    "**Learning Objectives:**\n",
    "1. Learn how to create a train and a validation split with BigQuery\n",
    "1. Learn how to wrap a machine learning model into a Docker container and train in on AI Platform\n",
    "1. Learn how to use the hyperparameter tunning engine on Google Cloud to find the best hyperparameters\n",
    "1. Learn how to deploy a trained machine learning model Google Cloud as a rest API and query it\n",
    "\n",
    "In this lab, you develop a  multi-class classification model, package the model as a docker image, and run on **AI Platform Training** as a training application. The training application trains a multi-class classification model that predicts the type of forest cover from cartographic data. The [dataset](../../../datasets/covertype/README.md) used in the lab is based on **Covertype Data Set** from UCI Machine Learning Repository.\n",
    "\n",
    "Scikit-learn is one of the most useful libraries for machine learning in Python. The training code uses `scikit-learn` for data pre-processing and modeling. \n",
    "\n",
    "The code is instrumented using the `hypertune` package so it can be used with **AI Platform** hyperparameter tuning job in searching for the best combination of hyperparameter values by optimizing the metrics you specified.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import uuid\n",
    "import time\n",
    "import tempfile\n",
    "\n",
    "from googleapiclient import discovery\n",
    "from googleapiclient import errors\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from jinja2 import Template\n",
    "from kfp.components import func_to_container_op\n",
    "from typing import NamedTuple\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the command in the cell below to install gcsfs package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install gcsfs==0.7.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare lab dataset\n",
    "\n",
    "Set environment variable so that we can use them throughout the entire lab.\n",
    "\n",<commit_msg>- Updated gcsfs version to 0.8 in solution notebook<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using custom containers with AI Platform Training\n",
    "\n",
    "**Learning Objectives:**\n",
    "1. Learn how to create a train and a validation split with BigQuery\n",
    "1. Learn how to wrap a machine learning model into a Docker container and train in on AI Platform\n",
    "1. Learn how to use the hyperparameter tunning engine on Google Cloud to find the best hyperparameters\n",
    "1. Learn how to deploy a trained machine learning model Google Cloud as a rest API and query it\n",
    "\n",
    "In this lab, you develop a  multi-class classification model, package the model as a docker image, and run on **AI Platform Training** as a training application. The training application trains a multi-class classification model that predicts the type of forest cover from cartographic data. The [dataset](../../../datasets/covertype/README.md) used in the lab is based on **Covertype Data Set** from UCI Machine Learning Repository.\n",
    "\n",
    "Scikit-learn is one of the most useful libraries for machine learning in Python. The training code uses `scikit-learn` for data pre-processing and modeling. \n",
    "\n",
    "The code is instrumented using the `hypertune` package so it can be used with **AI Platform** hyperparameter tuning job in searching for the best combination of hyperparameter values by optimizing the metrics you specified.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import uuid\n",
    "import time\n",
    "import tempfile\n",
    "\n",
    "from googleapiclient import discovery\n",
    "from googleapiclient import errors\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from jinja2 import Template\n",
    "from kfp.components import func_to_container_op\n",
    "from typing import NamedTuple\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the command in the cell below to install gcsfs package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install gcsfs==0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare lab dataset\n",
    "\n",
    "Set environment variable so that we can use them throughout the entire lab.\n",
    "\n",
{'fraction_non_alphanumeric': 0.12216197899017282, 'fraction_numerical': 0.0028803795323619113, 'mean_word_length': 3.2224606580829755, 'pattern_counts': {'":': 40, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6959791', 'n_tokens_mistral': 1784, 'n_tokens_neox': 1757, 'n_words': 671}