<jupyter_start><jupyter_text>Run in Colab                        View on GitHub        OverviewThis notebook illustrates the new feature of serving custom model prediction code on AI Platform. It allows us to execute arbitrary python pre-processing code prior to invoking a model, as well as post-processing on the produced predictions. In addition, you can use a model build by your **favourite Python-based ML framework**!This is all done server-side so that the client can pass data directly to AI Platform Serving in the unprocessed state.We will take advantage of this for text classification because it involves pre-processing that is not easily accomplished using native TensorFlow. Instead we will execute the the non TensorFlow pre-processing via python code on the server side.We will build a text classification model using [PyTorch](https://pytorch.org), while performing text preproessing using Keras. PyTorch is an open source deep learning platform that provides a seamless path from research prototyping to production deployment. Dataset[Hacker News](https://bigquery.cloud.google.com/table/fh-bigquery:hackernews.stories) is one of many public datasets available in [BigQuery](https://cloud.google.com/bigquery). This dataset includes titles of articles from several data sources. For the following tutorial, we extracted the titles that belong to either GitHub, The New York Times, or TechCrunch, and saved them as CSV files in a publicly shared Cloud Storage bucket at the following location: **gs://cloud-training-demos/blogs/CMLE_custom_prediction** ObjectiveThe goal of this tutorial is to:1. Process the data for text classification.2. Train a [PyTorch](https://pytorch.org) Text Classifier (locally).3. Deploy the [PyTorch](https://pytorch.org) Text Classifier, along with the preprocessing artifacts, to AI Platform Serving, using the Custom Online Prediction code.This tutorial focuses more on using this model with AI Platform Serving than on the design of the text classification model itself. For more details about text classification, please refer to [Google developer's Guide to Text Classification](https://developers.google.com/machine-learning/guides/text-classification/).  CostsThis tutorial uses billable components of Google Cloud Platform (GCP):* AI Platform* Cloud StorageLearn about [AI Platformpricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storagepricing](https://cloud.google.com/storage/pricing), and use the [PricingCalculator](https://cloud.google.com/products/calculator/)to generate a cost estimate based on your projected usage.  Authenticate your GCP account**If you are using AI Platform Notebooks**, your environment is alreadyauthenticated. Skip this step. **If you are using Colab**, run the cell below and follow the instructionswhen prompted to authenticate your account via oAuth.**Otherwise**, follow these steps:1. In the GCP Console, go to the [**Create service account key**   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).2. From the **Service account** drop-down list, select **New service account**.3. In the **Service account name** field, enter a name.4. From the **Role** drop-down list, select   **Machine Learning Engine > AI Platform Admin** and   **Storage > Storage Object Admin**.5. Click *Create*. A JSON file that contains your key downloads to yourlocal environment.6. Enter the path to your service account key as the`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell.<jupyter_code>import sys

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()

# If you are running this notebook locally, replace the string below with the
# path to your service account key and run this cell to authenticate your GCP
# account.
else:
  %env GOOGLE_APPLICATION_CREDENTIALS ''<jupyter_output><empty_output><jupyter_text>Run the following cell to install Python dependencies needed to train the model locally. When you run the training job in AI Platform,dependencies are preinstalled based on the [runtimeversion](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list)you choose.<jupyter_code>%load_ext autoreload
%autoreload 2
!pip install tensorflow==1.15.2 --user
!pip install torch --user
import tensorflow as tf
import torch
import os

print(tf.__version__) 
print(torch.__version__)<jupyter_output><empty_output><jupyter_text>Set up your GCP project**The following steps are required, regardless of your notebook environment.**1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)3. [Enable the AI Platform ("Cloud Machine Learning Engine") and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)4. Enter your project ID in the cell below. Then run the  cell to make sure theCloud SDK uses the right project for all the commands in this notebook.**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.<jupyter_code>PROJECT_ID = '[your-project-id]' # TODO (Set up your GCP Project name)
!gcloud config set project {PROJECT_ID}<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucket**The following steps are required, regardless of your notebook environment.**When you submit a training job using the Cloud SDK, you upload a Python packagecontaining your training code to a Cloud Storage bucket. AI Platform runsthe code from this package. In this tutorial, AI Platform also saves thetrained model that results from your job in the same bucket. You can thencreate an AI Platform model version based on this output in order to serveonline predictions.Set the name of your Cloud Storage bucket below. It must be unique across allCloud Storage buckets. You may also change the `REGION` variable, which is used for operationsthroughout the rest of this notebook. Make sure to [choose a region where CloudAI Platform services areavailable](https://cloud.google.com/ml-engine/docs/tensorflow/regions).<jupyter_code>BUCKET_NAME = '[your-bucket-name]' #@param {type:"string"}
REGION = 'us-central1' #@param {type:"string"}
ROOT='torch_text_classification'
MODEL_DIR=os.path.join(ROOT,'models')
PACKAGES_DIR=os.path.join(ROOT,'packages')
# Delete any previous artifacts from Google Cloud Storage
!gsutil rm -r gs://{BUCKET_NAME}/{ROOT}<jupyter_output><empty_output><jupyter_text>Download and Explore Data<jupyter_code>%%bash
gsutil cp gs://cloud-training-demos/blogs/CMLE_custom_prediction/keras_text_pre_processing/train.tsv .
gsutil cp gs://cloud-training-demos/blogs/CMLE_custom_prediction/keras_text_pre_processing/eval.tsv .
!head eval.tsv<jupyter_output><empty_output><jupyter_text>Preprocessing  Pre-processing class to be used in both training and serving<jupyter_code>%%writefile preprocess.py

from tensorflow.python.keras.preprocessing import sequence
from tensorflow.keras.preprocessing import text


class TextPreprocessor(object):
    def __init__(self, vocab_size, max_sequence_length):
        self._vocabb_size = vocab_size
        self._max_sequence_length = max_sequence_length
        self._tokenizer = None

    def fit(self, text_list):        
        # Create vocabulary from input corpus.
        tokenizer = text.Tokenizer(num_words=self._vocabb_size)
        tokenizer.fit_on_texts(text_list)
        self._tokenizer = tokenizer

    def transform(self, text_list):        
        # Transform text to sequence of integers
        text_sequence = self._tokenizer.texts_to_sequences(text_list)
        # Fix sequence length to max value. Sequences shorter than the length are
        # padded in the beginning and sequences longer are truncated
        # at the beginning.
        padded_text_sequence = sequence.pad_sequences(
          text_sequence, maxlen=self._max_sequence_length)
        return padded_text_sequence<jupyter_output><empty_output><jupyter_text>Test Prepocessing Locally<jupyter_code>from preprocess import TextPreprocessor

processor = TextPreprocessor(5, 5)
processor.fit(['hello machine learning'])
processor.transform(['hello machine learning'])<jupyter_output><empty_output><jupyter_text>Model Creation  Metadata<jupyter_code>CLASSES = {'github': 0, 'nytimes': 1, 'techcrunch': 2}  # label-to-int mapping
NUM_CLASSES = 3
VOCAB_SIZE = 20000  # Limit on the number vocabulary size used for tokenization
MAX_SEQUENCE_LENGTH = 50  # Sentences will be truncated/padded to this length<jupyter_output><empty_output><jupyter_text>Prepare data for training and evaluation<jupyter_code>import pandas as pd
import numpy as np
from preprocess import TextPreprocessor

def load_data(train_data_path, eval_data_path):
    # Parse CSV using pandas
    column_names = ('label', 'text')
    
    df_train = pd.read_csv(train_data_path, names=column_names, sep='\t')
    df_train = df_train.sample(frac=1)
    
    df_eval = pd.read_csv(eval_data_path, names=column_names, sep='\t')

    return ((list(df_train['text']), np.array(df_train['label'].map(CLASSES))),
            (list(df_eval['text']), np.array(df_eval['label'].map(CLASSES))))


((train_texts, train_labels), (eval_texts, eval_labels)) = load_data(
       'train.tsv', 'eval.tsv')

# Create vocabulary from training corpus.
processor = TextPreprocessor(VOCAB_SIZE, MAX_SEQUENCE_LENGTH)
processor.fit(train_texts)

# Preprocess the data
train_texts_vectorized = processor.transform(train_texts)
eval_texts_vectorized = processor.transform(eval_texts)<jupyter_output><empty_output><jupyter_text>Build the model<jupyter_code>%%writefile torch_model.py

import torch
import torch.nn as nn
import torch.nn.functional as F

class TorchTextClassifier(nn.Module):
    
    def __init__(self, vocab_size, embedding_dim, seq_length, num_classes, 
                 num_filters, kernel_size, pool_size, dropout_rate):
        super(TorchTextClassifier, self).__init__()

        self.embeddings = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim)
        
        self.conv1 = nn.Conv1d(seq_length, num_filters, kernel_size)
        self.max_pool1 = nn.MaxPool1d(pool_size)
        self.conv2 = nn.Conv1d(num_filters, num_filters*2, kernel_size)
        
        self.dropout = nn.Dropout(dropout_rate)
        self.dense = nn.Linear(num_filters*2, num_classes)
        

    def forward(self, x):
        
        x = self.embeddings(x)
        x = self.dropout(x)

        x = self.conv1(x)
        x = F.relu(x)
        x = self.max_pool1(x)
        
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool1d(x, x.size()[2]).squeeze(2)
        
        x = self.dropout(x)
        x = self.dense(x)
        x = F.softmax(x, 1)
        return x<jupyter_output><empty_output><jupyter_text>Train and save the model<jupyter_code>import torch
from torch.autograd import Variable
import torch.nn.functional as F

LEARNING_RATE=.001
FILTERS=64
DROPOUT_RATE=0.2
EMBEDDING_DIM=200
KERNEL_SIZE=3
POOL_SIZE=3

NUM_EPOCH=1
BATCH_SIZE=128

train_size = len(train_texts)
steps_per_epoch = int(len(train_labels)/BATCH_SIZE)

print("Train size: {}".format(train_size))
print("Batch size: {}".format(BATCH_SIZE))
print("Number of epochs: {}".format(NUM_EPOCH))
print("Steps per epoch: {}".format(steps_per_epoch))
print("Vocab Size: {}".format(VOCAB_SIZE))
print("Embed Dimensions: {}".format(EMBEDDING_DIM))
print("Sequence Length: {}".format(MAX_SEQUENCE_LENGTH))
print("")


def get_batch(step):
    start_index = step*BATCH_SIZE
    end_index = start_index + BATCH_SIZE
    x = Variable(torch.Tensor(train_texts_vectorized[start_index:end_index]).long())
    y = Variable(torch.Tensor(train_labels[start_index:end_index]).long())
    return x, y


from torch_model import TorchTextClassifier

model = TorchTextClassifier(VOCAB_SIZE, 
                            EMBEDDING_DIM, 
                            MAX_SEQUENCE_LENGTH, 
                            NUM_CLASSES, 
                            FILTERS, 
                            KERNEL_SIZE, 
                            POOL_SIZE, 
                            DROPOUT_RATE)

model.train()
loss_metric = F.cross_entropy
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)

for epoch in range(NUM_EPOCH):
    for step in range(steps_per_epoch):
        x, y = get_batch(step)
        optimizer.zero_grad()
        y_pred = model(x)
        loss = loss_metric(y_pred, y) 
        loss.backward()
        optimizer.step()
        if step % 50 == 0:
            print('Batch [{}/{}] Loss: {}'.format(step+1, steps_per_epoch, round(loss.item(),5)))
    print('Epoch [{}/{}] Loss: {}'.format(epoch+1, NUM_EPOCH, round(loss.item(),5)))
print('Final Loss: {}'.format(epoch+1, NUM_EPOCH, round(loss.item(),5)))

torch.save(model, 'torch_saved_model.pt')<jupyter_output><empty_output><jupyter_text>Save pre-processing objectWe need to save this so the same tokenizer used at training can be used to pre-process during serving<jupyter_code>import pickle
with open('./processor_state.pkl', 'wb') as f:
    pickle.dump(processor, f)<jupyter_output><empty_output><jupyter_text>Custom Model Prediction Preparation  Copy model and pre-processing object to GCS<jupyter_code>!gsutil cp torch_saved_model.pt gs://{BUCKET_NAME}/{MODEL_DIR}/
!gsutil cp processor_state.pkl gs://{BUCKET_NAME}/{MODEL_DIR}/<jupyter_output><empty_output><jupyter_text>Define Model Class<jupyter_code>%%writefile model.py

import os
import pickle
import numpy as np
import torch
from torch.autograd import Variable


class CustomModelPrediction(object):
    def __init__(self, model, processor):
        self._model = model
        self._processor = processor

    def _postprocess(self, predictions):
        labels = ['github', 'nytimes', 'techcrunch']
        label_indexes = [np.argmax(prediction) 
                             for prediction in predictions.detach().numpy()]
        return [labels[label_index] for label_index in label_indexes]


    def predict(self, instances, **kwargs):
        preprocessed_data = self._processor.transform(instances)
        predictions =  self._model(Variable(torch.Tensor(preprocessed_data).long()))
        labels = self._postprocess(predictions)
        return labels


    @classmethod
    def from_path(cls, model_dir):
        import torch 
        import torch_model
        model = torch.load(os.path.join(model_dir,'torch_saved_model.pt'))
        model.eval()
        with open(os.path.join(model_dir, 'processor_state.pkl'), 'rb') as f:
            processor = pickle.load(f)
        return cls(model, processor)<jupyter_output><empty_output><jupyter_text>Test Model Class Locally<jupyter_code># Headlines for Predictions

techcrunch=[
  'Uber shuts down self-driving trucks unit',
  'Grover raises €37M Series A to offer latest tech products as a subscription',
  'Tech companies can now bid on the Pentagon’s $10B cloud contract'
]
nytimes=[
  '‘Lopping,’ ‘Tips’ and the ‘Z-List’: Bias Lawsuit Explores Harvard’s Admissions',
  'A $3B Plan to Turn Hoover Dam into a Giant Battery',
  'A MeToo Reckoning in China’s Workplace Amid Wave of Accusations'
]
github=[
  'Show HN: Moon – 3kb JavaScript UI compiler',
  'Show HN: Hello, a CLI tool for managing social media',
  'Firefox Nightly added support for time-travel debugging'
]
requests = (techcrunch+nytimes+github)
from model import CustomModelPrediction

local_prediction = CustomModelPrediction.from_path('.')
local_prediction.predict(requests)<jupyter_output><empty_output><jupyter_text>Package up files and copy to GCSCreate a setup.py script to bundle **model.py**,**preprocess.py** and **torch_model.py**  in a tarball package. Notice that setup.py does not include the dependencies of `model.py` in the package. These dependencies are provided to your model version in other ways:`numpy` and `google-cloud-storage` are both included as part of AI Platform Prediction runtime version 1.15.`torch` is provided in a separate package, as described in a following section.<jupyter_code>%%writefile setup.py

from setuptools import setup

REQUIRED_PACKAGES = ['keras']

setup(
  name="text_classification",
  version="0.1",
  scripts=["preprocess.py", "model.py", "torch_model.py"],
  include_package_data=True,
  install_requires=REQUIRED_PACKAGES
)
!python setup.py sdist
!gsutil cp ./dist/text_classification-0.1.tar.gz gs://{BUCKET_NAME}/{PACKAGES_DIR}/text_classification-0.1.tar.gz<jupyter_output><empty_output><jupyter_text>Model Deployment to AI Platform<jupyter_code>MODEL_NAME='torch_text_classification'
MODEL_VERSION='v1'
RUNTIME_VERSION='1.15'
REGION='us-central1'
# Delete model version if any
! gcloud ai-platform versions delete {MODEL_VERSION} --model {MODEL_NAME} --quiet # run if version already created

# Delete model resource
! gcloud ai-platform models delete {MODEL_NAME} --quiet
!gcloud beta ai-platform models create {MODEL_NAME} --regions {REGION} --enable-logging --enable-console-logging<jupyter_output><empty_output><jupyter_text>Pytorch compatible packagesYou need to specify two Python packages when you create your version resource. One of these is the package containing `model.py` that you uploaded to Cloud Storage in a previous step. The other is a package containing the version of PyTorch that you need.Google Cloud provides a collection of PyTorch packages in the `gs://cloud-ai-pytorch` Cloud Storage bucket. These packages are mirrored from the official builds.For this tutorial, use `gs://cloud-ai-pytorch/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl` as your PyTorch package. This provides your version resource with PyTorch 1.3.1 for Python 3.7, built to run on a CPU in Linux.Use the following command to create your version resource:<jupyter_code>!gcloud beta ai-platform versions create {MODEL_VERSION} --model {MODEL_NAME} \
 --origin=gs://{BUCKET_NAME}/{MODEL_DIR}/ \
 --python-version=3.7 \
 --runtime-version={RUNTIME_VERSION} \
 --package-uris=gs://{BUCKET_NAME}/{PACKAGES_DIR}/text_classification-0.1.tar.gz,gs://cloud-ai-pytorch/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl \
 --machine-type=mls1-c4-m4 \
 --prediction-class=model.CustomModelPrediction<jupyter_output><empty_output><jupyter_text>Online Predictions from AI Platform Prediction<jupyter_code>from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json

# JSON format the requests
request_data = {'instances': requests}

# Authenticate and call CMLE prediction API 
credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials)

parent = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, MODEL_VERSION)
print("Model full name: {}".format(parent))
response = api.projects().predict(body=request_data, name=parent).execute()

print(response['predictions'])<jupyter_output><empty_output><jupyter_text>Cleaning upTo clean up all GCP resources used in this project, you can [delete the GCPproject](https://cloud.google.com/resource-manager/docs/creating-managing-projectsshutting_down_projects) you used for the tutorial.Alternatively, you can clean up individual resources by running the followingcommands:<jupyter_code># Delete model version resource
!gcloud ai-platform versions delete {MODEL_VERSION} --model {MODEL_NAME} --quiet

# Delete model resource
! gcloud ai-platform models delete {MODEL_NAME} --quiet<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09496868054152353, 'fraction_numerical': 0.006516467973327945, 'mean_word_length': 4.633750711439954, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 83, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '50433', 'license': 'Apache-2.0', 'n_tokens_mistral': 6010, 'n_tokens_neox': 5827, 'n_words': 1780, 'path': 'notebooks/samples/pytorch/text_classification/text_classification_using_pytorch_and_ai_platform.ipynb', 'repo_name': 'Google-Cloud-Tech-Career-Training/ai-platform-samples', 'size': '19796'}