<jupyter_start><jupyter_text>Text Classification using TensorFlow/Keras on Cloud ML Engine Leveraging a pre-trained embeddingThis notebook illustrates: Downloading a pre-trained text embedding Creating a text classification model using the Estimator API with a Keras model Training on Cloud ML Engine Deploying model Predicting with model<jupyter_code># change these to try this notebook out
BUCKET = 'cloud-training-demos-ml'
PROJECT = 'cloud-training-demos'
REGION = 'us-central1'
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION
os.environ['TFVERSION'] = '1.8'
import tensorflow as tf
print(tf.__version__)<jupyter_output><empty_output><jupyter_text>Pre-requisitesEnsure you have the training files generated from the txtcls_fromscratch notebook. If you don't, go back and run the txtcls_fromscratch notebook first.<jupyter_code>!wc -l data/txtcls/*.tsv<jupyter_output><empty_output><jupyter_text>Download Pre-trained EmbeddingIn previous notebook we trained our word embedding from scratch. Often times we get better performance and/or converge faster by leveraging a pre-trained embedding. This is a similar concept to transfer learning during image classification.We will use the popular GloVe embedding which is trained on Wikipedia as well as various news sources like the New York Times.You can read more about Glove at the project homepage: https://nlp.stanford.edu/projects/glove/You can download the embedding files directly from the stanford.edu site, but we've rehosted it in a GCS bucket for faster download speed.*Note: The emedding file is 660MB so the following cell may take several minutes to run if you have a slower internet connection*<jupyter_code>!gsutil cp gs://cloud-data-ml-text/glove.6B.200d.txt data/txtcls/glove.6B.200d.txt<jupyter_output><empty_output><jupyter_text>TensorFlow/Keras CodePlease explore the code in this directory: `model.py` contains the TensorFlow model and `task.py` parses command line arguments and launches off the training job. This is the **same code** as in the previous notebook. The only difference is we invoke it with the `--embedding_path` parameter  Run LocallyLet's make sure the code compiles by running for a fraction of an epoch locally. Note the new `--embedding_path` parameter<jupyter_code>%bash
OUTDIR=${PWD}/txtcls_trained
rm -rf $OUTDIR
gcloud ml-engine local train \
   --module-name=trainer.task \
   --package-path=${PWD}/txtclsmodel/trainer \
   -- \
   --output_dir=$OUTDIR \
   --train_data_path=${PWD}/data/txtcls/train.tsv \
   --eval_data_path=${PWD}/data/txtcls/eval.tsv \
   --embedding_path=${PWD}/data/txtcls/glove.6B.200d.txt \
   --num_epochs=0.1<jupyter_output><empty_output><jupyter_text>What was your accuracy? You should notice that even when training for only 1/10th of an epoch our model has started to perform better than random. This did not happen previously.Because our model no longer has to learn an embedding from scratch, it can converge on the classification problem at hand much more quickly.  Train on the CloudLet's first copy our embedding file to the cloud:<jupyter_code>!gsutil cp data/txtcls/glove.6B.200d.txt gs://$BUCKET/txtcls/
%%bash
OUTDIR=gs://${BUCKET}/txtcls/trained_finetune
JOBNAME=txtcls_$(date -u +%y%m%d_%H%M%S)
REGION=us-central1
gsutil -m rm -rf $OUTDIR
gcloud ml-engine jobs submit training $JOBNAME \
 --region=$REGION \
 --module-name=trainer.task \
 --package-path=${PWD}/txtclsmodel/trainer \
 --job-dir=$OUTDIR \
 --scale-tier=BASIC_GPU \
 --runtime-version=$TFVERSION \
 -- \
 --output_dir=$OUTDIR \
 --train_data_path=gs://${BUCKET}/txtcls/train.tsv \
 --eval_data_path=gs://${BUCKET}/txtcls/eval.tsv \
 --embedding_path=gs://${BUCKET}/txtcls/glove.6B.200d.txt \
 --num_epochs=5<jupyter_output><empty_output><jupyter_text>Monitor training with TensorBoardIf tensorboard appears blank try refreshing after 10 minutes<jupyter_code>from google.datalab.ml import TensorBoard
TensorBoard().start('gs://{}/txtcls/trained_finetune'.format(BUCKET))
for pid in TensorBoard.list()['pid']:
  TensorBoard().stop(pid)
  print 'Stopped TensorBoard with pid {}'.format(pid)<jupyter_output><empty_output><jupyter_text>ResultsWhat accuracy did you get? Was it an improvement over training the embedding from scratch? While the final accuracy may not change significantly, you should notice the model was able to converge to it more quickly given the pre-trained embedding.  Deploy trained model Once your training completes you will see your exported models in the output directory specified in Google Cloud Storage. You should see one model for each training checkpoint (default is every 1000 steps).<jupyter_code>%bash
gsutil ls gs://${BUCKET}/txtcls/trained_finetune/export/exporter/
%%bash
MODEL_NAME="txtcls"
MODEL_VERSION="v1_finetune"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/txtcls/trained_finetune/export/exporter/ | tail -1)
#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME} --quiet
#gcloud ml-engine models delete ${MODEL_NAME}
#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION
gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=$TFVERSION<jupyter_output><empty_output><jupyter_text>Get PredictionsHere are some actual hacker news headlines gathered from July 2018. These titles were not part of the training or evaluation datasets.<jupyter_code>techcrunch=[
  'Uber shuts down self-driving trucks unit',
  'Grover raises €37M Series A to offer latest tech products as a subscription',
  'Tech companies can now bid on the Pentagon’s $10B cloud contract'
]
nytimes=[
  '‘Lopping,’ ‘Tips’ and the ‘Z-List’: Bias Lawsuit Explores Harvard’s Admissions',
  'A $3B Plan to Turn Hoover Dam into a Giant Battery',
  'A MeToo Reckoning in China’s Workplace Amid Wave of Accusations'
]
github=[
  'Show HN: Moon – 3kb JavaScript UI compiler',
  'Show HN: Hello, a CLI tool for managing social media',
  'Firefox Nightly added support for time-travel debugging'
]<jupyter_output><empty_output><jupyter_text>Our serving input function expects the already tokenized representations of the headlines, so we do that pre-processing in the code before calling the REST API.<jupyter_code>import pickle
from tensorflow.python.keras.preprocessing import sequence
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json

requests = techcrunch+nytimes+github

# Tokenize and pad sentences using same mapping used in the deployed model
tokenizer = pickle.load( open( "txtclsmodel/tokenizer.pickled", "rb" ) )

requests_tokenized = tokenizer.texts_to_sequences(requests)
requests_tokenized = sequence.pad_sequences(requests_tokenized,maxlen=50)

# JSON format the requests
request_data = {'instances':requests_tokenized.tolist()}

# Authenticate and call CMLE prediction API 
credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials,
            discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json')

parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'txtcls', 'v1_finetune')
response = api.projects().predict(body=request_data, name=parent).execute()

# Format and print response
for i in xrange(len(requests)):
  print('\n{}'.format(requests[i]))
  print(' github    : {}'.format(response['predictions'][i]['dense_1'][0]))
  print(' nytimes   : {}'.format(response['predictions'][i]['dense_1'][1]))
  print(' techcrunch: {}'.format(response['predictions'][i]['dense_1'][2]))<jupyter_output><empty_output><jupyter_text>Bonus: Native Tensorflow Predictions  Why Native?Up until now we've been using python functions to do our data pre-processing. This is fine during training, but during serving it adds the limitation that we need a python client in the prediction pipeline.This limits your serving flexibility. For example, lets say you want to be able to serve this model locally (offline) on a mobile phone. How would you do it? It's non trivial to execute python code on Android. Plus if your tokenization vocabulary ever changed you'd need to update all the clients or else they would be out of sync with the model.A better way would be to have all of our serving pre-processing to be done using native Tensorflow operations. This way we can take advantage of Tensorflow's hardware agnostic execution engine, and leverage the engineering efforts the Tensorflow team put into making sure our code works whether we're running on a server, mobile, or an embedded device!  TensorFlow/Keras Code (Native)Please explore the code in this directory: `model_native.py` contains the TensorFlow model and `task.py` parses command line arguments and launches off the training job. In particular look for the follwing:1. [tf.keras.preprocessing.text.Tokenizer.fit_on_texts()](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizerfit_on_texts) to generate a mapping from our word vocabulary to integers2. [tf.gfile](https://www.tensorflow.org/api_docs/python/tf/gfile/GFile) to write the vocabulary mapping to disk3. [tf.contrib.lookup.index_table_from_file()](https://www.tensorflow.org/api_docs/python/tf/contrib/lookup/index_table_from_file) to encode our sentences into a tensor of their respective word-integers, based on the vocabulary mapping written to disk in the previous step4. [tf.pad](https://www.tensorflow.org/api_docs/python/tf/pad) to pad all sequences to be the same length  Train on the Cloud (Native)Note the new `--native` parameter. This tells `task.py` to call `model_native.py`<jupyter_code>%%bash
OUTDIR=gs://${BUCKET}/txtcls/trained_finetune_native
JOBNAME=txtcls_$(date -u +%y%m%d_%H%M%S)
REGION=us-central1
gsutil -m rm -rf $OUTDIR
gcloud ml-engine jobs submit training $JOBNAME \
 --region=$REGION \
 --module-name=trainer.task \
 --package-path=${PWD}/txtclsmodel/trainer \
 --job-dir=$OUTDIR \
 --scale-tier=BASIC_GPU \
 --runtime-version=$TFVERSION \
 -- \
 --output_dir=$OUTDIR \
 --train_data_path=gs://${BUCKET}/txtcls/train.tsv \
 --eval_data_path=gs://${BUCKET}/txtcls/eval.tsv \
 --embedding_path=gs://${BUCKET}/txtcls/glove.6B.200d.txt \
 --native \
 --num_epochs=5<jupyter_output><empty_output><jupyter_text>Deploy trained model (Native)<jupyter_code>%%bash
MODEL_NAME="txtcls"
MODEL_VERSION="v1_finetune_native"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/txtcls/trained_finetune_native/export/exporter/ | tail -1)
#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME} --quiet
#gcloud ml-engine models delete ${MODEL_NAME}
#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION
gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=$TFVERSION<jupyter_output><empty_output><jupyter_text>Get Predictions (Native) Note how we can now feed the titles directly to the model! The text tokenization is done for us inside of the Tensorflow serving_input_fn.<jupyter_code>from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json

# JSON format the requests
requests = (techcrunch+nytimes+github)
requests = [x.lower() for x in requests] 
request_data = {'instances': requests}

# Authenticate and call CMLE prediction API 
credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials,
            discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json')

parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'txtcls', 'v1_finetune_native')
response = api.projects().predict(body=request_data, name=parent).execute()

# Format and print response
for i in xrange(len(requests)):
  print('\n{}'.format(requests[i]))
  print(' github    : {}'.format(response['predictions'][i]['dense_1'][0]))
  print(' nytimes   : {}'.format(response['predictions'][i]['dense_1'][1]))
  print(' techcrunch: {}'.format(response['predictions'][i]['dense_1'][2]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10821792851811925, 'fraction_numerical': 0.007048677336429223, 'mean_word_length': 5.504854368932039, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '223089', 'license': 'Apache-2.0', 'n_tokens_mistral': 3711, 'n_tokens_neox': 3651, 'n_words': 1159, 'path': 'courses/machine_learning/deepdive/09_sequence/txtcls_finetuning.ipynb', 'repo_name': 'fullstackenviormentss/training-data-analyst', 'size': '12059'}