<jupyter_start><jupyter_text>Using batch prediction with Keras models on AI Platform Functional API example **Use a TensorFlow Enterprise 2.3 instance to run this notebook.** Define environment variables and create a storage bucket.<jupyter_code># Change these to try this notebook out.
BUCKET = 'your-project-id'
PROJECT = 'your-project-id'
REGION = 'us-central1'
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION
%%bash
if ! gsutil ls | grep -q gs://${BUCKET}/; then
    gsutil mb -l ${REGION} gs://${BUCKET}
fi<jupyter_output><empty_output><jupyter_text>Import modules and prepare training data. (You use the [Boston Housing price regression dataset](https://keras.io/api/datasets/boston_housing/load_data-function) as an example.)<jupyter_code>import numpy as np
from pandas import DataFrame

import tensorflow as tf
from tensorflow.keras import layers, models, datasets
(x_train, y_train), (x_test, y_test) = datasets.boston_housing.load_data(
    path="boston_housing.npz", test_split=0.2, seed=113
)
df = DataFrame(x_train)
df.columns = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']
df.describe()<jupyter_output><empty_output><jupyter_text>Create a model to predict average prices and train it.<jupyter_code>model = models.Sequential()
model.add(layers.Input(shape=(13,), name='features'))
model.add(layers.Dense(8, activation='relu', name='hidden'))
model.add(layers.Dense(1, activation=None, name='output'))
model.summary()
model.compile(optimizer='adam', loss='mse')<jupyter_output><empty_output><jupyter_text>Training will take a few minutes.<jupyter_code>history = model.fit(
    x_train, y_train, validation_data=(x_test, y_test),
    batch_size=64, epochs=1000, verbose=0)
DataFrame({'train_loss': history.history['loss'], 
           'test_loss': history.history['val_loss']}).plot()
DataFrame({'train_rmse': np.sqrt(history.history['loss']), 
           'test_rmse': np.sqrt(history.history['val_loss'])}).plot(ylim=[0, 15])<jupyter_output><empty_output><jupyter_text>Create a wrapper model so that you can add an identifier key to each example.<jupyter_code>key = layers.Input(shape=(), name='key', dtype='int32')
pred = layers.Concatenate(name='prediction_with_key')(
    [model.output, tf.cast(layers.Reshape((1,))(key), tf.float32)])
wrapper_model = models.Model(inputs=[model.input, key], outputs=pred)
wrapper_model.compile()
wrapper_model.summary()
tf.keras.utils.plot_model(
    model=wrapper_model, to_file="dnn_model.png", show_shapes=False, rankdir="LR")<jupyter_output><empty_output><jupyter_text>Use the predict method of the wrapper model.<jupyter_code>wrapper_model.predict([x_test[:10], tf.constant(range(10))])<jupyter_output><empty_output><jupyter_text>Export the wrapper model in the saved_model format.<jupyter_code>export_path = "./export"
wrapper_model.save(export_path, save_format='tf')
%%bash
saved_model_cli show --dir "./export" --tag_set serve --signature_def serving_default<jupyter_output>The given SavedModel SignatureDef contains the following input(s):
  inputs['features'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 13)
      name: serving_default_features:0
  inputs['key'] tensor_info:
      dtype: DT_INT32
      shape: (-1)
      name: serving_default_key:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['prediction_with_key'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 2)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict<jupyter_text>Make local predictions with the exported model.<jupyter_code>serving_fn = models.load_model(export_path).signatures['serving_default']
serving_fn(features=tf.constant(x_test[:10].tolist()), key=tf.constant(range(10)))<jupyter_output><empty_output><jupyter_text>Deploy the exported model to AI Platform.**Note**: the latest runtime that supports batch prediction is 2.1https://cloud.google.com/ai-platform/training/docs/runtime-version-list<jupyter_code>%%bash
MODEL_NAME="housing_price1"
MODEL_VERSION="v1"
MODEL_LOCATION="./export"
gcloud ai-platform models create ${MODEL_NAME} --regions $REGION
gcloud ai-platform versions create ${MODEL_VERSION} --model ${MODEL_NAME} \
  --origin ${MODEL_LOCATION} --runtime-version 2.1 \
  --staging-bucket gs://$BUCKET --region global<jupyter_output>Using endpoint [https://ml.googleapis.com/]
Created ml engine model [projects/your-project-id/models/housing_price1].
Using endpoint [https://ml.googleapis.com/]
Creating version (this might take a few minutes)......
...............................................................................................................................................................................................................................................done.<jupyter_text>Make online predictions with the deployed model.<jupyter_code>from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json

credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials, cache_discovery=False)

request_data =  {'instances':
  [
    {"features": [18.0846, 0, 18.1, 0, 0.679, 6.434, 100, 1.8347, 24, 666, 20.2, 27.25, 29.05], "key": 0},
    {"features": [0.12329, 0, 10.01, 0, 0.547, 5.913, 92.9, 2.3534, 6.0, 432, 17.8, 394.95, 16.21], "key": 1},
  ]
}

parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'housing_price1', 'v1')
response = api.projects().predict(body=request_data, name=parent).execute()
print(json.dumps(response, sort_keys = True, indent = 4))<jupyter_output>{
    "predictions": [
        {
            "prediction_with_key": [
                7.977111339569092,
                0.0
            ]
        },
        {
            "prediction_with_key": [
                21.491554260253906,
                1.0
            ]
        }
    ]
}<jupyter_text>Create an input file and execute a batch prediction job.<jupyter_code>with open('./input.json', 'w') as f:
    for features, key in zip(x_test[:10].tolist(), np.arange(10).tolist()):
        print(json.dumps({'features': features, 'key':key}), file=f)
%%bash
gsutil cp ./input.json gs://$BUCKET/input/input.json
gsutil cat gs://$BUCKET/input/input.json
%%bash
MODEL_NAME="housing_price1"
MODEL_VERSION="v1"
JOB_NAME="batch_pred_$(date +'%Y%m%d%H%M%S')"
INPUT_PATHS="gs://$BUCKET/input/*"
OUTPUT_PATH="gs://$BUCKET/output"
DATA_FORMAT="text" # JSON data format
gcloud ai-platform jobs submit prediction $JOB_NAME \
    --model $MODEL_NAME \
    --input-paths $INPUT_PATHS \
    --output-path $OUTPUT_PATH \
    --region $REGION \
    --data-format $DATA_FORMAT<jupyter_output>jobId: batch_pred_20210112112025
state: QUEUED<jupyter_text>Open [Cloud Console](https://console.cloud.google.com/ai-platform/jobs) and wait for the batch prediction job to finish. When it has successfully completed, the prediction results are stored in the storage bucket.<jupyter_code>!gsutil cat gs://$BUCKET/output/prediction.results*<jupyter_output>{"prediction_with_key": [7.977108955383301, 0.0]}
{"prediction_with_key": [21.491567611694336, 1.0]}
{"prediction_with_key": [23.832067489624023, 2.0]}
{"prediction_with_key": [29.424741744995117, 3.0]}
{"prediction_with_key": [26.124998092651367, 4.0]}
{"prediction_with_key": [20.89533042907715, 5.0]}
{"prediction_with_key": [29.16432762145996, 6.0]}
{"prediction_with_key": [26.358930587768555, 7.0]}
{"prediction_with_key": [20.769208908081055, 8.0]}
{"prediction_with_key": [20.85109519958496, 9.0]}
{'fraction_non_alphanumeric': 0.1757631822386679, 'fraction_numerical': 0.05167173252279635, 'mean_word_length': 4.382645803698435, 'pattern_counts': {'":': 17, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '264400', 'license': 'Apache-2.0', 'n_tokens_mistral': 2782, 'n_tokens_neox': 2548, 'n_words': 533, 'path': 'tutorials/ai-platform-batch-prediction-keras/notebooks/functional_API_example.ipynb', 'repo_name': 'agmsb/community', 'size': '7567'}