<jupyter_start><jupyter_text>Consume Hybrid Keras/TF Model served by Tensorflow ServingThis notebook shows the client code needed to consume a hybrid Keras-Tensorflow model served over Tensorflow Serving. The Tensorflow Serving Model Server needs to be started against our MNIST CNN test model at `EXPORT_DIR_ROOT/EXPORT_MODEL_NAME` using the following command:     bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \        --port=9000 --model_name=ktf-mnist-cnn \        --model_base_path=/home/sujit/Projects/polydlot/data/tf-export/ktf-mnist-cnnCode for the client is adapted from the [mnist_client.py](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_client.py) code provided as part of the TF-Serving examples.<jupyter_code>from __future__ import division, print_function
from grpc.beta import implementations
from sklearn.preprocessing import OneHotEncoder
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2
import os
import sys
import time
import numpy as np
import tensorflow as tf
# NUM_TESTS = 10
NUM_TESTS = 1
SERVER_HOST = "localhost"
SERVER_PORT = 9000
WORK_DIR = "/tmp"

DATA_DIR = "../../data"
TEST_FILE = os.path.join(DATA_DIR, "mnist_test.csv")

IMG_SIZE = 28
NUM_CLASSES = 10
BATCH_SIZE = 1

MODEL_NAME = "ktf-mnist-cnn"<jupyter_output><empty_output><jupyter_text>Prepare Data<jupyter_code>def parse_file(filename):
    xdata, ydata = [], []
    fin = open(filename, "rb")
    i = 0
    for line in fin:
        if i % 10000 == 0:
            print("{:s}: {:d} lines read".format(
                os.path.basename(filename), i))
        cols = line.strip().split(",")
        ydata.append(int(cols[0]))
        xdata.append(np.reshape(np.array([float(x) / 255. for x in cols[1:]]), 
                                (IMG_SIZE, IMG_SIZE, 1)))
        i += 1
    fin.close()
    print("{:s}: {:d} lines read".format(os.path.basename(filename), i))
    X = np.array(xdata, dtype="float32")
    y = np.array(ydata, dtype="int32")
    return X, y

Xtest, ytest = parse_file(TEST_FILE)
print(Xtest.shape, ytest.shape)<jupyter_output>mnist_test.csv: 0 lines read
mnist_test.csv: 10000 lines read
(10000, 28, 28, 1) (10000,)<jupyter_text>Make PredictionsSince my Keras/TF hybrid model computes validation set accuracy during training, I need to pass in the labels in the original model as well, hence the labels input. In an actual prediction situation, you could simply pass a zero vector. However, the error message seems to indicate that it is looking for another placeholder (which in this case has been declared to be the output placeholder Y\_. I believe it is looking for the Keras learning phase flag, which I cannot figure out how to pass in this model.<jupyter_code>channel = implementations.insecure_channel(SERVER_HOST, SERVER_PORT)
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

for i in range(NUM_TESTS):
    request = predict_pb2.PredictRequest()
    request.model_spec.name = MODEL_NAME
    request.model_spec.signature_name = "predict"

    Xbatch, ybatch = Xtest[i], ytest[i]
    Ybatch = np.zeros((NUM_CLASSES), dtype="int32")
    print(Xbatch.shape, Ybatch.shape)
    request.inputs["images"].CopyFrom(
        tf.contrib.util.make_tensor_proto(Xbatch, shape=[1, IMG_SIZE, IMG_SIZE, 1]))
    request.inputs["labels"].CopyFrom(
        tf.contrib.util.make_tensor_proto(Ybatch, shape=[1, NUM_CLASSES]))

    result = stub.Predict(request, 10.0)<jupyter_output>(28, 28, 1) (10,)
{'fraction_non_alphanumeric': 0.10420785088957922, 'fraction_numerical': 0.020898051397910195, 'mean_word_length': 4.247407407407407, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '174843', 'license': 'Apache-2.0', 'n_tokens_mistral': 1216, 'n_tokens_neox': 1129, 'n_words': 320, 'path': 'src/tf-serving/03a-consume-model.ipynb', 'repo_name': 'sujitpal/polydlot', 'size': '3541'}