title serving tensorflow models with tfserving dimitre oliveirahttps www linkedin comindimitreoliveira7a1a0113a date created 20230102 last modified 20230102 description how to serve tensorflow models with tensorflow serving accelerator none introduction once you build a machine learning model the next step is to serve it you may want to do that by exposing your model as an endpoint service there are many frameworks that you can use to do that but the tensorflow ecosystem has its own solution called tensorflow servinghttps www tensorflow orgtfxguideserving from the tensorflow serving github pagehttps github comtensorflowserving tensorflow serving is a flexible highperformance serving system for machine learning models designed for production environments it deals with the inference aspect of machine learning taking models after training and managing their lifetimes providing clients with versioned access via a highperformance referencecounted lookup table tensorflow serving provides outofthebox integration with tensorflow models but can be easily extended to serve other types of models and data to note a few features it can serve multiple models or multiple versions of the same model simultaneously it exposes both grpc as well as http inference endpoints it allows deployment of new model versions without changing any client code it supports canarying new versions and ab testing experimental models it adds minimal latency to inference time due to efficient lowoverhead implementation it features a scheduler that groups individual inference requests into batches for joint execution on gpu with configurable latency controls it supports many servables tensorflow models embeddings vocabularies feature transformations and even nontensorflowbased machine learning models this guide creates a simple mobilenethttps arxiv orgabs1704 04861 model using the keras applications apihttps keras ioapiapplications and then serves it with tensorflow servinghttps www tensorflow orgtfxguideserving the focus is on tensorflow serving rather than the modeling and training in tensorflow note you can find a colab notebook with the full working code at this linkhttps colab research google comdrive1nwuija4so1xzyu0ngq8txsgto295mu uspsharing dependencies model here we load a pretrained mobilenethttps arxiv orgabs1704 04861 from the keras applicationshttps keras ioapiapplications this is the model that we are going to serve preprocessing most models don t work out of the box on raw data they usually require some kind of preprocessing step to adjust the data to the model requirements in the case of this mobilenet we can see from its api pagehttps keras ioapiapplicationsmobilenet that it requires three basic steps for its input images pixel values normalized to the 0 1 range pixel values scaled to the 1 1 range images with the shape of 224 224 3 meaning height width channels we can do all of that with the following function scale normalize and resizes images image image 255 0 scale image image mean std normalize image tf image resizeimage shape resize return image postprocessing download humanreadable labels for imagenet imagenetlabelsurl https storage googleapis comdownload tensorflow orgdataimagenetlabels txt response requests getimagenetlabelsurl skiping backgroung class labels x for x in response text splitn if x 1 convert the labels to the tensorflow data format tflabels tf constantlabels dtypetf string def postprocessprediction labelstflabels now let s download a banana picture and see how everything comes together save the model to load our trained model into tensorflow serving we first need to save it in savedmodelhttps www tensorflow orgversionsr1 15apidocspythontfsavedmodel format this will create a protobuf file in a welldefined directory hierarchy and will include a version number tensorflow servinghttps www tensorflow orgtfxguideserving allows us to select which version of a model or servable we want to use when we make inference requests each version will be exported to a different subdirectory under the given path examine your saved model we ll use the command line utility savedmodelcli to look at the metagraphdefshttps www tensorflow orgversionsr1 15apidocspythontfmetagraphdef the models and signaturedefshttps www tensorflow orgtfxservingsignaturedefs the methods you can call in our savedmodel see this discussion of the savedmodel clihttps github comtensorflowdocsblobmastersiteenr1guidesavedmodel mdclitoinspectandexecutesavedmodel in the tensorflow guide shell savedmodelcli show dir modelexportpath tagset serve signaturedef servingdefault that tells us a lot about our model for instance we can see that its inputs have a 4d shape 1 224 224 3 which means batchsize height width channels also note that this model requires a specific image shape 224 224 3 this means that we may need to reshape our images before sending them to the model we can also see that the model s outputs have a 1 1000 shape which are the logits for the 1000 classes of the imagenethttps www imagenet org dataset this information doesn t tell us everything like the fact that the pixel values needs to be in the 1 1 range but it s a great start serve your model with tensorflow serving install tfserving we re preparing to install tensorflow serving using aptitudehttps wiki debian orgaptitude since this colab runs in a debian environment we ll add the tensorflowmodelserver package to the list of packages that aptitude knows about note that we re running as root note this example is running tensorflow serving natively but you can also run it in a docker containerhttps www tensorflow orgtfxservingdocker which is one of the easiest ways to get started using tensorflow serving shell wget http storage googleapis comtensorflowservingaptpooltensorflowmodelserveruniversal2 8 0ttensorflowmodelserveruniversaltensorflowmodelserveruniversal2 8 0all deb dpkg i tensorflowmodelserveruniversal2 8 0all deb start running tensorflow serving this is where we start running tensorflow serving and load our model after it loads we can start making inference requests using rest there are some important parameters port the port that you ll use for grpc requests restapiport the port that you ll use for rest requests modelname you ll use this in the url of rest requests it can be anything modelbasepath this is the path to the directory where you ve saved your model check the tfserving api referencehttps github comtensorflowservingblobmastertensorflowservingmodelserversmain cc to get all the parameters available environment variable with the path to the model shell bash bg nohup tensorflowmodelserver port8500 restapiport8501 modelnamemodel modelbasepathmodeldir server log 21 shell we can check the logs to the server to help troubleshooting cat server log outputs warn getaddrinfo address family for nodename not supported evhttpserver cc 245 netlog entering the event loop shell now we can check if tensorflow is in the active services sudo lsof i p n grep listen outputs node 7 root 21u ipv6 19100 0t0 tcp 8080 listen kernelma 34 root 7u ipv4 18874 0t0 tcp 172 28 0 12 6000 listen colabfil 63 root 5u ipv4 17975 0t0 tcp 3453 listen colabfil 63 root 6u ipv6 17976 0t0 tcp 3453 listen jupytern 81 root 6u ipv4 18092 0t0 tcp 172 28 0 12 9000 listen python3 101 root 23u ipv4 18252 0t0 tcp 127 0 0 1 44915 listen python3 132 root 3u ipv4 20548 0t0 tcp 127 0 0 1 15264 listen python3 132 root 4u ipv4 20549 0t0 tcp 127 0 0 1 37977 listen python3 132 root 9u ipv4 20662 0t0 tcp 127 0 0 1 40689 listen tensorflo 1101 root 5u ipv4 35543 0t0 tcp 8500 listen tensorflo 1101 root 12u ipv4 35548 0t0 tcp 8501 listen make a request to your model in tensorflow serving now let s create the json object for an inference request and see how well our model classifies it rest api newest version of the servable we ll send a predict request as a post to our server s rest endpoint and pass it as an example we ll ask our server to give us the latest version of our servable by not specifying a particular version python restoutputs predictrestdata url printfrest output shape restoutputs shape printfpredicted class postprocessrestoutputs outputs rest output shape 1 1000 predicted class b banana grpc api grpchttps grpc io is based on the remote procedure call rpc model and is a technology for implementing rpc apis that uses http 2 0 as its underlying transport protocol grpc is usually preferred for lowlatency highly scalable and distributed systems if you wanna know more about the rest vs grpc tradeoffs checkout this articlehttps cloud google comblogproductsapimanagementunderstandinggrpcopenapiandrestandwhentousethem create a channel that will be connected to the grpc port of the container shell pip install q tensorflowservingapi python from tensorflowserving apis import predictpb2 predictionservicepb2grpc create a stub made for prediction this stub will be used to send the grpcrequest to the tf server stub predictionservicepb2grpc predictionservicestubchannel get the servinginput key python def predictgrpcdata inputname stub create a grpc request made for prediction request predictpb2 predictrequest set the name of the model for this use case it is model request modelspec name model set which signature is used to format the grpc query here the default one servingdefault request modelspec signaturename servingdefault set the input as the data tf maketensorproto turns a tensorflow tensor into a protobuf tensor request inputsinputname copyfromtf maketensorprotodata numpy tolist send the grpc request to the tf server result stub predictrequest return result grpcoutputs predictgrpcbatchedimg inputname stub grpcoutputs np arraygrpcoutputs outputs predictions floatval printfgrpc output shape grpcoutputs shape printfpredicted class postprocessgrpcoutputs outputs grpc output shape 1 1000 predicted class b banana custom signature note that for this model we always need to preprocess and postprocess all samples to get the desired output this can get quite tricky if are maintaining and serving several models developed by a large team and each one of them might require different processing logic tensorflow allows us to customize the model graph to embed all of that processing logic which makes model serving much easier there are different ways to achieve this but since we are going to server the models using tfserving we can customize the model graph straight into the serving signature we can just use the following code to export the same model that already contains the preprocessing and postprocessing logic as the default signature this allows this model to make predictions on raw data shell savedmodelcli show dir modelsigexportpath tagset serve signaturedef servingdefault note that this model has a different signature its input is still 4d but now with a 1 1 1 3 shape which means that it supports images with any height and width size its output also has a different shape it no longer outputs the 1000long logits we can test the model s prediction using a specific signature using this api below prediction using a particular version of the servable now let s specify a particular version of our servable note that when we saved the model with a custom signature we used a different folder the first model was saved in folder 1 version 1 and the one with a custom signature in folder 2 version 2 by default tfserving will serve all models that share the same base parent folder rest api python printfrest output shape restoutputs shape printfpredicted class restoutputs outputs rest output shape 1 predicted class banana grpc api python channel grpc insecurechannellocalhost 8500 stub predictionservicepb2grpc predictionservicestubchannel python grpcoutputs predictgrpcbatchedrawimg inputname stub grpcoutputs np arraygrpcoutputs outputs label stringval printfgrpc output shape grpcoutputs shape printfpredicted class grpcoutputs outputs grpc output shape 1 1 predicted class b banana additional resources colab notebook with the full working codehttps colab research google comdrive1nwuija4so1xzyu0ngq8txsgto295mu uspsharing train and serve a tensorflow model with tensorflow serving tensorflow bloghttps www tensorflow orgtfxtutorialsservingrestsimplemakearequesttoyourmodelintensorflowserving tensorflow serving playlist tensorflow youtube channelhttps www youtube complaylist listplqy2h8rroyvwhdpvqvohy7qcyf2s1uyk introduction once you build a machine learning model the next step is to serve it you may want to do that by exposing your model as an endpoint service there are many frameworks that you can use to do that but the tensorflow ecosystem has its own solution called tensorflow serving https www tensorflow org tfx guide serving from the tensorflow serving github page https github com tensorflow serving tensorflow serving is a flexible high performance serving system for machine learning models designed for production environments it deals with the inference aspect of machine learning taking models after training and managing their lifetimes providing clients with versioned access via a high performance reference counted lookup table tensorflow serving provides out of the box integration with tensorflow models but can be easily extended to serve other types of models and data to note a few features it can serve multiple models or multiple versions of the same model simultaneously it exposes both grpc as well as http inference endpoints it allows deployment of new model versions without changing any client code it supports canarying new versions and a b testing experimental models it adds minimal latency to inference time due to efficient low overhead implementation it features a scheduler that groups individual inference requests into batches for joint execution on gpu with configurable latency controls it supports many servables tensorflow models embeddings vocabularies feature transformations and even non tensorflow based machine learning models this guide creates a simple mobilenet https arxiv org abs 1704 04861 model using the keras applications api https keras io api applications and then serves it with tensorflow serving https www tensorflow org tfx guide serving the focus is on tensorflow serving rather than the modeling and training in tensorflow note you can find a colab notebook with the full working code at this link https colab research google com drive 1nwuija4so1xzyu0ngq8tx_ sgto295mu usp sharing dependencies model here we load a pre trained mobilenet https arxiv org abs 1704 04861 from the keras applications https keras io api applications this is the model that we are going to serve preprocessing most models don t work out of the box on raw data they usually require some kind of preprocessing step to adjust the data to the model requirements in the case of this mobilenet we can see from its api page https keras io api applications mobilenet that it requires three basic steps for its input images pixel values normalized to the 0 1 range pixel values scaled to the 1 1 range images with the shape of 224 224 3 meaning height width channels we can do all of that with the following function scale normalize and resizes images scale normalize resize a note regarding preprocessing and postprocessing using the keras applications api all models that are available at the keras applications https keras io api applications api also provide preprocess_input and decode_predictions functions those functions are respectively responsible for the preprocessing and postprocessing of each model and already contains all the logic necessary for those steps that is the recommended way to process inputs and outputs when using keras applications models for this guide we are not using them to present the advantages of custom signatures in a clearer way postprocessing in the same context most models output values that need extra processing to meet the user requirements for instance the user does not want to know the logits values for each class given an image what the user wants is to know from which class it belongs for our model this translates to the following transformations on top of the model outputs get the index of the class with the highest prediction get the name of the class from that index download human readable labels for imagenet skiping backgroung class convert the labels to the tensorflow data format convert from probs to labels index with highest prediction class name now let s download a banana picture and see how everything comes together save the model to load our trained model into tensorflow serving we first need to save it in savedmodel https www tensorflow org versions r1 15 api_docs python tf saved_model format this will create a protobuf file in a well defined directory hierarchy and will include a version number tensorflow serving https www tensorflow org tfx guide serving allows us to select which version of a model or servable we want to use when we make inference requests each version will be exported to a different sub directory under the given path examine your saved model we ll use the command line utility saved_model_cli to look at the metagraphdefs https www tensorflow org versions r1 15 api_docs python tf metagraphdef the models and signaturedefs https www tensorflow org tfx serving signature_defs the methods you can call in our savedmodel see this discussion of the savedmodel cli https github com tensorflow docs blob master site en r1 guide saved_model md cli to inspect and execute savedmodel in the tensorflow guide shell saved_model_cli show dir model_export_path tag_set serve signature_def serving_default that tells us a lot about our model for instance we can see that its inputs have a 4d shape 1 224 224 3 which means batch_size height width channels also note that this model requires a specific image shape 224 224 3 this means that we may need to reshape our images before sending them to the model we can also see that the model s outputs have a 1 1000 shape which are the logits for the 1000 classes of the imagenet https www image net org dataset this information doesn t tell us everything like the fact that the pixel values needs to be in the 1 1 range but it s a great start serve your model with tensorflow serving install tfserving we re preparing to install tensorflow serving using aptitude https wiki debian org aptitude since this colab runs in a debian environment we ll add the tensorflow model server package to the list of packages that aptitude knows about note that we re running as root note this example is running tensorflow serving natively but you can also run it in a docker container https www tensorflow org tfx serving docker which is one of the easiest ways to get started using tensorflow serving shell wget http storage googleapis com tensorflow serving apt pool tensorflow model server universal 2 8 0 t tensorflow model server universal tensorflow model server universal_2 8 0_all deb dpkg i tensorflow model server universal_2 8 0_all deb start running tensorflow serving this is where we start running tensorflow serving and load our model after it loads we can start making inference requests using rest there are some important parameters port the port that you ll use for grpc requests rest_api_port the port that you ll use for rest requests model_name you ll use this in the url of rest requests it can be anything model_base_path this is the path to the directory where you ve saved your model check the tfserving api reference https github com tensorflow serving blob master tensorflow_serving model_servers main cc to get all the parameters available environment variable with the path to the model shell bash bg nohup tensorflow_model_server port 8500 rest_api_port 8501 model_name model model_base_path model_dir server log 2 1 shell we can check the logs to the server to help troubleshooting cat server log outputs warn getaddrinfo address family for nodename not supported evhttp_server cc 245 net_log entering the event loop shell now we can check if tensorflow is in the active services sudo lsof i p n grep listen outputs node 7 root 21u ipv6 19100 0t0 tcp 8080 listen kernel_ma 34 root 7u ipv4 18874 0t0 tcp 172 28 0 12 6000 listen colab fil 63 root 5u ipv4 17975 0t0 tcp 3453 listen colab fil 63 root 6u ipv6 17976 0t0 tcp 3453 listen jupyter n 81 root 6u ipv4 18092 0t0 tcp 172 28 0 12 9000 listen python3 101 root 23u ipv4 18252 0t0 tcp 127 0 0 1 44915 listen python3 132 root 3u ipv4 20548 0t0 tcp 127 0 0 1 15264 listen python3 132 root 4u ipv4 20549 0t0 tcp 127 0 0 1 37977 listen python3 132 root 9u ipv4 20662 0t0 tcp 127 0 0 1 40689 listen tensorflo 1101 root 5u ipv4 35543 0t0 tcp 8500 listen tensorflo 1101 root 12u ipv4 35548 0t0 tcp 8501 listen make a request to your model in tensorflow serving now let s create the json object for an inference request and see how well our model classifies it rest api newest version of the servable we ll send a predict request as a post to our server s rest endpoint and pass it as an example we ll ask our server to give us the latest version of our servable by not specifying a particular version python rest_outputs predict_rest data url print f rest output shape rest_outputs shape print f predicted class postprocess rest_outputs outputs rest output shape 1 1000 predicted class b banana grpc api grpc https grpc io is based on the remote procedure call rpc model and is a technology for implementing rpc apis that uses http 2 0 as its underlying transport protocol grpc is usually preferred for low latency highly scalable and distributed systems if you wanna know more about the rest vs grpc tradeoffs checkout this article https cloud google com blog products api management understanding grpc openapi and rest and when to use them create a channel that will be connected to the grpc port of the container shell pip install q tensorflow_serving_api python from tensorflow_serving apis import predict_pb2 prediction_service_pb2_grpc create a stub made for prediction this stub will be used to send the grpcrequest to the tf server stub prediction_service_pb2_grpc predictionservicestub channel get the serving_input key python def predict_grpc data input_name stub create a grpc request made for prediction request predict_pb2 predictrequest set the name of the model for this use case it is model request model_spec name model set which signature is used to format the grpc query here the default one serving_default request model_spec signature_name serving_default set the input as the data tf make_tensor_proto turns a tensorflow tensor into a protobuf tensor request inputs input_name copyfrom tf make_tensor_proto data numpy tolist send the grpc request to the tf server result stub predict request return result grpc_outputs predict_grpc batched_img input_name stub grpc_outputs np array grpc_outputs outputs predictions float_val print f grpc output shape grpc_outputs shape print f predicted class postprocess grpc_outputs outputs grpc output shape 1 1000 predicted class b banana custom signature note that for this model we always need to preprocess and postprocess all samples to get the desired output this can get quite tricky if are maintaining and serving several models developed by a large team and each one of them might require different processing logic tensorflow allows us to customize the model graph to embed all of that processing logic which makes model serving much easier there are different ways to achieve this but since we are going to server the models using tfserving we can customize the model graph straight into the serving signature we can just use the following code to export the same model that already contains the preprocessing and postprocessing logic as the default signature this allows this model to make predictions on raw data shell saved_model_cli show dir model_sig_export_path tag_set serve signature_def serving_default note that this model has a different signature its input is still 4d but now with a 1 1 1 3 shape which means that it supports images with any height and width size its output also has a different shape it no longer outputs the 1000 long logits we can test the model s prediction using a specific signature using this api below prediction using a particular version of the servable now let s specify a particular version of our servable note that when we saved the model with a custom signature we used a different folder the first model was saved in folder 1 version 1 and the one with a custom signature in folder 2 version 2 by default tfserving will serve all models that share the same base parent folder rest api python print f rest output shape rest_outputs shape print f predicted class rest_outputs outputs rest output shape 1 predicted class banana grpc api python channel grpc insecure_channel localhost 8500 stub prediction_service_pb2_grpc predictionservicestub channel python grpc_outputs predict_grpc batched_raw_img input_name stub grpc_outputs np array grpc_outputs outputs label string_val print f grpc output shape grpc_outputs shape print f predicted class grpc_outputs outputs grpc output shape 1 1 predicted class b banana additional resources colab notebook with the full working code https colab research google com drive 1nwuija4so1xzyu0ngq8tx_ sgto295mu usp sharing train and serve a tensorflow model with tensorflow serving tensorflow blog https www tensorflow org tfx tutorials serving rest_simple make_a_request_to_your_model_in_tensorflow_serving tensorflow serving playlist tensorflow youtube channel https www youtube com playlist list plqy2h8rroyvwhdpvqvohy7 qcyf2s1uyk
import os

os.environ["KERAS_BACKEND"] = "tensorflow"

import json
import shutil
import requests
import numpy as np
import tensorflow as tf
import keras
import matplotlib.pyplot as plt



model = keras.applications.MobileNet()




def preprocess(image, mean=0.5, std=0.5, shape=(224, 224)):
    
    image = image / 255.0  
    image = (image - mean) / std  
    image = tf.image.resize(image, shape)  
    return image








imagenet_labels_url = "https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt"
response = requests.get(imagenet_labels_url)

labels = [x for x in response.text.split("\n") if x != ""][1:]

tf_labels = tf.constant(labels, dtype=tf.string)


def postprocess(prediction, labels=tf_labels):
    
    indices = tf.argmax(prediction, axis=-1)  
    label = tf.gather(params=labels, indices=indices)  
    return label




response = requests.get("https://i.imgur.com/j9xCCzn.jpeg", stream=True)

with open("banana.jpeg", "wb") as f:
    shutil.copyfileobj(response.raw, f)

sample_img = plt.imread("./banana.jpeg")
print(f"Original image shape: {sample_img.shape}")
print(f"Original image pixel range: ({sample_img.min()}, {sample_img.max()})")
plt.imshow(sample_img)
plt.show()

preprocess_img = preprocess(sample_img)
print(f"Preprocessed image shape: {preprocess_img.shape}")
print(
    f"Preprocessed image pixel range: ({preprocess_img.numpy().min()},",
    f"{preprocess_img.numpy().max()})",
)

batched_img = tf.expand_dims(preprocess_img, axis=0)
batched_img = tf.cast(batched_img, tf.float32)
print(f"Batched image shape: {batched_img.shape}")

model_outputs = model(batched_img)
print(f"Model output shape: {model_outputs.shape}")
print(f"Predicted class: {postprocess(model_outputs)}")



model_dir = "./model"
model_version = 1
model_export_path = f"{model_dir}/{model_version}"

tf.saved_model.save(
    model,
    export_dir=model_export_path,
)

print(f"SavedModel files: {os.listdir(model_export_path)}")











os.environ["MODEL_DIR"] = f"{model_dir}"



data = json.dumps(
    {
        "signature_name": "serving_default",
        "instances": batched_img.numpy().tolist(),
    }
)
url = "http://localhost:8501/v1/models/model:predict"


def predict_rest(json_data, url):
    json_response = requests.post(url, data=json_data)
    response = json.loads(json_response.text)
    rest_outputs = np.array(response["predictions"])
    return rest_outputs




import grpc


channel = grpc.insecure_channel("localhost:8500")




loaded_model = tf.saved_model.load(model_export_path)
input_name = list(
    loaded_model.signatures["serving_default"]
    .structured_input_signature[1]
    .keys()
)[0]







def export_model(model, labels):
    @tf.function(
        input_signature=[tf.TensorSpec([None, None, None, 3], tf.float32)]
    )
    def serving_fn(image):
        processed_img = preprocess(image)
        probs = model(processed_img)
        label = postprocess(probs)
        return {"label": label}

    return serving_fn


model_sig_version = 2
model_sig_export_path = f"{model_dir}/{model_sig_version}"

tf.saved_model.save(
    model,
    export_dir=model_sig_export_path,
    signatures={"serving_default": export_model(model, labels)},
)





batched_raw_img = tf.expand_dims(sample_img, axis=0)
batched_raw_img = tf.cast(batched_raw_img, tf.float32)

loaded_model = tf.saved_model.load(model_sig_export_path)
loaded_model.signatures["serving_default"](**{"image": batched_raw_img})



data = json.dumps(
    {
        "signature_name": "serving_default",
        "instances": batched_raw_img.numpy().tolist(),
    }
)
url_sig = "http://localhost:8501/v1/models/model/versions/2:predict"





input_name = list(
    loaded_model.signatures["serving_default"]
    .structured_input_signature[1]
    .keys()
)[0]