<jupyter_start><jupyter_text>Dynamic Inference (Serving Models in Production via REST)Inference is the term used to describe the process of using a pre-trained model to make predictions for unseen data.Dynamic Inference is the term used to describe making predictions on demand, using a server. This notebook is a walk through for how to serve a machine learning model using a low latency prediction servering system called **[clipper.ai](http://clipper.ai/)**. clipper can be hosted on your favorite cloud provider or on-premise.Overview + Model training+ Clipper cluster creation+ App creation & model deployment+ Model query (single row, multiple rows) via Python requests & curl+ Model versioning update + Model versioning rollback+ Model replicationReferences:+ [clipper.ai documentation](http://clipper.ai/)+ [clipper @github](https://github.com/ucbrise/clipper)  Model TrainingBefore we can serve a model, we must first train it. Model training is an iterative process often persisting artifacts to disk. After model has been trained, we can load the model back into memory (RAM) when serving.<jupyter_code>import logging, xgboost as xgb, numpy as np
from sklearn.metrics import mean_absolute_error
import joblib
import pandas as pd
from datetime import datetime
import pickle
import time
import matplotlib.pyplot as plt
plt.show(block=True)

training_examples = pd.read_pickle("../data/processed/airlines_training_examples.pkl")
f1=open("../data/processed/airlines_training_targets.pkl",'rb')
training_targets = pickle.load(f1) 
f1.close()
test_examples = pd.read_pickle("../data/processed/airlines_test_examples.pkl")

def get_train_points():
     return training_examples.values.tolist()

def get_test_points(start_row_index,end_row_index):
    return test_examples.iloc[start_row_index:end_row_index].values.tolist()

def get_test_point(row_index):
     return test_examples.iloc[row_index].tolist()
    
# Create a training matrix.
dtrain = xgb.DMatrix(get_train_points(), label=training_targets)
# We then create parameters, watchlist, and specify the number of rounds
# This is code that we use to build our XGBoost Model, and your code may differ.
param = {'max_depth': 2, 'eta': 1, 'silent': 1, 'objective': 'binary:logistic'}
watchlist = [(dtrain, 'train')]
num_round = 2
bst = xgb.train(param, dtrain, num_round, watchlist)
def predict(xs):
    result = bst.predict(xgb.DMatrix(xs))
    return result 
# make predictions
predictions = predict(test_examples.values)
print("Predict instances in test set using custom defined scoring function...")
predictions<jupyter_output>Predict instances in test set using custom defined scoring function...<jupyter_text>Clipper Cluster Creation<jupyter_code>from clipper_admin import ClipperConnection, DockerContainerManager
clipper_conn = ClipperConnection(DockerContainerManager())
print("Start Clipper...")
clipper_conn.start_clipper()<jupyter_output>18-08-22:00:19:15 INFO     [docker_container_manager.py:151] [default-cluster] Starting managed Redis instance in Docker<jupyter_text>App Creation & Model Deployment<jupyter_code>from clipper_admin.deployers import python as python_deployer
print("Register Clipper application...")
clipper_conn.register_application('xgboost-airlines', 'doubles', 'default_pred', 100000)

# We specify which packages to install in the pkgs_to_install arg.
# For example, if we wanted to install xgboost and psycopg2, we would use
# pkgs_to_install = ['xgboost', 'psycopg2']
print("Deploy predict function closure using Clipper...")
python_deployer.deploy_python_closure(clipper_conn, name='xgboost-model', version=1,
    input_type="doubles", func=predict, pkgs_to_install=['xgboost'])
print("Link Clipper connection to model application...")
clipper_conn.link_model_to_app('xgboost-airlines', 'xgboost-model')<jupyter_output>18-08-22:00:19:34 INFO     [clipper_admin.py:277] [default-cluster] Model xgboost-model is now linked to application xgboost-airlines<jupyter_text>Query Model via Python requests module<jupyter_code>import requests, json
# Get Address
addr = clipper_conn.get_query_addr()
print("Model predict for a single instance via Python requests POST request & parse response...")

# Post Query
response = requests.post(
     "http://%s/%s/predict" % (addr, 'xgboost-airlines'),
     headers={"Content-type": "application/json"},
     data=json.dumps({
         'input': get_test_point(0)
     }))
result = response.json() 
result<jupyter_output>Model predict for a single instance via Python requests POST request & parse response...<jupyter_text>Query Model (single row)<jupyter_code>import requests, json, numpy as np
print("Model predict for a single instance via Python requests POST request...")
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/xgboost-airlines/predict", headers=headers, data=json.dumps({"input": get_test_point(0)})).json()<jupyter_output>Model predict for a single instance via Python requests POST request...<jupyter_text>Query Model (multiple rows)<jupyter_code>import requests, json, numpy as np
print("Model predict for a batch of instances via Python requests POST request...")
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/xgboost-airlines/predict", headers=headers, 
              data=json.dumps({"input_batch": get_test_points(0,2)})).json()<jupyter_output>Model predict for a batch of instances via Python requests POST request...<jupyter_text>Query Model via curl<jupyter_code>get_test_point(0)
print("Model predict for a single instance via curl...")
!curl -X POST --header "Content-Type:application/json" -d '{"input": [1987.0, 10.0, 1.0, 4.0, 1250.0, 1340.0, 1509.0, 50.0, 226.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}' 127.0.0.1:1337/xgboost-airlines/predict<jupyter_output>Model predict for a single instance via curl...
{"query_id":4,"output":0.9041093,"default":false}<jupyter_text>Model UpdateSuppose you found a new set of hyper-parameters which increase the predictive power of your model. You found that it yields better prediction results than the first model you deployed. We decide to deploy version 2 of our XGBoost model.Let's first retrain v2...<jupyter_code># Create a training matrix.
dtrain = xgb.DMatrix(get_train_points(), label=training_targets)
# We then create parameters, watchlist, and specify the number of rounds
# This is code that we use to build our XGBoost Model, and your code may differ.
param = {'max_depth': 3, 'eta': 1, 'silent': 1, 'objective': 'binary:logistic'}
watchlist = [(dtrain, 'train')]
num_round = 2
bst_v2 = xgb.train(param, dtrain, num_round, watchlist)
def predict(xs):
    result = bst_v2.predict(xgb.DMatrix(xs))
    return result 
# make predictions
predictions = predict(test_examples.values)
print("Predict instances in test set using custom defined scoring function...")
predictions
# Deploy the 'predict' function as a model (to a new container)... observe the name change
python_deployer.deploy_python_closure(clipper_conn, name='xgboostv2-model', version=2,
    input_type="doubles", func=predict, pkgs_to_install=['xgboost'])<jupyter_output>18-08-22:00:26:38 INFO     [deployer_utils.py:41] Saving function to /var/folders/kv/w56d6z9j4c79zvw8c8jsn6hw0000gn/T/tmplfrzglliclipper
18-08-22:00:26:38 INFO     [deployer_utils.py:51] Serialized and supplied predict function
18-08-22:00:26:38 INFO     [python.py:192] Python closure saved
18-08-22:00:26:38 INFO     [python.py:206] Using Python 3.6 base image
18-08-22:00:26:38 INFO     [clipper_admin.py:467] [default-cluster] Building model Docker image with model data from /var/folders/kv/w56d6z9j4c79zvw8c8jsn6hw0000gn/T/tmplfrzglliclipper
18-08-22:00:26:39 INFO     [clipper_admin.py:472] [default-cluster] Step 1/3 : FROM clipper/python36-closure-container:develop
18-08-22:00:26:39 INFO     [clipper_admin.py:472] [default-cluster]  ---> 0fac6e6e8242
18-08-22:00:26:39 INFO     [clipper_admin.py:472] [default-cluster] Step 2/3 : RUN apt-get -y install build-essential && pip install xgboost
18-08-22:00:26:39 INFO     [clipper_admin.py:472] [default-cluster]  ---> Using cache
18-08-22:00[...]<jupyter_text>Query Model (single row)<jupyter_code>import requests, json, numpy as np
print("Model predict for a single instance via Python requests POST request...")
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/xgboost-airlines/predict", headers=headers, 
              data=json.dumps({"input": get_test_point(0)})).json()<jupyter_output>Model predict for a single instance via Python requests POST request...<jupyter_text>Model RollbackSuppose you find out that model v2 is overfitting, here's how you can roll it back to v1...<jupyter_code># rollback
clipper_conn.set_model_version(name='xgboostv2-model', version='2')
import requests, json, numpy as np
print("Model predict for a single instance via Python requests POST request...")
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/xgboost-airlines/predict", headers=headers, data=json.dumps({"input": get_test_point(0)})).json()<jupyter_output>Model predict for a single instance via Python requests POST request...<jupyter_text>Model Replication Machine learning models can be computationally expensive. A single instance of the model hosting machine may not meet the throughput requirements of a serving workload. In order to increase the prediction throughput you can add additional replicas...<jupyter_code>clipper_conn.set_num_replicas('xgboost-model', num_replicas=10, version='1')
!docker ps<jupyter_output>CONTAINER ID        IMAGE                                 COMMAND                  CREATED              STATUS                        PORTS                                            NAMES
2e14fab54dde        default-cluster-xgboost-model:1       "/container/containe…"   57 seconds ago       Up 56 seconds (healthy)                                                        xgboost-model_1-50045
cedd0a671508        default-cluster-xgboost-model:1       "/container/containe…"   58 seconds ago       Up 57 seconds (healthy)                                                        xgboost-model_1-58540
325a91f5e787        default-cluster-xgboost-model:1       "/container/containe…"   59 seconds ago       Up 58 seconds (healthy)                                                        xgboost-model_1-55105
529cf633fc3d        default-cluster-xgboost-model:1       "/container/containe…"   About a minute ago   Up 59 seconds (healthy)                                                        xgboost-m[...]<jupyter_text>Clipper Troubleshooting Guide[This](http://clipper.ai/tutorials/troubleshooting/) is a helpful guide for troubleshooting common issues with clipper.ai.<jupyter_code># clipper_conn.inspect_instance()
# clipper_conn.get_clipper_logs()<jupyter_output><empty_output><jupyter_text>Shut down clipper<jupyter_code>print("Shutting down Clipper connection.")
clipper_conn.stop_all()
# stop all containers:
!docker rm $(docker ps -a -q)
!docker ps<jupyter_output>CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES<jupyter_text>Some other useful system commands<jupyter_code># stop all containers:
# docker kill $(docker ps -q)

# remove all containers
# !docker rm $(docker ps -a -q)

# remove all docker images
# docker rmi $(docker images -q)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13619286161552913, 'fraction_numerical': 0.07365372573575454, 'mean_word_length': 3.8416066691928763, 'pattern_counts': {'":': 13, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 41, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '220891', 'license': 'MIT', 'n_tokens_mistral': 5109, 'n_tokens_neox': 4512, 'n_words': 1274, 'path': 'notebooks/Dynamic_Model_Inference.ipynb', 'repo_name': 'ShreeshaJay/ML-to-Production-Toolkit', 'size': '12776'}