<jupyter_start><jupyter_text>Friendly MLOps with DRUM Considering the [10k Diabetes dataset](./data/readmissions_train.csv) and a number of pretrained models, we'll get our hands dirty by * Using DRUM for performance testing of models* Using DRUM for validation of models * Using DRUM to get a REST API endpoint* Show ease of swapping models out (different framewokrs - H2O GLM, DataRobot LGMB, Python Catboost, Python XGBoost* Instrument humility rules  Grab the `friendly-mlops` repo<jupyter_code>!git clone https://github.com/timsetsfire/friendly-mlops.git<jupyter_output>Cloning into 'friendly-mlops'...
remote: Enumerating objects: 56, done.[K
remote: Counting objects: 100% (56/56), done.[K
remote: Compressing objects: 100% (56/56), done.[K
remote: Total 56 (delta 24), reused 0 (delta 0), pack-reused 0[K
Unpacking objects: 100% (56/56), done.<jupyter_text>Install requirements<jupyter_code>!pip install -U -r /content/friendly-mlops/colab_requirements.txt -q<jupyter_output>[K     |████████████████████████████████| 20.1MB 1.4MB/s 
[K     |████████████████████████████████| 276kB 25.5MB/s 
[K     |████████████████████████████████| 296kB 44.2MB/s 
[K     |████████████████████████████████| 10.4MB 39.7MB/s 
[K     |████████████████████████████████| 25.9MB 116kB/s 
[K     |████████████████████████████████| 7.0MB 16.3MB/s 
[K     |████████████████████████████████| 66.0MB 137kB/s 
[K     |████████████████████████████████| 8.7MB 42.5MB/s 
[K     |████████████████████████████████| 148.9MB 80kB/s 
[K     |████████████████████████████████| 788kB 35.7MB/s 
[K     |████████████████████████████████| 276kB 37.6MB/s 
[K     |████████████████████████████████| 51kB 5.5MB/s 
[K     |████████████████████████████████| 17.7MB 208kB/s 
[K     |████████████████████████████████| 153kB 41.2MB/s 
[K     |████████████████████████████████| 204kB 37.2MB/s 
[K     |████████████████████████████████| 808kB 37.2MB/s 
[K     |████████████████████████████████| 61kB 6.6MB/s 
[...]<jupyter_text>Restart KERNEL!go to Runtime -> Restart RuntimeThis ensures we used the version so libraries we just installed  ModelsSeveral models have been been trained in advanced and are available in the model folder of this repo.  Each models is has been trained to predict the probability of readmission for a given patient.  Several frameworks have been used to train aforementioned models to highlight versatility of DRUM when it comes to ease of usage<jupyter_code>! ls -l /content/friendly-mlops/models<jupyter_output>total 20
drwxr-xr-x 2 root root 4096 Mar 28 18:18 catboost
drwxr-xr-x 2 root root 4096 Mar 28 18:18 datarobot
drwxr-xr-x 2 root root 4096 Mar 28 18:18 h2o
drwxr-xr-x 2 root root 4096 Mar 28 18:18 xgboost
drwxr-xr-x 2 root root 4096 Mar 28 18:18 xgboost-with-humility<jupyter_text>There are a lot of [frameworks](https://github.com/datarobot/datarobot-user-modelsbuilt-in-model-support)  and models which DRUM supports natively, but for those which DRUM doesn't support of these shelf, we'll just need to create some custom hooks so DRUM.  In this example, we'll highlight some very simple custom hooks, and will provide links to more complex examples.Those (Python and R) hooks include* `init`* `load_model`* `read_input_data`* `transform` aka preprocessing* `score`* `post_process`* `unstructured_predict` - scoring arbitrary data and returning arbitrary output.    ValidationYou can validate the model on a set of various checks. It is highly recommended to run these checks, as they are performed in DataRobot before the model can be deployed.List of checks:* null values imputation - each feature of the provided dataset is set to missing and fed to the model.This takes some time to run but will not be pursued.  __It is highly recommended to run these checks__<jupyter_code># %%sh
# drum validation --code-dir ./friendly-mlops/models/xgboost \
# --input ./friendly-mlops/data/readmissions_test.csv \
# --target-type binary \
# --positive-class-label True \
# --negative-class-label False<jupyter_output><empty_output><jupyter_text>Performance Testing<jupyter_code>%%sh
drum perf-test --code-dir ./friendly-mlops/models/xgboost \
--input ./friendly-mlops/data/readmissions_test.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \
--verbose<jupyter_output>Detected perf-test mode
DRUM performance test
Model:      /content/friendly-mlops/models/xgboost
Data:       /content/friendly-mlops/data/readmissions_test.csv
# Features: 48
Preparing test data...
Running drum using: [drum server --code-dir /content/friendly-mlops/models/xgboost --address localhost:58185 --logging-level warning --show-perf --target-type binary --positive-class-label True --negative-class-label False]



Running test case with timeout: 600
Running test case: 264 bytes - 1 samples, 100 iterations
Running test case with timeout: 600
Running test case: 0.1MB - 396 samples, 50 iterations
Running test case with timeout: 600
Running test case: 10MB - 39685 samples, 5 iterations
Running test case with timeout: 600
Running test case: 50MB - 198428 samples, 1 iterations
Test is done stopping drum server

  size      samples   iters    min      avg      max     used (MB)      total   
                                                                      physical  
              [...]<jupyter_text>Batch Scoring with DRUMAt this point our model has been written to disk and we want to start making predictions with it.  To do this, we'll leverage DRUM and it's ability to natively handle our scikit learn model, all we need to do is tell DRUM where it resides as well as the data we wish to score.  There are a lot of frameworks which DRUM supports nateively, but for those which DRUM doesn't support of these shelf, we'll just need to create some custom hooks so DRUM.  In this example, we'll highlight some very simple custom hooks, and will provide links to more complex examples.<jupyter_code>%%sh 
drum score --code-dir ./friendly-mlops/models/xgboost \
--input ./friendly-mlops/data/readmissions_test.csv \
--output ./friendly-mlops/data/predictions.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \
import pandas as pd
pd.read_csv("./friendly-mlops/data/predictions.csv").head()<jupyter_output><empty_output><jupyter_text>Serving via API with DRUMBatch scoring can be very useful, but the utility DRUM offers does not stop there.  We can also leverage DRUM to serve our model as a RESTful API endpoint.  The only thing that changes is the way we will structure the command - using the `server` mode instead of `score` model.  We'll also need to provide an address which is NOT in use.  When starting the server, we'll use `subprocess.Popen` so we may interact with the server in this notebook<jupyter_code>import subprocess
import requests
from io import BytesIO
import yaml
import time
import os
from pprint import pprint
run_inference_server = ["drum",
              "server",
              "--code-dir","/content/friendly-mlops/models/xgboost", 
              "--address", "0.0.0.0:6789", 
              "--show-perf",
              "--target-type", "binary",
              "--positive-class-label", "True",
              "--negative-class-label", "False",
              "--logging-level", "info",
              "--show-stacktrace",
              "--verbose",
              # "--production", 
              # "--max-workers", "5"
              ]
inference_server = subprocess.Popen(run_inference_server, stdout=subprocess.PIPE)
# !sudo service nginx status<jupyter_output><empty_output><jupyter_text>Ping the Server to make sure it is running<jupyter_code>## confirm the server is running
time.sleep(5) ## snoozing before pinging the server to give it time to actually start
print('check status')
requests.request("GET", "http://0.0.0.0:6789/").content<jupyter_output>check status<jupyter_text>Send data to server for inferenceThe request must provide our dataset as form data.  In order to do so, we'll create a simple python function to pass the data over appropriately.  We'll leverage the same function in our simple flask app a little later.<jupyter_code>def score(data, port = "6789"):
    b_buf = BytesIO()
    b_buf.write(data.to_csv(index=False).encode("utf-8"))
    b_buf.seek(0)
  
    url = "http://localhost:{}/predict/".format(port)
    files = [
        ('X', b_buf)
    ]
    response = requests.request("POST", url, files = files, timeout=None, verify=False)
    return response
# %%timeit
scoring_data = pd.read_csv("./friendly-mlops/data/readmissions_test.csv")
predictions = score(scoring_data).json() ## score entire dataset but only show first 5 records
pd.DataFrame(predictions["predictions"]).head()
requests.request("POST", "http://0.0.0.0:6789/shutdown").content
inference_server.terminate()
inference_server.stdout.readlines()
# %%sh
# nginx -s stop
# sudo service nginx status<jupyter_output>* nginx is not running<jupyter_text>Value PropOne may ask, what is the benefit to be had here?  Well, first of, there is not need for me to write an api to get the model up and running.  Second, DRUM allows me to abstract the framework away (provided I'm using one that is natively supported, or I can write enough python so that DRUM understands how to hook up to the model.  For example, I could hot swap models as I see fit (see exampels in `./friendly-mlops/models`)While we will run through several other frameworks with in `score` you can bet they are supported in `server` mode as well!  H2O GLM Mojo<jupyter_code>%%sh
drum score \
--code-dir ./friendly-mlops/models/h2o \
--input ./friendly-mlops/data/readmissions_test.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \<jupyter_output>FALSE      TRUE
0    0.713687  0.286313
1    0.791950  0.208050
2    0.451245  0.548755
3    0.632735  0.367265
4    0.757495  0.242505
..        ...       ...
495  0.591182  0.408818
496  0.657564  0.342436
497  0.699272  0.300728
498  0.324734  0.675266
499  0.302756  0.697244

[500 rows x 2 columns]<jupyter_text>DataRobot Light GBM<jupyter_code>%%sh
drum score \
--code-dir ./friendly-mlops/models/datarobot \
--input ./friendly-mlops/data/readmissions_test.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \<jupyter_output>True     False
0    0.335276  0.664724
1    0.293130  0.706870
2    0.493023  0.506977
3    0.353853  0.646147
4    0.212991  0.787009
..        ...       ...
495  0.547511  0.452489
496  0.346488  0.653512
497  0.550543  0.449457
498  0.520717  0.479283
499  0.732253  0.267747

[500 rows x 2 columns]<jupyter_text>Python Catboost<jupyter_code>%%sh
drum score \
--code-dir ./friendly-mlops/models/catboost \
--input ./friendly-mlops/data/readmissions_test.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \<jupyter_output>True     False
0    0.735228  0.264772
1    0.735228  0.264772
2    0.419437  0.580563
3    0.591693  0.408307
4    0.735228  0.264772
..        ...       ...
495  0.591693  0.408307
496  0.591693  0.408307
497  0.486772  0.513228
498  0.419437  0.580563
499  0.419437  0.580563

[500 rows x 2 columns]<jupyter_text>HumilityWhether or not your model can handle missing values for features and new levels for categorical data is a seperate issue from what to do when missing values and new levels occur within the data. Machine learning models are not infaliable.  They can learn biases and get things wrong.  We'll consider adding a layer of humility to our model as they reside in production by introducing actions to take in certain instances.  * Should you model make predictions as usual when an outlying input is observed?  * Your model can handle new levels of categoricals, but should you let it??  Should you throw provide a prediction but log the observation?* In the case of prediction, how should you handle uncertain predictions (classification) or unreasonably high or low predictions (regression)  Humility on 10K DiabetesTo introduce humility to our XGBoost model, a simple class was written to be involked in our custom.py script, specifically, we'll have a humility check that runs on input features within the `transform` hook, and a humility check that happens on the predictions within the `post_processing` hook.  For example, in our humility rules for 10k diabetes, we'll consider a rule for * outlying input for `time_in_hospital`* `prediction` override* known / unknown levels for `race````- feature: race  rule: categorical    known_values:    - Caucasian    - AfricanAmerican    - Hispanic    - Other    - Asian    - "?"  action:    key: do nothing    value: null - feature: time_in_hospital  rule: outlying_input  bounds:    lower_bound: 1    upper_bound: 100  action:    key: do nothing    value: null- prediction_column: "True"  bounds:    lower_bound: 0.48    upper_bound: 0.52  rule: uncertain_prediction  action:     key: override_prediction    value: 0.0```<jupyter_code>%%sh
drum score \
--code-dir ./friendly-mlops/models/xgboost-with-humility \
--input ./friendly-mlops/data/readmissions_test_humility.csv \
--target-type binary \
--positive-class-label True \
--negative-class-label False \<jupyter_output>True     False
0    0.609406  0.390594
1    0.716724  0.283276
2    0.647538  0.352462
3    0.595122  0.404878
4    0.816218  0.183782
..        ...       ...
495  0.586178  0.413822
496  0.458597  0.541403
497  0.466822  0.533178
498  0.478825  0.521175
499  0.349466  0.650534

[500 rows x 2 columns]<jupyter_text>Monitoring DeploymentsWhat follows will require a DataRobot account.  You can get a trial account at [https://www.datarobot.com/trial/](https://www.datarobot.com/trial/)Also, JDK 11 or 12 will be required.The main idea: we'll will start an agent service locally.  This agent will be monitoring a spooler.  The spooler could be something as simple as local file system, or a little more realistic like a message broker (pubsub, rabbitmq, sqs).  Once, this agent is spun up locally, we'll enable a few environment variables to let DRUM know that there is an agent present and that it needs to buffer data to defined spool.    Getting the monitoring agents Currently - have to go in through the [UI](https://app2.datarobot.com/account/developer-tools) to grab the agents<jupyter_code>import datarobot as dr
token = "token"
endpoint = "https://app2.datarobot.com"
## connect to DataRobot platform with python client. 
client = dr.Client(token, "{}/api/v2".format(endpoint))
# mlops_agents_tb = client.get("mlopsInstaller")
# with open("/content/odsc-ml-drum/mlops-agent.tar.gz", "wb") as f:
#     f.write(mlops_agents_tb.content)
# !tar -xf /content/mlops-agent.tar.gz -C ..
!tar -xf /content/datarobot-mlops-agent-6.2.4-399.tar.gz -C .<jupyter_output><empty_output><jupyter_text>Configuring the AgentWhen we'll configure the agent, we just need to define the DataRobot MLOPS location, our api token.  By default, the agent will expect the data to be spooled on the local file system.  Specifically, the default location will be `/tmp/ta` so we just need to make sure that location exists<jupyter_code>!mkdir -p /tmp/ta
agents_dir = "/content/datarobot-mlops-agent-6.2.4"
with open(r'{}/conf/mlops.agent.conf.yaml'.format(agents_dir)) as file:
    documents = yaml.load(file, Loader=yaml.FullLoader)
## configure the loaction of the mlops instance with which we'll communcate
documents['mlopsUrl'] = endpoint
# Set your API token
documents['apiToken'] = token
## write the configuration back to disk
with open('../{}/conf/mlops.agent.conf.yaml'.format(agents_dir), "w") as f:
    yaml.dump(documents, f)<jupyter_output><empty_output><jupyter_text>Start the Agent ServiceChecking to make sure we can start up the agents service.  This will require a JDK - tested with 11 and 12<jupyter_code>## run agents service
subprocess.call("{}/bin/start-agent.sh".format(agents_dir))
## check status
check = subprocess.Popen(["../{}/bin/status-agent.sh".format(agents_dir)], stdout=subprocess.PIPE)
print(check.stdout.readlines())
check.terminate()
## check log to see that the agent connected to DR MLOps
check = subprocess.Popen(["cat", "../{}/logs/mlops.agent.log".format(agents_dir)], stdout=subprocess.PIPE)
for line in check.stdout.readlines():
    print(line)
check.terminate()<jupyter_output>b'2020-11-16 19:01:02,449 INFO  com.datarobot.mlops.agent.config.channels.YamlBuilder        [] - Found spooler of type FILESYSTEM\n'
b'2020-11-16 19:01:02,452 INFO  com.datarobot.mlops.agent.config.channels.YamlBuilder        [] - Setting directory = /tmp/ta\n'
b'2020-11-16 19:01:02,453 INFO  com.datarobot.mlops.agent.config.channels.YamlBuilder        [] - Setting CHANNEL_NAME = filesystem\n'
b"2020-11-16 19:01:03,699 INFO  com.datarobot.mlops.common.client.MLOpsClient                [] - DataRobot Server API Version found: '2.22'\n"
b"2020-11-16 19:01:04,258 INFO  com.datarobot.mlops.common.client.MLOpsClient                [] - DataRobot Server API Version found: '2.22'\n"
b"2020-11-16 19:01:04,259 INFO  com.datarobot.mlops.agent.Agent                              [] - DataRobot server at 'https://app2.datarobot.com' is reachable.\n"
b'2020-11-16 19:01:04,259 INFO  com.datarobot.mlops.agent.Agent                              [] - DataRobot Monitoring Agent will process 100 records [...]<jupyter_text>DataRobot MLOps - Deploying External Model  To communication with DataRobot MLOps, with need to MLOps python client installed which came in the downloaded tarball<jupyter_code>!pip install /content/datarobot-mlops-*/lib/datarobot_mlops-*.whl -q
from datarobot.mlops.mlops import MLOps
from datarobot.mlops.common.enums import OutputType
from datarobot.mlops.connected.client import MLOpsClient
from datarobot.mlops.common.exception import DRConnectedException
from datarobot.mlops.constants import Constants
DEPLOYMENT_NAME="10 Diabetes Readmissions Prediction"
TRAINING_DATA = '/content/friendly-mlops/data/readmissions_train.csv'
model_info = {
        "name": "<NAME>",
        "modelDescription": {
            "description": "prediction price of home"
        },
        "target": {
            "type": "Binary",
            "name": "readmitted",
            "positiveClassLabel": "True",
            "negativeClassLabel": "False"
        }
}
# Create connected client
mlops_client = MLOpsClient(endpoint, token)

# Add training_data to model configuration
print("Uploading training data - {}. This may take some time...".format(TRAINING_DATA))
dataset_id = mlops_client.upload_dataset(TRAINING_DATA)
print("Training dataset uploaded. Catalog ID {}.".format(dataset_id))
model_info["datasets"] = {"trainingDataCatalogId": dataset_id}

# Create the model package
print('Create model package')
model_pkg_id = mlops_client.create_model_package(model_info)
model_pkg = mlops_client.get_model_package(model_pkg_id)
model_id = model_pkg["modelId"]

# Deploy the model package
print('Deploy model package')
deployment_id = mlops_client.deploy_model_package(model_pkg["id"],
                                                            DEPLOYMENT_NAME)

# Enable data drift tracking
print('Enable feature drift')
enable_feature_drift = TRAINING_DATA is not None
mlops_client.update_deployment_settings(deployment_id, target_drift=True,
                                                  feature_drift=enable_feature_drift)
_ = mlops_client.get_deployment_settings(deployment_id)

print("\nDone.")
print("DEPLOYMENT_ID=%s, MODEL_ID=%s" % (deployment_id, model_id))

DEPLOYMENT_ID = deployment_id
MODEL_ID = model_id
from IPython.core.display import display, HTML
link = "{}/deployments/{}/overview".format(endpoint,deployment_id)
# display(HTML("""<a href="{link}">{link}</a>""".format( link=link )))
print(link)<jupyter_output>https://app2.datarobot.com/deployments/5fb2ccba6a2cd70255b0fa2c/overview<jupyter_text>Adding Monitoring with MLOps Monitoring Agents  Monitoring With DRUMThere are a few addition parameters we should set for the command line utility, or we may just create environment variables, and allow the drum utility to pick up the details from there.  ```  --monitor             Monitor predictions using DataRobot MLOps. True or                        False. (env: MONITOR).Monitoring can not be used in                        unstructured mode.  --deployment-id DEPLOYMENT_ID                        Deployment id to use for monitoring model predictions                        (env: DEPLOYMENT_ID)  --model-id MODEL_ID   MLOps model id to use for monitoring predictions (env:                        MODEL_ID)  --monitor-settings MONITOR_SETTINGS                        MLOps setting to use for connecting with the MLOps                        Agent (env: MONITOR_SETTINGS)```For today, we'll set environment variables to add monitoring.<jupyter_code>os.environ["MONITOR"] = "True"
os.environ["DEPLOYMENT_ID"] = deployment_id
os.environ["MODEL_ID"] = model_id
os.environ["MONITOR_SETTINGS"] = "spooler_type=filesystem;directory=/tmp/ta;max_files=5;file_max_size=1045876000"
run_inference_server = ["drum",
              "server",
              "--code-dir","friendly-mlops/models/xgboost-with-humility", 
              "--address", "0.0.0.0:43210", 
              "--show-perf",
              "--target-type", "binary",
              "--positive-class-label", "True",
              "--negative-class-label", "False",
              "--logging-level", "info",
              "--show-stacktrace",
#               "--verbose"
              ]
inference_server_with_monitoring = subprocess.Popen(run_inference_server, stdout=subprocess.PIPE)
predictions = score(
    pd.read_csv("/content/friendly-mlops/data/readmissions_test.csv").head(100),
    "43210")
pd.DataFrame(predictions.json()).head()
requests.post("http://localhost:43210/shutdown/").content
subprocess.call("../{}/bin/stop-agent.sh".format(agents_dir))
## check that agent is stopped 
check = subprocess.Popen(["../{}/bin/status-agent.sh".format(agents_dir)], stdout=subprocess.PIPE)
print(check.stdout.readlines())
check.terminate()
deployment = dr.Deployment.get(deployment_id)
deployment.get_service_stats()
service_stats = deployment.get_service_stats()
service_stats.metrics<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1267840259330962, 'fraction_numerical': 0.05128089685291072, 'mean_word_length': 3.625572678050812, 'pattern_counts': {'":': 9, '<': 64, '<?xml version=': 0, '>': 65, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '649020', 'license': 'Apache-2.0', 'n_tokens_mistral': 8200, 'n_tokens_neox': 7035, 'n_words': 2145, 'path': 'Colab - Friendly MLOps.ipynb', 'repo_name': 'datarobot-community/friendly-mlops', 'size': '22227'}