<jupyter_start><jupyter_text>Getting Started with BentoML[BentoML](http://bentoml.ai) is an open-source framework for machine learning **model serving**, aiming to **bridge the gap between Data Science and DevOps**.Data Scientists can easily package their models trained with any ML framework using BentoMl and reproduce the model for serving in production. BentoML helps with managing packaged models in the BentoML format, and allows DevOps to deploy them as online API serving endpoints or offline batch inference jobs, on any cloud platform.This getting started guide demonstrates how to use BentoML to serve a sklearn modeld via a REST API server, and then containerize the model server for production deployment.![Impression](https://www.google-analytics.com/collect?v=1&tid=UA-112879361-3&cid=555&t=event&ec=guides&ea=bentoml-quick-start-guide&dt=bentoml-quick-start-guide)BentoML requires python 3.6 or above, install dependencies via `pip`:<jupyter_code># Install PyPI packages required in this guide, including BentoML
!pip install -q bentoml
!pip install -q 'scikit-learn>=0.23.2' 'pandas>=1.1.1'<jupyter_output><empty_output><jupyter_text>Before started, let's discuss how BentoML's project structure would look like. For most use-cases, users can follow this minimal scaffoldfor deploying with BentoML to avoid any potential errors (example project structure can be found under [guides/quick-start](https://github.com/bentoml/BentoML/tree/master/guides/quick-start)):    bento_deploy/    ├── bento_packer.py         responsible for packing BentoService    ├── bento_service.py        BentoService definition    ├── model.py                DL Model definitions    ├── train.py                training scripts    └── requirements.txtLet's prepare a trained model for serving with BentoML. Train a classifier model on the [Iris data set](https://en.wikipedia.org/wiki/Iris_flower_data_set):<jupyter_code>from sklearn import svm
from sklearn import datasets

# Load training data
iris = datasets.load_iris()
X, y = iris.data, iris.target

# Model Training
clf = svm.SVC(gamma='scale')
clf.fit(X, y)<jupyter_output><empty_output><jupyter_text>Create a Prediction Service with BentoMLModel serving with BentoML comes after a model is trained. The first step is creating aprediction service class, which defines the models required and the inference APIs whichcontains the serving logic. Here is a minimal prediction service created for servingthe iris classifier model trained above:<jupyter_code>%%writefile bento_service.py
import pandas as pd

from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.frameworks.sklearn import SklearnModelArtifact

@env(infer_pip_packages=True)
@artifacts([SklearnModelArtifact('model')])
class IrisClassifier(BentoService):
    """
    A minimum prediction service exposing a Scikit-learn model
    """

    @api(input=DataframeInput(), batch=True)
    def predict(self, df: pd.DataFrame):
        """
        An inference API named `predict` with Dataframe input adapter, which codifies
        how HTTP requests or CSV files are converted to a pandas Dataframe object as the
        inference API function input
        """
        return self.artifacts.model.predict(df)<jupyter_output>Overwriting iris_classifier.py<jupyter_text>This code defines a prediction service that packages a scikit-learn model and providesan inference API that expects a `pandas.Dataframe` object as its input. BentoML also supports other API input data types including `JsonInput`, `ImageInput`, `FileInput` and [more](https://docs.bentoml.org/en/latest/api/adapters.html).In BentoML, **all inference APIs are suppose to accept a list of inputs and return a list of results**. In the case of `DataframeInput`, each row of the dataframe is mappingto one prediction request received from the client. BentoML will convert HTTP JSON requests into :code:`pandas.DataFrame` object before passing it to the user-defined inference API function. This design allows BentoML to group API requests into small batches while serving onlinetraffic. Comparing to a regular flask or FastAPI based model server, this can increasesthe overall throughput of the API server by 10-100x depending on the workload.The following code packages the trained model with the prediction service class`IrisClassifier` defined above, and then saves the IrisClassifier instance to disk in the BentoML format for distribution and deployment:<jupyter_code># import the IrisClassifier class defined above
from bento_service import IrisClassifier

# Create a iris classifier service instance
iris_classifier_service = IrisClassifier()

# Pack the newly trained model artifact
iris_classifier_service.pack('model', clf)
# Prepare input data for testing the prediction service
import pandas as pd
test_input_df = pd.DataFrame(X).sample(n=5)
test_input_df.to_csv("./test_input.csv", index=False)
test_input_df
# Test the service's inference API python interface
iris_classifier_service.predict(test_input_df)
# Start a dev model server to test out everything
iris_classifier_service.start_dev_server()
import requests
response = requests.post(
    "http://127.0.0.1:5000/predict",
    json=test_input_df.values.tolist()
)
print(response.text)
# Stop the dev model server
iris_classifier_service.stop_dev_server()
# Save the prediction service to disk for deployment
saved_path = iris_classifier_service.save()<jupyter_output>[2021-03-19 02:36:06,764] INFO - BentoService bundle 'IrisClassifier:20210319023551_84AAF6' saved to: /Users/chaoyu/bentoml/repository/IrisClassifier/20210319023551_84AAF6<jupyter_text>BentoML stores all packaged model files under the`~/bentoml/{service_name}/{service_version}` directory by default.The BentoML file format contains all the code, files, and configs required to deploy the model for serving.  REST API Model ServingTo start a REST API model server with the `IrisClassifier` saved above, use the `bentoml serve` command:<jupyter_code>!bentoml serve IrisClassifier:latest<jupyter_output>[2021-03-19 02:37:09,964] INFO - Getting latest version IrisClassifier:20210319023551_84AAF6
[2021-03-19 02:37:09,965] INFO - Starting BentoML API server in development mode..
 * Serving Flask app "IrisClassifier" (lazy loading)
 * Environment: production
[31m   WARNING: This is a development server. Do not use it in a production deployment.[0m
[2m   Use a production WSGI server instead.[0m
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET / HTTP/1.1[0m" 200 -
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET /static_content/main.css HTTP/1.1[0m" 200 -
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET /static_content/readme.css HTTP/1.1[0m" 200 -
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET /static_content/swagger-ui.css HTTP/1.1[0m" 200 -
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET /static_content/marked.min.js HTTP/1.1[0m" 200 -
127.0.0.1 - - [19/Mar/2021 02:37:19] "[37mGET /static_content/swagger-ui[...]<jupyter_text>If you are running this notebook from Google Colab, you can start the dev server with `--run-with-ngrok` option, to gain acccess to the API endpoint via a public endpoint managed by [ngrok](https://ngrok.com/):<jupyter_code>!bentoml serve IrisClassifier:latest --run-with-ngrok<jupyter_output><empty_output><jupyter_text>The `IrisClassifier` model is now served at `localhost:5000`. Use `curl` command to senda prediction request:```bashcurl -i \--header "Content-Type: application/json" \--request POST \--data '[[5.1, 3.5, 1.4, 0.2]]' \localhost:5000/predict```Or with `python` and [request library](https://requests.readthedocs.io/):```pythonimport requestsresponse = requests.post("http://127.0.0.1:5000/predict", json=[[5.1, 3.5, 1.4, 0.2]])print(response.text)```Note that BentoML API server automatically converts the Dataframe JSON format into a`pandas.DataFrame` object before sending it to the user-defined inference API function.The BentoML API server also provides a simple web UI dashboard.Go to http://localhost:5000 in the browser and use the Web UI to sendprediction request:![BentoML API Server Web UI Screenshot](https://raw.githubusercontent.com/bentoml/BentoML/master/guides/quick-start/bento-api-server-web-ui.png)  Containerize model server with DockerOne common way of distributing this model API server for production deployment, is viaDocker containers. And BentoML provides a convenient way to do that.Note that `docker` is __not available in Google Colab__. You will need to download and run this notebook locally to try out this containerization with docker feature.If you already have docker configured, simply run the follow command to product a docker container serving the `IrisClassifier` prediction service created above:<jupyter_code>!bentoml containerize IrisClassifier:latest -t iris-classifier:v1<jupyter_output>[2021-03-19 02:37:37,423] INFO - Getting latest version IrisClassifier:20210319023551_84AAF6
[39mFound Bento: /Users/chaoyu/bentoml/repository/IrisClassifier/20210319023551_84AAF6[0m
Containerizing IrisClassifier:20210319023551_84AAF6 with local YataiService and docker daemon from local environment-[32mBuild container image: iris-classifier:v1[0m
<jupyter_text>Start a container with the docker image built in the previous step:<jupyter_code>!docker run -p 5000:5000 iris-classifier:v1 --workers=2<jupyter_output>[2021-03-19 09:37:42,276] INFO - Starting BentoML API server in production mode..
[2021-03-19 09:37:42 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2021-03-19 09:37:42 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2021-03-19 09:37:42 +0000] [1] [INFO] Using worker: sync
[2021-03-19 09:37:42 +0000] [12] [INFO] Booting worker with pid: 12
[2021-03-19 09:37:42 +0000] [13] [INFO] Booting worker with pid: 13
[2021-03-19 09:37:42,700] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.8, but current environment version is 3.7.6.
[2021-03-19 09:37:42,771] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.8, but current environment version is 3.7.6.
[2021-03-19 09:38:35,545] INFO - {'service_name': 'IrisClassifier', 'service_version': '20210319023551_84AAF6', 'api': 'predict', 'task': {'data': '[[5.5, 2.3, 4.0, 1.3], [5.2, 4.1, 1.5, 0.1], [4.4, 2.9, 1.4, 0.2][...]<jupyter_text>This made it possible to deploy BentoML bundled ML models with platforms such as[Kubeflow](https://www.kubeflow.org/docs/components/serving/bentoml/),[Knative](https://knative.dev/community/samples/serving/machinelearning-python-bentoml/),[Kubernetes](https://docs.bentoml.org/en/latest/deployment/kubernetes.html), whichprovides advanced model deployment features such as auto-scaling, A/B testing,scale-to-zero, canary rollout and multi-armed bandit. Load saved BentoService`bentoml.load` is the API for loading a BentoML packaged model in python:<jupyter_code>import bentoml
import pandas as pd

bento_svc = bentoml.load(saved_path)

# Test loaded bentoml service:
bento_svc.predict(test_input_df)<jupyter_output>[2021-03-19 02:38:54,032] WARNING - Module `iris_classifier` already loaded, using existing imported module.
[2021-03-19 02:38:54,071] WARNING - pip package requirement pandas already exist
[2021-03-19 02:38:54,072] WARNING - pip package requirement scikit-learn already exist<jupyter_text>The BentoML format is pip-installable and can be directly distributed as aPyPI package for using in python applications:<jupyter_code>!pip install -q {saved_path}
# The BentoService class name will become packaged name
import IrisClassifier

installed_svc = IrisClassifier.load()
installed_svc.predict(test_input_df)<jupyter_output><empty_output><jupyter_text>This also allow users to upload their BentoService to pypi.org as public python packageor to their organization's private PyPi index to share with other developers.`cd {saved_path} & python setup.py sdist upload`*You will have to configure ".pypirc" file before uploading to pypi index.    You can find more information about distributing python package at:    https://docs.python.org/3.7/distributing/index.htmldistributing-index* Launch inference job from CLI BentoML cli supports loading and running a packaged model from CLI. With the `DataframeInput` adapter, the CLI command supports reading input Dataframe data from CLI argument or local `csv` or `json` files:<jupyter_code>!bentoml run IrisClassifier:latest predict --input '{test_input_df.to_json()}' --quiet
!bentoml run IrisClassifier:latest predict \
    --input-file "./test_input.csv" --format "csv" --quiet
# run inference with the docker image built above
!docker run -v $(PWD):/tmp iris-classifier:v1 \
        bentoml run /bento predict --input-file "/tmp/test_input.csv" --format "csv" --quiet<jupyter_output>[1, 0, 0, 0, 2]
{'fraction_non_alphanumeric': 0.0971932266295523, 'fraction_numerical': 0.05598082424804763, 'mean_word_length': 5.2908560311284045, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 40, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 10, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '3898', 'license': 'Apache-2.0', 'n_tokens_mistral': 4507, 'n_tokens_neox': 3962, 'n_words': 1343, 'path': 'guides/quick-start/bentoml-quick-start-guide.ipynb', 'repo_name': 'andy51002000/BentoML', 'size': '12933'}