<jupyter_start><jupyter_text>Pachyderm-Seldon Integration: Version Controlled Models > By Dr. <NAME>, CEO of [Winder Research](https://WinderResearch.com)[Pachyderm](https://pachyderm.io/) is a data science platform that combines Data Lineage with End-to-End Pipelines. [Seldon-Core](https://www.seldon.io/tech/products/core/) is an open-source platform for rapidly deploying machine learning models.This notebook specialises in showing you how you can automate the provisioning of a Seldon deployment using Pachyderm pipelines when new training data enters a Pachyderm repository. Additionally, this example uses Pachyderm's S3 gateway feature to provide convenient access to version controlled model parameters. HighlightsIn ML, provenance describes the ability to pinpoint all of the factors that contributed to a deployed model. People often talk about the provenance of the data, but what about the provenance of a model? Not only does model provenance provide the ability to do post-mortems on bad models, it is also an integral to GitOps, which provides repeatability, and resiliency.Similary, automation is the art of encoding a process to produce a deployed model. Without a process you can't have provenance, since several people will peform the task in several different ways. If you can automate the process then you can remove the human-induced variance entirelly and improve efficiency to boot. Example SituationImagine you work for an organization that uses the UK's retail services index (RSI), similar to the consumer prices index in the US, to help predict demand for your products. The RSI is often quoted as a proxy for consumer demand, so you could base your stock levels upon a future RSI prediction.But the RSI is a dynamic index. It is influenced by external events, like COVID-19, for example. Because of this you want to make sure that your model is up to date. One potential solution is scheduled retraining, but this is a poor proxy because it is not reactive. Scheduling is a poor-mans proxy because you end up retraining too much, which adds risk, or too little, where important changes to the model parameters are delayed.The solution then, if you pardon the buzz-word, is to become data-driven. When new data is available, it should kick-off a new training run. That should result in a model which is then automatically deployed. And ideally, all of this should be tracked and version controlled for provenance.I achieve this with two Pachyderm pipelines (on for training, one for deployment) and a simple custom Seldon deployment to serve the model. From an outsiders perspective, this is still a pure Seldon solution. But internally you will have full data-driven automation and provenance. This is depicted in the image below.![overview image](./images/pachyderm-cd4ml.png) PrerequisitesThis is a follow-on example from the simpler Pachyderm demo. I recommend that you review that first before following this one. * A kubernetes cluster with kubectl configured * [A seldon-core cluster installed and configured](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html) * Pachyderm's `pachctl` binary installed following the [official documentation](https://docs.pachyderm.com/latest/getting_started/local_installation/install-pachctl) * The ability to build docker images and push them to your clusterPachyderm is controlled and deployed via the `pachctl`. This document assumes that you have version `1.11.8` installed, although it should work with any `1.x.x` version.<jupyter_code>!pachctl version --client-only
!kubectl get po -A<jupyter_output>1.11.8
NAMESPACE            NAME                                          READY   STATUS    RESTARTS   AGE
kube-system          coredns-f9fd979d6-5ddkn                       1/1     Running   0          7m53s
kube-system          coredns-f9fd979d6-lbz46                       1/1     Running   0          7m53s
kube-system          etcd-kind-control-plane                       1/1     Running   0          8m1s
kube-system          kindnet-pfxnk                                 1/1     Running   0          7m53s
kube-system          kube-apiserver-kind-control-plane             1/1     Running   0          8m1s
kube-system          kube-controller-manager-kind-control-plane    1/1     Running   0          8m1s
kube-system          kube-proxy-s5z84                              1/1     Running   0          7m53s
kube-system          kube-scheduler-kind-control-plane             1/1     Running   0          8m1s
kube-system          seldon-spartakus-volunteer-5b57b95596-7hjb5   1/1     Runnin[...]<jupyter_text>1. Install PachydermFirst install a Pachyderm cluster. This uses the standard `local` install method which creates a cluster using your current kubernetes context.The second step is to give all Pachyderm pielines permission to deploy the `SeldonDeployment` to the `seldon` namespace. If you alter the deployment you will need to check that you provide the correct permissions.<jupyter_code>!kubectl create ns pachyderm
!pachctl deploy local --no-expose-docker-socket --no-dashboard --namespace pachyderm
!kubectl rollout status deployment -n pachyderm pachd
%%writefile rbac.yaml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: seldon
  name: allow-all
rules:
# To write the Pachyderm S3 gateway secret
- apiGroups: ["", ]
  resources: ["secrets"]
  verbs: ["*"]
# To create the Seldon deployment 
- apiGroups: ["machinelearning.seldon.io", ]
  resources: ["seldondeployments"]
  verbs: ["*"]
# So Helm can do an upgrade
- apiGroups: ["", ]
  resources: ["serviceaccounts"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pachyderm-deployer
  namespace: seldon
subjects:
- kind: ServiceAccount
  name: pachyderm-worker
  namespace: pachyderm
roleRef:
  kind: Role
  name: allow-all
  apiGroup: rbac.authorization.k8s.io
!kubectl apply -f rbac.yaml<jupyter_output>role.rbac.authorization.k8s.io/allow-all configured
rolebinding.rbac.authorization.k8s.io/pachyderm-deployer unchanged<jupyter_text>2. Add Initial Training Data and Build the Necessary Containers> Note: If you are using a non-standard Pachyderm namespace, you will need to run `pachctl port-forward` in a separate terminalBelow I create a training data repository and populate it with some initial RSI data. This data comes from the [UK's Office of National Statistics](https://www.ons.gov.uk/businessindustryandtrade/retailindustry/timeseries/j5c4/drsi). I have altered the original data by adding headers, removing quotes for floats, and organized into pre, mid, and post corona datasets for demonstration purposes.You'll have to wait a moment to see the data. I want to setup the automation so you can see it in action and first we need to create a data repo.<jupyter_code>!pachctl create repo rsi_data
!cd data && pachctl put file rsi_data@master:/rsi.csv -f rsi_before.csv
!pachctl list repo
!pachctl list file rsi_data@master<jupyter_output>NAME     TYPE SIZE     
/rsi.csv file 1.173KiB<jupyter_text>There are three containers used in this demo, one to train, one to deploy and the custom seldon deployment. You'll need to make sure that these containers are present in this cluster. I provide a script to upload to a local `kind` cluster but if you're using something else, you'll have to upload those containers yourself.<jupyter_code>!make kind_load<jupyter_output>make -C RSIServing build
docker build . -t seldonio/pachyderm-rsi-server:0.1.0
Sending build context to Docker daemon  7.168kB
Step 1/6 : FROM python:3.8-slim
 ---> be5d294735c6
Step 2/6 : RUN pip install pandas==1.1.4 statsmodels==0.12.1 seldon-core==1.5.0 joblib==1.0.0 minio==7.0.0
 ---> Running in 06fb3520d480
Collecting joblib==1.0.0
  Downloading joblib-1.0.0-py3-none-any.whl (302 kB)
Collecting minio==7.0.0
  Downloading minio-7.0.0-py3-none-any.whl (72 kB)
Collecting pandas==1.1.4
  Downloading pandas-1.1.4-cp38-cp38-manylinux1_x86_64.whl (9.3 MB)
Collecting seldon-core==1.5.0
  Downloading seldon_core-1.5.0-py3-none-any.whl (127 kB)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.8/site-packages (from seldon-core==1.5.0) (51.1.2)
Collecting statsmodels==0.12.1
  Downloading statsmodels-0.12.1-cp38-cp38-manylinux1_x86_64.whl (9.5 MB)
Collecting Flask<2.0.0
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting click>=5.1
  Downloading click[...]<jupyter_text>3. Create a Training PipelineThe training pipeline accepts data from a Pachyderm repository, trains a regression forecasting model using the `statsmodels` library, and produces a `joblib` file that holds the model parameters.The pipeline uses a custom container and python file to train the model (which you uploaded in the previous step). I'm not going to show it because it's really straightforward and builds a predictive model based upon an exponentially weighted moving average.<jupyter_code>%%writefile train.json
{
  "pipeline": {
    "name": "rsi_training"
  },
  "description": "A pipeline that trains a statsmodels model based on the provided input data.",
  "transform": {
    "image": "seldonio/pachyderm-rsi-trainer:0.1.0",
    "cmd": [
      "python3",
      "/code/RSITrain.py",
      "/pfs/rsi_data/rsi.csv",
      "/pfs/out/model.joblib"
    ]
  },
  "parallelism_spec": {
    "constant": "1"
  },
  "input": {
    "pfs": {
      "repo": "rsi_data",
      "glob": "/"
    }
  }
}
!pachctl create pipeline -f train.json<jupyter_output><empty_output><jupyter_text>4. Create a Custom Seldon DeploymentThis is where things get interesting. If you are using an sklearn or TF based model, you could use one of the out of the box Seldon servers. But `statsmodels` doesn't have one, and more to the point there is a key piece of functionality missing from the built in containers.Pachyderm exposes the commit hash of the objects in the repository via the `version` field of the S3 API. This isn't supported by KFServing/Seldon yet, so I created a custom container that does just that. This means that the code is slightly longer because of the inclusion of the minio code/library, but I think you should still be able to understand it.In summary, you need to pass the commit hash into the container, then use that to [call the S3 API](https://docs.min.io/docs/python-client-api-reference.htmlfget_object), setting the `version` parameter. After that you need to implement the standard [Seldon API](https://docs.seldon.io/projects/seldon-core/en/latest/python/python_component.html), which basically consists of a `predict` function, and make sure it returns the right type (a numpy array).Check out the accompanying video if you want a walkthrough of the code.Finally, this is all wrapped in a helm chart which consists of a `SeldonDeployment` and a `Secret` to store the S3 credentials. The trick here is that I use helm to template the S3 URI and the commit hash of the model that was trained by the pipeline.Again, check out the video for a walkthrough. 5. Create a Deployment PipelineAt this point you could kick off any old CD pipeline to deploy, but Pachyderm is perfectly capable, and you gain from the tight integration. For example, you could argue that the pipeline definition is static, and can reside in traditional CD systems, but the deployment is dynamic and data driven, so it makes more sense to keep it close to the data.To do this I create a custom deployer container which contains my custom helm chart. Typically you wouldn't need to do this, because your chart is likely to be hosted somewhere, so you can use the standard `alpine/helm` container. But for this demo the chart is local, so I bake it.Also there is a tiny `deploy.sh` script to `helm install` if it doesn't exist, else `helm upgrade`.The key section here is the parameters passed into the deploy script. Here I am hardcoding the master branch from the `rsi_training` repository, and the model name. But the commit hash is a special environmental variable provided to us by Pachyderm, specifying the new commit hash.  ImportantThis is what provides the provenance. At any point in the future, from anywhere in the world, you can deploy an old model using `deploy.sh in_some_namespace from_some_repo and_some_commit`. Beautifully simple and repeatable.Once you run the code below, the `rsi_deploy` pipeline will see that there is a new trained model, then run helm install to deploy the Seldon Deployment. (Lots of deploys!)<jupyter_code>%%writefile deploy.json
{
  "pipeline": {
    "name": "rsi_deploy"
  },
  "description": "A pipeline deploys a seldon core model using the most recent model parameters.",
  "transform": {
    "image": "seldonio/pachyderm-rsi-deployer:0.1.0",
    "cmd": ["/bin/ash"],
    "stdin": [
      "/deploy.sh seldon s3://master.rsi_training/model.joblib ${rsi_training_COMMIT}"
    ]
  },
  "parallelism_spec": {
    "constant": "1"
  },
  "input": {
    "pfs": {
      "repo": "rsi_training",
      "glob": "/"
    }
  }
}
!pachctl create pipeline -f deploy.json
!pachctl list pipeline
!pachctl list job<jupyter_output>NAME         VERSION INPUT          CREATED        STATE / LAST JOB  DESCRIPTION                                                                    
rsi_deploy   1       rsi_training:/ 17 seconds ago [32mrunning[0m / [32msuccess[0m A pipeline deploys a seldon core model using the most recent model parameters. 
rsi_training 1       rsi_data:/     19 seconds ago [32mrunning[0m / [32msuccess[0m A pipeline that trains a statsmodels model based on the provided input data.   
ID                               PIPELINE     STARTED        DURATION  RESTART PROGRESS  DL       UL      STATE   
61e0dd166c1c41499cb064d4ade222a4 rsi_deploy   9 seconds ago  2 seconds 0       1 + 0 / 1 15.9KiB  0B      [32msuccess[0m 
0e689e1838ec4b499eb57c10bad4c391 rsi_training 11 seconds ago 2 seconds 0       1 + 0 / 1 1.173KiB 15.9KiB [32msuccess[0m<jupyter_text>6. Call the Initial Model to Predict the Future (pre-COVID)By now you should have a fully operational Seldon Deployment, driven by your initial data commit. But it's worth checking. Often there are resource limit issues or silly things like that. If you have any issues, fix them first.<jupyter_code>!kubectl rollout status deployment -n seldon rsi-predictor-default-0-rsi-predictor<jupyter_output>Waiting for deployment "rsi-predictor-default-0-rsi-predictor" rollout to finish: 0 of 1 updated replicas are available...
deployment "rsi-predictor-default-0-rsi-predictor" successfully rolled out<jupyter_text>Now it's time to take a look at the data. In this notebook I want to actually show you the data, so I'm going to parse it into a pandas array. Below I create two helper methods to do this.Make sure you've port-forwarded the ingress controller service, or replace localhost with something routable from your machine. For exmaple:```shkubectl port-forward $(kubectl get pods -n seldon-system -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon-system 8003:8080```<jupyter_code>from urllib import request, parse
import json
from pprint import pprint

def get_predictions(num_periods: int = 6):
    data = json.dumps({"data":{"ndarray": num_periods}}).encode()
    req =  request.Request("http://localhost:8003/seldon/seldon/rsi-predictor/api/v1.0/predictions", data=data)
    req.add_header('Content-Type', 'application/json')
    resp = request.urlopen(req)
    print(resp.status)
    resp_data = json.loads(resp.read())
    pprint(resp_data)
    return resp_data["data"]["ndarray"]

def get_metadata() -> None:
    req =  request.Request("http://localhost:8003/seldon/seldon/rsi-predictor/api/v1.0/metadata")
    req.add_header('Content-Type', 'application/json')
    resp = request.urlopen(req)
    print(resp.status)
    resp_data = json.loads(resp.read())
    pprint(resp_data)
predictions = get_predictions(6)
get_metadata()<jupyter_output>200
{'data': {'names': ['t:0', 't:1'],
          'ndarray': [['Sat, 01 Feb 2020 00:00:00 GMT', 104.4494780932981],
                      ['Sun, 01 Mar 2020 00:00:00 GMT', 104.69971373078019],
                      ['Wed, 01 Apr 2020 00:00:00 GMT', 104.9499493682623],
                      ['Fri, 01 May 2020 00:00:00 GMT', 105.20018500574439],
                      ['Mon, 01 Jun 2020 00:00:00 GMT', 105.4504206432265],
                      ['Wed, 01 Jul 2020 00:00:00 GMT', 105.70065628070859]]},
 'meta': {'requestPath': {'rsi-predictor': 'seldonio/pachyderm-rsi-server:0.1.0'}}}
200
{'graphinputs': [{'messagetype': 'ndarray',
                  'schema': {'names': ['periods'], 'shape': [1]}}],
 'graphoutputs': [{'messagetype': 'ndarray', 'schema': {'shape': [2]}}],
 'models': {'rsi-predictor': {'custom': {'MODEL_COMMIT_HASH': 'ee7ff0e85615451e8bf4f03ba399ff04',
                                         'MODEL_URI': 's3://master.rsi_training/model.joblib'},
                              'in[...]<jupyter_text>Inspect that data. See how the prediction has returned 6 months into the future. Also note the metadata, which contains the commit hash of the model that generated this prediction. Cool huh! You can tracing predictions back to their model/data.This is implemented as custom metadata. It's not a standard feature of Seldon.Now let me load and plot all the data to see how our notional business is going to do in the future...<jupyter_code>import pandas as pd

rsi = pd.read_csv("./data/rsi_before.csv", index_col="Date", parse_dates=True) # Load data
rsi_prediction = pd.DataFrame(predictions).set_index(0) # Move timestamp to index
rsi_prediction.index = pd.to_datetime(rsi_prediction.index) # Parse timestamp
ax = rsi.plot()
rsi_prediction.plot(ax=ax)<jupyter_output>/Users/phil/.pyenv/versions/3.8.1/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py:1266: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.
  warnings.warn(<jupyter_text>Not too bad ehy? Minor blip in the last couple of months in 2019, but sentiment continues to be positive...> BUT WAIT...> > WHAT IS... GOING ON...> > WHY...> > UH OH...<jupyter_code>rsi = pd.read_csv("./data/rsi_mid.csv", index_col="Date", parse_dates=True) # Load more recent data
rsi_prediction = pd.DataFrame(predictions).set_index(0)
rsi_prediction.index = pd.to_datetime(rsi_prediction.index)
ax = rsi.plot()
rsi_prediction.plot(ax=ax)<jupyter_output>/Users/phil/.pyenv/versions/3.8.1/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py:1266: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.
  warnings.warn(<jupyter_text>7. Panic-Driven Manual Deploys Always FailYes, Coronavirus happened. It's the latest in a string of many so-called "1-in-100 year events" and affected nearly every aspect of life.Thankfully, in our wisdom, we have a pipeline. All we need to do is to stream the data into our training data repository, then watch the deployment bloom. Don't Panic!<jupyter_code>!cd data && pachctl put file rsi_data@master:/rsi.csv -f rsi_mid.csv --overwrite
!pachctl list pipeline
!pachctl list job
!kubectl rollout status deployment -n seldon rsi-predictor-default-0-rsi-predictor
get_metadata()
predictions = get_predictions(6)<jupyter_output>200
{'graphinputs': [{'messagetype': 'ndarray',
                  'schema': {'names': ['periods'], 'shape': [1]}}],
 'graphoutputs': [{'messagetype': 'ndarray', 'schema': {'shape': [2]}}],
 'models': {'rsi-predictor': {'custom': {'MODEL_COMMIT_HASH': 'f6d17cefdfe94c1f96b1bab58fdef9db',
                                         'MODEL_URI': 's3://master.rsi_training/model.joblib'},
                              'inputs': [{'messagetype': 'ndarray',
                                          'schema': {'names': ['periods'],
                                                     'shape': [1]}}],
                              'name': 'RSIModel',
                              'outputs': [{'messagetype': 'ndarray',
                                           'schema': {'shape': [2]}}],
                              'platform': 'seldon',
                              'versions': ['s3://master.rsi_training/model.joblib@f6d17cefdfe94c1f96b1bab58fdef9db']}},
 'name': 'default'}
200
{'data': {'names':[...]<jupyter_text>First, notice the commit hash in the metadata. See that it has changed. Your old model still exists in the repo, but it's not at the tip any more.Secondly, let's look at the predictions now...<jupyter_code>rsi = pd.read_csv("./data/rsi_mid.csv", index_col="Date", parse_dates=True)
rsi_prediction = pd.DataFrame(predictions).set_index(0)
rsi_prediction.index = pd.to_datetime(rsi_prediction.index)
ax = rsi.plot()
rsi_prediction.plot(ax=ax)<jupyter_output>/Users/phil/.pyenv/versions/3.8.1/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py:1266: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.
  warnings.warn(<jupyter_text>Oh dear! Negative RSI's that's not right! This highlights that our model isn't fit for purpose any more, but that's a conversation for another day.After some time, we get more data...<jupyter_code>!cd data && pachctl put file rsi_data@master:/rsi.csv -f rsi_after.csv --overwrite
!pachctl list pipeline
!pachctl list job
!kubectl rollout status deployment -n seldon rsi-predictor-default-0-rsi-predictor
get_metadata()
predictions = get_predictions(6)
rsi = pd.read_csv("./data/rsi_after.csv", index_col="Date", parse_dates=True)
rsi_prediction = pd.DataFrame(predictions).set_index(0)
rsi_prediction.index = pd.to_datetime(rsi_prediction.index)
ax = rsi.plot()
rsi_prediction.plot(ax=ax)<jupyter_output>/Users/phil/.pyenv/versions/3.8.1/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py:1266: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.
  warnings.warn(
{'fraction_non_alphanumeric': 0.09368002559064113, 'fraction_numerical': 0.030388886350134807, 'mean_word_length': 3.676068376068376, 'pattern_counts': {'":': 27, '<': 43, '<?xml version=': 0, '>': 56, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '129537', 'license': 'Apache-2.0', 'n_tokens_mistral': 7085, 'n_tokens_neox': 6550, 'n_words': 2358, 'path': 'examples/pachyderm-cd4ml/index.ipynb', 'repo_name': 'taka-sawada/seldon-core', 'size': '21888'}