<jupyter_start><jupyter_text>Canary Models with Seldon and IstioThis folder provides resources to illustrate how to do a canary	roll out of one	MNIST model to another using the canary	pattern where a small amount	of traffic is sent to the new model to validate	it before sending all traffic to the new model.We utilize two MNIST digit classification models.  * Version 1 of the model using scikit-learn * Version 2 of the model using Tensorflow. After deploying Istio and Seldon to a kubernetes cluster we will: * Deploy version 1 scikit-learn based model using seldon-core * Create an istio routing rule to direct all traffic to this version * Create a canary deployment with both version 1 and version 2 (the Tensorflow model) * Update the istio routing to send 10% of the traffic to version 2 * Update the istio routing to send 100% of the traffic to version 2    SetupThe steps below will install istio and seldon onto a GKE Cluster. If you wish to use your own setup then you need to ensure * You allow istio egress to the internet as the load test downloads MNIST images * Ensure you give your user cluster-admin privledges * Install seldon into a namespace seldon To follow the steps below you will need:   * a Google project running a K8S cluster  * gcloud and kubectl installed with kubectl authorized for your cluster  * [istio download](https://github.com/istio/istio/releases/)  Setup Environment Variables<jupyter_code># Set this to your istio download folder, e.g. 
# %env ISTIO_HOME=~/istio-1.0.0
%env ISTIO_HOME=/home/clive/work/istio/istio-1.0.6
# Set this to your GKE Zone for your cluster, e.g.
# %env ZONE=europe-west3-b
%env ZONE=europe-west3-b
# Set this to the GCP Project within which your cluster is running
%env PROJECT=seldon-demos
# Set this to the name of your kubernetes cluster
%env CLUSTER_NAME=istio-seldon<jupyter_output>env: CLUSTER_NAME=istio-seldon<jupyter_text>Determine CIDR ranges<jupyter_code>!gcloud container clusters describe ${CLUSTER_NAME} --zone ${ZONE} --project ${PROJECT} | grep -e clusterIpv4Cidr -e servicesIpv4Cidr<jupyter_output>clusterIpv4Cidr: 10.56.0.0/14
servicesIpv4Cidr: 10.59.240.0/20<jupyter_text>Install Helm<jupyter_code>!kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format="value(config.account)")
!kubectl create -f ${ISTIO_HOME}/install/kubernetes/helm/helm-service-account.yaml
!helm init --service-account tiller<jupyter_output>$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!<jupyter_text>Install Istio ** Replace the CIDR 10.48.0.0/14\,10.51.240.0/20 values with those you got above **For more details see [istio docs on egress](https://istio.io/docs/tasks/traffic-management/egress/calling-external-services-directly)<jupyter_code>!helm install ${ISTIO_HOME}/install/kubernetes/helm/istio --name istio --namespace istio-system \
    --set global.proxy.includeIPRanges="10.56.0.0/14\,10.59.240.0/20" --set grafana.enabled=true<jupyter_output>NAME:   istio
LAST DEPLOYED: Fri Mar 15 15:05:11 2019
NAMESPACE: istio-system
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/ClusterRoleBinding
NAME                                                    AGE
istio-galley-admin-role-binding-istio-system            58s
istio-ingressgateway-istio-system                       58s
istio-egressgateway-istio-system                        58s
istio-grafana-post-install-role-binding-istio-system    58s
istio-mixer-admin-role-binding-istio-system             58s
istio-pilot-istio-system                                58s
prometheus-istio-system                                 58s
istio-citadel-istio-system                              58s
istio-security-post-install-role-binding-istio-system   58s
istio-sidecar-injector-admin-role-binding-istio-system  58s

==> v1alpha2/prometheus
NAME     AGE
handler  56s

==> v1/Pod(related)
NAME                                     READY  STATUS   RESTARTS  AGE
istio-galley-6c68c5dbcf-kcsxc            1/1    Running  0 [...]<jupyter_text>To view the istio Grafana dashboard:```kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000```http://localhost:3000  Install Seldon<jupyter_code>!kubectl create namespace seldon
!helm install ../../../helm-charts/seldon-core-crd --name seldon-core-crd
!helm install ../../../helm-charts/seldon-core --name seldon-core --namespace seldon \
    --set ambassador.enabled=true
!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \
    --set grafana_prom_admin_password=password \
    --set persistence.enabled=false \
    --namespace seldon<jupyter_output>NAME:   seldon-core-analytics
LAST DEPLOYED: Fri Mar 15 15:06:51 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                      READY  STATUS             RESTARTS  AGE
grafana-prom-import-dashboards-nnd85      0/1    ContainerCreating  0         1s
alertmanager-deployment-5bff7cbb4d-5rr66  0/1    ContainerCreating  0         1s
grafana-prom-deployment-5dbc55b564-nxtxl  0/1    ContainerCreating  0         1s
prometheus-node-exporter-dzh66            0/1    ContainerCreating  0         1s
prometheus-node-exporter-n657d            0/1    Pending            0         1s
prometheus-node-exporter-wpwbf            0/1    ContainerCreating  0         1s
prometheus-deployment-78f855c4b-zpzrb     0/1    ContainerCreating  0         1s

==> v1/Secret
NAME                 TYPE    DATA  AGE
grafana-prom-secret  Opaque  1     1s

==> v1/ServiceAccount
NAME        SECRETS  AGE
prometheus  1        1s

==> v1beta1/ClusterRole
NAME        AGE
prometheu[...]<jupyter_text>To send requests to Ambassador ingress in another terminal run:    ```kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8002:8080``` To view the Seldon Grafana dashboard in another terminal run:```kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3001:3000```Use user:admin and the password passed in above in the helm install.http://localhost:3001/dashboard/db/prediction-analytics?refresh=5s&orgId=1<jupyter_code>!kubectl label namespace seldon istio-injection=enabled
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
%matplotlib inline
import utils
from visualizer import get_graph
mnist = utils.download_mnist()<jupyter_output>WARNING:tensorflow:From /home/clive/work/seldon-core/fork-seldon-core/examples/istio/canary_update/utils.py:65: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /home/clive/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From /home/clive/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to imp[...]<jupyter_text>Launch Version 1 ModelA sklearn MNIST model.<jupyter_code>!pygmentize mnist_v1.json
get_graph("mnist_v1.json")
!kubectl apply -f mnist_v1.json<jupyter_output>seldondeployment.machinelearning.seldon.io/mnist-classifier created<jupyter_text>** Wait until new pods are running **<jupyter_code>!kubectl rollout status deploy/mnist-deployment-sk-mnist-predictor-78b83e6
utils.predict_rest_mnist(mnist,"mnist-classifier")<jupyter_output><empty_output><jupyter_text>Start a Load Test<jupyter_code>!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust
!helm install seldon-core-loadtesting --name loadtest  \
    --namespace seldon \
    --repo https://storage.googleapis.com/seldon-charts \
    --set locust.script=mnist_rest_locust.py \
    --set locust.host=http://mnist-deployment-mnist-classifier:8000 \
    --set oauth.enabled=false \
    --set oauth.key=oauth-key \
    --set oauth.secret=oauth-secret \
    --set locust.hatchRate=1 \
    --set locust.clients=1 \
    --set loadtest.sendFeedback=1 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=1 \
    --set data.size=784<jupyter_output>NAME:   loadtest
LAST DEPLOYED: Fri Mar 15 15:40:06 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                   READY  STATUS    RESTARTS  AGE
locust-slave-1-4kptp   0/2    Init:0/1  0         0s
locust-master-1-qd9zg  0/2    Init:0/1  0         0s

==> v1/ReplicationController
NAME             DESIRED  CURRENT  READY  AGE
locust-slave-1   1        1        0      0s
locust-master-1  1        1        0      0s

==> v1/Service
NAME             TYPE      CLUSTER-IP    EXTERNAL-IP  PORT(S)                                       AGE
locust-master-1  NodePort  10.59.244.63  <none>       5557:31879/TCP,5558:32461/TCP,8089:31585/TCP  0s<jupyter_text>Setup Version 1 Istio RoutingJust route to version 1 of our model<jupyter_code>!pygmentize istio_canary_v1.yaml
!istioctl create -f istio_canary_v1.yaml<jupyter_output>Updated config virtual-service/seldon/mnist-deployment to revision 9948
Updated config destination-rule/seldon/mnist-deployment to revision 9949<jupyter_text>Launch Version 2 of ModelVersion 2 is a Tensorflow Deep Learning Model for MNIST. Hopefully, better...<jupyter_code>!pygmentize mnist_v2.json
get_graph("mnist_v2.json")
!kubectl apply -f mnist_v2.json<jupyter_output>seldondeployment.machinelearning.seldon.io/mnist-classifier configured<jupyter_text>** Wait until new pods are running **<jupyter_code>!kubectl rollout status deploy/mnist-deployment-tf-mnist-predictor-0b33413
utils.predict_rest_mnist(mnist,"mnist-classifier")<jupyter_output><empty_output><jupyter_text>Setup Canary RoutingLet's be cautious and route 30% of traffic to version 2<jupyter_code>!pygmentize istio_canary_v2.yaml
!istioctl replace -f istio_canary_v2.yaml<jupyter_output>Updated config virtual-service/seldon/mnist-deployment to revision 11056
Updated config destination-rule/seldon/mnist-deployment to revision 10741<jupyter_text>Viewing the Seldon Grafana dashboard we see both models getting traffic.![dashboard](dashboard.png)  Setup Routing to Version 2After testing we are happy and want to switch 100% of traffic to version 2 our Tensorflow model.<jupyter_code>!pygmentize istio_canary_v3.yaml
!istioctl replace -f istio_canary_v3.yaml<jupyter_output>Updated config virtual-service/seldon/mnist-deployment to revision 12704
Updated config destination-rule/seldon/mnist-deployment to revision 10741<jupyter_text>The dashboard now show traffic 100% to our new model.![dashboard2](dashboard2.png)  Switch to final modelFinally, lets save resources and remove version 1 of our model<jupyter_code>!pygmentize mnist_v3.json
get_graph("mnist_v3.json")
!kubectl apply -f mnist_v3.json<jupyter_output>seldondeployment.machinelearning.seldon.io/mnist-classifier configured
{'fraction_non_alphanumeric': 0.09225686208065062, 'fraction_numerical': 0.03524229074889868, 'mean_word_length': 3.7696969696969695, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 59, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 7, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '361857', 'license': 'Apache-2.0', 'n_tokens_mistral': 4179, 'n_tokens_neox': 3850, 'n_words': 1007, 'path': 'examples/istio/canary_update/canary.ipynb', 'repo_name': 'dtrawins/seldon-core', 'size': '11804'}