<jupyter_start><jupyter_text>Stateful Model Feedback Metrics ServerIn this example we will add statistical performance metrics capabilities by levering the Seldon metrics server.Dependencies* Seldon Core installed* KNative eventing v0.11.0* KNative serving v0.11.1 (optional)<jupyter_code>!mkdir -p config<jupyter_output><empty_output><jupyter_text>Create a simple modelWe create a multiclass classification model - iris classifier.The iris classifier takes an input array, and returns the prediction of the 4 classes.The prediction can be done as numeric or as a probability array.<jupyter_code>%%bash
kubectl apply -f - << END
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: multiclass-model
spec:
  predictors:
  - graph:
      children: []
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/iris
      name: classifier
      logger:
        mode: all
    name: default
    replicas: 1
END<jupyter_output>seldondeployment.machinelearning.seldon.io/multiclass-model configured<jupyter_text>Send test request<jupyter_code>!kubectl run --quiet=true -it --rm curl --image=radial/busyboxplus:curl --restart=Never -- \
    curl -X POST -v "http://multiclass-model-default.default.svc.cluster.local:8000/api/v1.0/predictions" \
        -H "Content-Type: application/json" -d '{"data": { "ndarray": [[1,2,3,4]]}, "meta": { "puid": "hello" }}'<jupyter_output>> POST /api/v1.0/predictions HTTP/1.1


> User-Agent: curl/7.35.0


> Host: multiclass-model-default.default.svc.cluster.local:8000


> Accept: */*


> Content-Type: application/json


> Content-Length: 64


> 


< HTTP/1.1 200 OK


< Access-Control-Allow-Headers: Accept, Accept-Encoding, Authorization, Content-Length, Content-Type, X-CSRF-Token


< Access-Control-Allow-Methods: OPTIONS,POST


< Access-Control-Allow-Origin: *


< Content-Type: application/json


< Seldon-Puid: b8c38c50-2bda-419b-88c3-62a74068eea0


< X-Content-Type-Options: nosniff


< Date: Fri, 25 Sep 2020 07:47:01 GMT


< Content-Length: 139


< 


{"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.0006985194531162841,0.003668039039435755,0.9956334415074478]]},"meta":{"puid":"hello"}}<jupyter_text>Create Metrics Service<jupyter_code>%%writefile config/multiclass-service.yaml
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: seldon-multiclass-model-metrics
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - image: "seldonio/alibi-detect-server:1.3.0-dev"
        imagePullPolicy: Never
        args:
        - --model_name
        - multiclassserver
        - --http_port
        - '8080'
        - --protocol
        - seldonfeedback.http
        - --storage_uri
        - "adserver.cm_models.multiclass_one_hot.MulticlassOneHot"
        - --reply_url
        - http://message-dumper.default        
        - --event_type
        - io.seldon.serving.feedback.metrics
        - --event_source
        - io.seldon.serving.feedback
        - MetricsServer
        env:
        - name: "SELDON_DEPLOYMENT_ID"
          value: "multiclass-model"
        - name: "PREDICTIVE_UNIT_ID"
          value: "classifier"
        - name: "PREDICTIVE_UNIT_IMAGE"
          value: "alibi-detect-server:1.3.0-dev"
        - name: "PREDICTOR_ID"
          value: "default"
        securityContext:
            runAsUser: 8888
!kubectl apply -f config/multiclass-service.yaml
!kubectl get kservice<jupyter_output>NAME                                URL                                                            LATESTCREATED                             LATESTREADY   READY   REASON
multiclass-model-metrics-kservice   http://multiclass-model-metrics-kservice.default.example.com   multiclass-model-metrics-kservice-8nh2p                 False   RevisionMissing<jupyter_text>(Alternative) Kubernetes DeploymentAlternatively you can also create a kubernetes deployment instead of a kservice with the yaml below.<jupyter_code>%%writefile config/multiclass-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-multiclass-model-metrics
  labels:
    app: seldon-multiclass-model-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seldon-multiclass-model-metrics
  template:
    metadata:
      labels:
        app: seldon-multiclass-model-metrics
    spec:
      securityContext:
          runAsUser: 8888
      containers:
      - name: user-container
        image: seldonio/alibi-detect-server:1.3.0-dev
        imagePullPolicy: Never
        args:
        - --model_name
        - multiclassserver
        - --http_port
        - '8080'
        - --protocol
        - seldonfeedback.http
        - --storage_uri
        - "adserver.cm_models.multiclass_one_hot.MulticlassOneHot"
        - --reply_url
        - http://message-dumper.default        
        - --event_type
        - io.seldon.serving.feedback.metrics
        - --event_source
        - io.seldon.serving.feedback
        - MetricsServer
        env:
        - name: "SELDON_DEPLOYMENT_ID"
          value: "multiclass-model"
        - name: "PREDICTIVE_UNIT_ID"
          value: "classifier"
        - name: "PREDICTIVE_UNIT_IMAGE"
          value: "alibi-detect-server:1.3.0-dev"
        - name: "PREDICTOR_ID"
          value: "default"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-multiclass-model-metrics
  labels:
    app: seldon-multiclass-model-metrics
spec:
  selector:
    app: seldon-multiclass-model-metrics
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
!kubectl apply -f config/multiclass-deployment.yaml
!kubectl get pods<jupyter_output>NAME                                               READY   STATUS        RESTARTS   AGE
seldon-multiclass-model-metrics-5f9776bf69-25dxk   1/1     Running       0          20s
seldon-multiclass-model-metrics-5f9776bf69-55jzn   1/1     Terminating   0          10m<jupyter_text>Create Trigger<jupyter_code>!kubectl label namespace default knative-eventing-injection=enabled --overwrite=true
!kubectl get broker
%%writefile config/trigger.yaml
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
  name: multiclass-model-metrics-trigger
  namespace: default
spec:
  filter:
    sourceAndType:
      type: io.seldon.serving.feedback
  subscriber:
    ref:
      apiVersion: v1
      kind: Service
      name: seldon-multiclass-model-metrics
!kubectl apply -f config/trigger.yaml
!kubectl get trigger<jupyter_output>NAME                               READY   REASON   BROKER    SUBSCRIBER_URI                                                      AGE
multiclass-model-metrics-trigger   True             default   http://seldon-multiclass-model-metrics.default.svc.cluster.local/   1s<jupyter_text>Send feedback<jupyter_code>!kubectl run --quiet=true -it --rm curl --image=radial/busyboxplus:curl --restart=Never -- \
    curl -X POST -v "http://multiclass-model-default.default.svc.cluster.local:8000/api/v1.0/feedback" \
        -H "Content-Type: application/json" \
        -d '{"response": {"data": {"ndarray": [[0.0006985194531162841,0.003668039039435755,0.9956334415074478]]}}, "truth":{"data": {"ndarray": [[0,0,1]]}}}'<jupyter_output>> POST /api/v1.0/feedback HTTP/1.1


> User-Agent: curl/7.35.0


> Host: multiclass-model-default.default.svc.cluster.local:8000


> Accept: */*


> Content-Type: application/json


> Content-Length: 144


> 


< HTTP/1.1 200 OK


< Access-Control-Allow-Headers: Accept, Accept-Encoding, Authorization, Content-Length, Content-Type, X-CSRF-Token


< Access-Control-Allow-Methods: OPTIONS,POST


< Access-Control-Allow-Origin: *


< Content-Type: application/json


< Seldon-Puid: 7a7c95e5-70fe-4464-8b63-9e0bafe00cb1


< X-Content-Type-Options: nosniff


< Date: Fri, 25 Sep 2020 07:49:57 GMT


< Content-Length: 44


< 


{"data":{"tensor":{"shape":[0]}},"meta":{}}<jupyter_text>Check that metrics are recorded<jupyter_code>!kubectl run --quiet=true -it --rm curl --image=radial/busyboxplus:curl --restart=Never -- \
    curl -X GET -v "http://seldon-multiclass-model-metrics.default.svc.cluster.local:80/v1/metrics"<jupyter_output>> GET /v1/metrics HTTP/1.1


> User-Agent: curl/7.35.0


> Host: seldon-multiclass-model-metrics.default.svc.cluster.local


> Accept: */*


> 


< HTTP/1.1 200 OK


< Server: TornadoServer/6.0.4


< Content-Type: text/plain; version=0.0.4; charset=utf-8


< Date: Fri, 25 Sep 2020 08:07:41 GMT


< Etag: "91440f4e5e783afe116d54978ccc0e7ae6f6e252"


< Content-Length: 495


< 


# HELP seldon_metric_true_positive_total 

# TYPE seldon_metric_true_positive_total counter

seldon_metric_true_positive_total{class="CLASS_0",deployment_name="multiclass-model",image_name="alibi-detect-server",image_version="1.3.0-dev",method="ce_server",model_image="alibi-detect-server",model_name="classifier",model_version="1.3.0-dev",predictor_name="default",predictor_version="NOT_IMPLEMENTED",seldon_deployment_name="multiclass-model",worker_id="b53f1f2a-ff03-11ea-9e97-f68d14edffe0"} 1.0
{'fraction_non_alphanumeric': 0.11794476840136428, 'fraction_numerical': 0.04554956540873584, 'mean_word_length': 2.8161209068010074, 'pattern_counts': {'":': 19, '<': 55, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '537803', 'license': 'Apache-2.0', 'n_tokens_mistral': 3418, 'n_tokens_neox': 3129, 'n_words': 530, 'path': 'examples/feedback/metrics-server/README.ipynb', 'repo_name': 'kkuhu302/seldon-core', 'size': '9089'}