<jupyter_start><jupyter_text>Distributed Tracing Template Illustrate the configuration for allowing distributed tracing using Jaeger.  Dependencies * [Helm](https://github.com/kubernetes/helm) * [Minikube](https://github.com/kubernetes/minikube)  Test using Minikube**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need [s2i >= 1.1.13](https://github.com/openshift/source-to-image/releases/tag/v1.1.13)**<jupyter_code>!minikube start --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
!kubectl create namespace seldon
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
!helm init
!kubectl rollout status deploy/tiller-deploy -n kube-system<jupyter_output><empty_output><jupyter_text>Install JaegerWe will use the Jaeger All-in-1 resource found at the [Jaeger Kubernetes repo](https://github.com/jaegertracing/jaeger-kubernetes).<jupyter_code>!kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml -n seldon<jupyter_output>deployment.extensions/jaeger created
service/jaeger-query created
service/jaeger-collector created
service/jaeger-agent created
service/zipkin created<jupyter_text>Start Jaeger UI```minikube service jaeger-query -n seldon```  Install Seldon<jupyter_code>!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system<jupyter_output>partitioned roll out complete: 1 new pods have been updated...<jupyter_text>Setup IngressThere are gRPC issues with the latest Ambassador, so we rewcommend 0.40.2 until these are fixed.<jupyter_code>!helm install stable/ambassador --name ambassador --set crds.keep=false
!kubectl rollout status deployment.apps/ambassador<jupyter_output>Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out<jupyter_text>Create Jaeger ConfigMap<jupyter_code>!pygmentize tracing-configmap.yaml
!kubectl apply -f tracing-configmap.yaml -n seldon<jupyter_output>configmap/tracing-config unchanged<jupyter_text>Run Example REST Deployment<jupyter_code>!pygmentize deployment_rest.json
!kubectl create -f deployment_rest.json
!kubectl rollout status deployment/tracing-example-tracing-535f3a8
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    tracing-example --namespace seldon -p<jupyter_output>----------------------------------------
SENDING NEW REQUEST:

[[ 0.871  0.027 -0.599]]
RECEIVED RESPONSE:
meta {
  puid: "kl0gv9mc7pe39kst3n7loeu6va"
  requestPath {
    key: "model1"
    value: "seldonio/mock_classifier_rest:1.1"
  }
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.056412411380098754
        }
      }
    }
  }
}<jupyter_text>Check the Jaeger UI. You should be able to find traces like below:![rest](jaeger-ui-rest-example.png)<jupyter_code>!kubectl delete -f deployment_rest.json<jupyter_output>seldondeployment.machinelearning.seldon.io "tracing-example" deleted<jupyter_text>Run Example GRPC Deployment<jupyter_code>!kubectl create -f deployment_grpc.json
!kubectl rollout status deployment/tracing-example-tracing-d240ae0
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    tracing-example --namespace seldon -p --grpc<jupyter_output>----------------------------------------
SENDING NEW REQUEST:

[[-0.024 -0.489 -0.362]]
RECEIVED RESPONSE:
meta {
  puid: "7m4ikainug51qqsq626cos499o"
  requestPath {
    key: "model1"
    value: "seldonio/mock_classifier_grpc:1.1"
  }
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.03885332622207222
        }
      }
    }
  }
}<jupyter_text>Check the Jaeger UI. You should be able to find traces like below:![grpc](jaeger-ui-grpc-example.png)<jupyter_code>!kubectl delete -f deployment_grpc.json<jupyter_output>seldondeployment.machinelearning.seldon.io "tracing-example" deleted
{'fraction_non_alphanumeric': 0.12324627671055471, 'fraction_numerical': 0.02698035829915821, 'mean_word_length': 4.576413959085439, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '585422', 'license': 'Apache-2.0', 'n_tokens_mistral': 1676, 'n_tokens_neox': 1551, 'n_words': 334, 'path': 'examples/models/template_model_tracing/tracing.ipynb', 'repo_name': 'Diffblue-benchmarks/Mjlodge-seldon-core', 'size': '4633'}