<jupyter_start><jupyter_text>Tensorflow Load and Benchmark TestsUsing a pretrained model for [Tensorflow flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) * Load test the model at fixed rate * Benchmark the model to find maximum throughput and saturation handling   Setup   * Create a 3 node GCP cluster with n1-standard-8 node  * Install Seldon Core    TODO   * gRPC  * Run vegeta on separate node to model servers using affinity/taints<jupyter_code>!kubectl create namespace seldon
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
import sys

sys.path.append("../")
from vegeta_utils import *<jupyter_output><empty_output><jupyter_text>Put Taint on Nodes<jupyter_code>raw=!kubectl get nodes -o jsonpath='{.items[0].metadata.name}'
firstNode = raw[0]
raw=!kubectl get nodes -o jsonpath='{.items[1].metadata.name}'
secondNode = raw[0]
raw=!kubectl get nodes -o jsonpath='{.items[2].metadata.name}'
thirdNode = raw[0]
!kubectl taint nodes '{firstNode}' loadtester=active:NoSchedule
!kubectl taint nodes '{secondNode}' model=active:NoSchedule 
!kubectl taint nodes '{thirdNode}' model=active:NoSchedule<jupyter_output>node/pool-triv8uq93-3oaz0 tainted
error: Node pool-triv8uq93-3oaz1 already has model taint(s) with same effect(s) and --overwrite is false
error: Node pool-triv8uq93-3oazd already has model taint(s) with same effect(s) and --overwrite is false<jupyter_text>Benchmark with Saturation Test<jupyter_code>%%writefile tf_flowers.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: tf-flowers
spec:
  protocol: tensorflow
  transport: rest
  predictors:
  - graph:
      implementation: TENSORFLOW_SERVER
      modelUri: gs://kfserving-samples/models/tensorflow/flowers
      name:  flowers
      parameters:
        - name: model_name
          type: STRING
          value: flowers
    componentSpecs:
    - spec:
        containers:
        - name: flowers
          resources:
            requests:
              cpu: '2'
        tolerations:
        - key: model
          operator: Exists
          effect: NoSchedule
    name: default
    replicas: 1
run_model("tf_flowers.yaml")<jupyter_output>Available with 1 pods<jupyter_text>Run test to gather the max throughput of the model<jupyter_code>results = run_vegeta_test("tf_vegeta_cfg.yaml", "vegeta_max.yaml", "11m")
print(json.dumps(results, indent=4))
saturation_throughput = int(results["throughput"])
print("Max Throughtput=", saturation_throughput)<jupyter_output>Max Throughtput= 24<jupyter_text>Load Tests with HPARun with an HPA at saturation rate to check:  * Latencies affected by scaling<jupyter_code>%%writefile tf_flowers.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: tf-flowers
spec:
  protocol: tensorflow
  transport: rest
  predictors:
  - graph:
      implementation: TENSORFLOW_SERVER
      modelUri: gs://kfserving-samples/models/tensorflow/flowers
      name:  flowers
      parameters:
        - name: model_name
          type: STRING
          value: flowers
    componentSpecs:
    - hpaSpec:
        minReplicas: 1
        maxReplicas: 5
        metrics:
        - resource:
            name: cpu
            targetAverageUtilization: 70
          type: Resource
      spec:
        containers:
        - name: flowers
          resources:
            requests:
              cpu: '1'
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 5
            successThreshold: 1
            tcpSocket:
              port: http
            timeoutSeconds: 5
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 20
            periodSeconds: 5
            successThreshold: 1
            tcpSocket:
              port: http
            timeoutSeconds: 5
        tolerations:
        - key: model
          operator: Exists
          effect: NoSchedule
    name: default
    replicas: 1
run_model("tf_flowers.yaml")
rate = saturation_throughput
duration = "10m"
%env DURATION=$duration
%env RATE=$rate/1s
!cat vegeta_cfg.tmpl.yaml | envsubst > vegeta.tmp.yaml
!cat vegeta.tmp.yaml
results = run_vegeta_test("tf_vegeta_cfg.yaml", "vegeta.tmp.yaml", "11m")
print(json.dumps(results, indent=4))
print_vegeta_results(results)<jupyter_output>Latencies:
	mean: 259.990239 ms
	50th: 131.917169 ms
	90th: 310.053255 ms
	95th: 916.684759 ms
	99th: 2775.05271 ms

Throughput: 23.997572337989126/s
Errors: False
{'fraction_non_alphanumeric': 0.08715798764342454, 'fraction_numerical': 0.02846425419240953, 'mean_word_length': 3.1097008159564825, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '219065', 'license': 'Apache-2.0', 'n_tokens_mistral': 1528, 'n_tokens_neox': 1434, 'n_words': 359, 'path': 'testing/benchmarking/tensorflow/tensorflow.ipynb', 'repo_name': 'edbighead/seldon-core', 'size': '4532'}