<jupyter_start><jupyter_text>HyperParameter tunning using  CMA-ESIn this example you will deploy 3 Katib Experiments with Covariance Matrix Adaptation Evolution Strategy (CMA-ES) using Jupyter Notebook and Katib SDK. These Experiments have various resume policies.The notebook shows how to create, get, check status and delete an Experiment.  Install required package<jupyter_code>pip install kubeflow-katib==0.10.1<jupyter_output>Defaulting to user installation because normal site-packages is not writeable
Collecting kubeflow-katib==0.10.1
  Downloading kubeflow_katib-0.10.1-py3-none-any.whl (113 kB)
[K     |████████████████████████████████| 113 kB 28.0 MB/s eta 0:00:01
[?25hCollecting table-logger>=0.3.5
  Downloading table_logger-0.3.6-py3-none-any.whl (14 kB)
Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.6/dist-packages (from kubeflow-katib==0.10.1) (2019.11.28)
Requirement already satisfied: urllib3>=1.15.1 in /usr/local/lib/python3.6/dist-packages (from kubeflow-katib==0.10.1) (1.25.8)
Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kubeflow-katib==0.10.1) (45.1.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.6/dist-packages (from kubeflow-katib==0.10.1) (2.8.1)
Requirement already satisfied: six>=1.10 in /usr/lib/python3/dist-packages (from kubeflow-katib==0.10.1) (1.11.0)
Requirement alread[...]<jupyter_text>Restart the Notebook kernel to use SDK package<jupyter_code>from IPython.display import display_html
display_html("<script>Jupyter.notebook.kernel.restart()</script>",raw=True)<jupyter_output><empty_output><jupyter_text>Import required packages<jupyter_code>import copy

from kubeflow.katib import KatibClient
from kubernetes.client import V1ObjectMeta
from kubeflow.katib import V1beta1Experiment
from kubeflow.katib import V1beta1AlgorithmSpec
from kubeflow.katib import V1beta1ObjectiveSpec
from kubeflow.katib import V1beta1FeasibleSpace
from kubeflow.katib import V1beta1ExperimentSpec
from kubeflow.katib import V1beta1ObjectiveSpec
from kubeflow.katib import V1beta1ParameterSpec
from kubeflow.katib import V1beta1TrialTemplate
from kubeflow.katib import V1beta1TrialParameterSpec<jupyter_output><empty_output><jupyter_text>Define your ExperimentYou have to create your Experiment object before deploying it. This Experiment is similar to [this](https://github.com/kubeflow/katib/blob/master/examples/v1beta1/cmaes-example.yaml) example.<jupyter_code># Experiment name and namespace.
namespace = "anonymous"
experiment_name = "cmaes-example"

metadata = V1ObjectMeta(
    name=experiment_name,
    namespace=namespace
)

# Algorithm specification.
algorithm_spec=V1beta1AlgorithmSpec(
    algorithm_name="cmaes"
)

# Objective specification.
objective_spec=V1beta1ObjectiveSpec(
    type="maximize",
    goal= 0.99,
    objective_metric_name="Validation-accuracy",
    additional_metric_names=["Train-accuracy"]
)

# Experiment search space. In this example we tune learning rate, number of layer and optimizer.
parameters=[
    V1beta1ParameterSpec(
        name="lr",
        parameter_type="double",
        feasible_space=V1beta1FeasibleSpace(
            min="0.01",
            max="0.06"
        ),
    ),
    V1beta1ParameterSpec(
        name="num-layers",
        parameter_type="int",
        feasible_space=V1beta1FeasibleSpace(
            min="2",
            max="5"
        ),
    ),
    V1beta1ParameterSpec(
        name="optimizer",
        parameter_type="categorical",
        feasible_space=V1beta1FeasibleSpace(
            list=["sgd", "adam", "ftrl"]
        ),
    ),
]



# JSON template specification for the Trial's Worker Kubernetes Job.
trial_spec={
    "apiVersion": "batch/v1",
    "kind": "Job",
    "spec": {
        "template": {
            "metadata": {
                "annotations": {
                    "sidecar.istio.io/inject": "false"
                }
            },
            "spec": {
                "containers": [
                    {
                        "name": "training-container",
                        "image": "docker.io/kubeflowkatib/mxnet-mnist:v1beta1-45c5727",
                        "command": [
                            "python3",
                            "/opt/mxnet-mnist/mnist.py",
                            "--batch-size=64",
                            "--lr=${trialParameters.learningRate}",
                            "--num-layers=${trialParameters.numberLayers}",
                            "--optimizer=${trialParameters.optimizer}"
                        ]
                    }
                ],
                "restartPolicy": "Never"
            }
        }
    }
}

# Configure parameters for the Trial template.
trial_template=V1beta1TrialTemplate(
    primary_container_name="training-container",
    trial_parameters=[
        V1beta1TrialParameterSpec(
            name="learningRate",
            description="Learning rate for the training model",
            reference="lr"
        ),
        V1beta1TrialParameterSpec(
            name="numberLayers",
            description="Number of training model layers",
            reference="num-layers"
        ),
        V1beta1TrialParameterSpec(
            name="optimizer",
            description="Training model optimizer (sdg, adam or ftrl)",
            reference="optimizer"
        ),
    ],
    trial_spec=trial_spec
)


# Experiment object.
experiment = V1beta1Experiment(
    api_version="kubeflow.org/v1beta1",
    kind="Experiment",
    metadata=metadata,
    spec=V1beta1ExperimentSpec(
        max_trial_count=7,
        parallel_trial_count=3,
        max_failed_trial_count=3,
        algorithm=algorithm_spec,
        objective=objective_spec,
        parameters=parameters,
        trial_template=trial_template,
    )
)<jupyter_output><empty_output><jupyter_text>Define Experiments with resume policyWe will define another 2 Experiments with ResumePolicy = Never and ResumePolicy = FromVolume.Experiment with _Never_ resume policy can't be resumed, the Suggestion resources will be deleted.Experiment with _FromVolume_ resume policy can be resumed, volume is attached to the Suggestion. Suggestion's PVC and PV should be created for the Suggestion.<jupyter_code>experiment_never_resume_name = "never-resume-cmaes"
experiment_from_volume_resume_name = "from-volume-resume-cmaes"

# Create new Experiments from the previous Experiment info.
# Define Experiment with never resume.
experiment_never_resume = copy.deepcopy(experiment)
experiment_never_resume.metadata.name = experiment_never_resume_name
experiment_never_resume.spec.resume_policy = "Never"
experiment_never_resume.spec.max_trial_count = 4

# Define Experiment with from volume resume.
experiment_from_volume_resume = copy.deepcopy(experiment)
experiment_from_volume_resume.metadata.name = experiment_from_volume_resume_name
experiment_from_volume_resume.spec.resume_policy = "FromVolume"
experiment_from_volume_resume.spec.max_trial_count = 4<jupyter_output><empty_output><jupyter_text>You can print the Experiment's info to verify it before submission.<jupyter_code>print(experiment.metadata.name)
print(experiment.spec.algorithm.algorithm_name)
print("-----------------")
print(experiment_never_resume.metadata.name)
print(experiment_never_resume.spec.resume_policy)
print("-----------------")
print(experiment_from_volume_resume.metadata.name)
print(experiment_from_volume_resume.spec.resume_policy)<jupyter_output>cmaes-example
cmaes
-----------------
never-resume-cmaes
Never
-----------------
from-volume-resume-cmaes
FromVolume<jupyter_text>Create your ExperimentYou have to create Katib client to use the SDK.<jupyter_code># Create client.
kclient = KatibClient()

# Create your Experiment.
kclient.create_experiment(experiment,namespace=namespace)<jupyter_output><empty_output><jupyter_text>Create other Experiments.<jupyter_code># Create Experiment with never resume.
kclient.create_experiment(experiment_never_resume,namespace=namespace)
# Create Experiment with from volume resume.
kclient.create_experiment(experiment_from_volume_resume,namespace=namespace)<jupyter_output><empty_output><jupyter_text>Get your ExperimentYou can get your Experiment by name and receive required data.<jupyter_code>exp = kclient.get_experiment(name=experiment_name, namespace=namespace)
print(exp)
print("-----------------\n")

# Get the max trial count and latest status.
print(exp["spec"]["maxTrialCount"])
print(exp["status"]["conditions"][-1])<jupyter_output>{'apiVersion': 'kubeflow.org/v1beta1', 'kind': 'Experiment', 'metadata': {'creationTimestamp': '2020-11-30T19:02:20Z', 'finalizers': ['update-prometheus-metrics'], 'generation': 1, 'name': 'cmaes-example', 'namespace': 'anonymous', 'resourceVersion': '170779219', 'selfLink': '/apis/kubeflow.org/v1beta1/namespaces/anonymous/experiments/cmaes-example', 'uid': '6d8b16d3-3778-4fc1-ba3d-d524ec487450'}, 'spec': {'algorithm': {'algorithmName': 'cmaes'}, 'maxFailedTrialCount': 3, 'maxTrialCount': 7, 'metricsCollectorSpec': {'collector': {'kind': 'StdOut'}}, 'objective': {'additionalMetricNames': ['Train-accuracy'], 'goal': 0.99, 'metricStrategies': [{'name': 'Validation-accuracy', 'value': 'max'}, {'name': 'Train-accuracy', 'value': 'max'}], 'objectiveMetricName': 'Validation-accuracy', 'type': 'maximize'}, 'parallelTrialCount': 3, 'parameters': [{'feasibleSpace': {'max': '0.06', 'min': '0.01'}, 'name': 'lr', 'parameterType': 'double'}, {'feasibleSpace': {'max': '5', 'min': '2'}, 'name': 'num-[...]<jupyter_text>Get all ExperimentsYou can get list of the current Experiments.<jupyter_code># Get names from the running Experiments.
exp_list = kclient.get_experiment(namespace=namespace)

for exp in exp_list["items"]:
    print(exp["metadata"]["name"])<jupyter_output>cmaes-example
from-volume-resume-cmaes
never-resume-cmaes<jupyter_text>Get the current Experiment statusYou can check the current Experiment status.<jupyter_code>kclient.get_experiment_status(name=experiment_name, namespace=namespace)<jupyter_output><empty_output><jupyter_text>You can check if your Experiment is succeeded.<jupyter_code>kclient.is_experiment_succeeded(name=experiment_name, namespace=namespace)<jupyter_output><empty_output><jupyter_text>List of the current TrialsYou can get list of the current trials with the latest status.<jupyter_code># Trial list.
kclient.list_trials(name=experiment_name, namespace=namespace)<jupyter_output><empty_output><jupyter_text>Get the optimal HyperParametersYou can get the current optimal Trial from your Experiment. For the each metric you can see the max, min and latest value.<jupyter_code># Optimal HPs.
kclient.get_optimal_hyperparameters(name=experiment_name, namespace=namespace)<jupyter_output><empty_output><jupyter_text>Status for the Suggestion objectsYou can check the Suggestion object status for more information about resume status.For Experiment with FromVolume you should be able to check created PVC and PV.<jupyter_code># Get the current Suggestion status for the never resume Experiment.
suggestion = kclient.get_suggestion(name=experiment_never_resume_name, namespace=namespace)

print(suggestion["status"]["conditions"][-1]["message"])
print("-----------------")

# Get the current Suggestion status for the from volume Experiment.
suggestion = kclient.get_suggestion(name=experiment_from_volume_resume_name, namespace=namespace)

print(suggestion["status"]["conditions"][-1]["message"])<jupyter_output>Suggestion is succeeded, can't be restarted
-----------------
Suggestion is succeeded, suggestion volume is not deleted, can be restarted<jupyter_text>Delete your ExperimentsYou can delete your Experiments.<jupyter_code>kclient.delete_experiment(name=experiment_name, namespace=namespace)
kclient.delete_experiment(name=experiment_never_resume_name, namespace=namespace)
kclient.delete_experiment(name=experiment_from_volume_resume_name, namespace=namespace)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11859883236030025, 'fraction_numerical': 0.021601334445371143, 'mean_word_length': 4.3843735967669515, 'pattern_counts': {'":': 13, '<': 62, '<?xml version=': 0, '>': 68, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '90146', 'license': 'Apache-2.0', 'n_tokens_mistral': 3856, 'n_tokens_neox': 3665, 'n_words': 762, 'path': 'examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb', 'repo_name': 'evilnick/katib', 'size': '11990'}