<jupyter_start><jupyter_text>Installation and imports<jupyter_code>!pip install kfmd --upgrade --user
!pip install pandas --upgrade --user

from kfmd import metadata
import pandas
from datetime import datetime<jupyter_output>Requirement already up-to-date: kfmd in ./.local/lib/python3.6/site-packages (0.1.8)
Requirement already up-to-date: pandas in ./.local/lib/python3.6/site-packages (1.0.1)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas) (2.8.1)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.18.1)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas) (2019.3)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.6.1->pandas) (1.11.0)<jupyter_text>Create a workspace, run and execution<jupyter_code>ws1 = metadata.Workspace(
    # Connect to metadata-service in namesapce kubeflow in k8s cluster.
    backend_url_prefix="metadata-service.kubeflow.svc.cluster.local:8080",
    name="ws1",
    description="a workspace for testing",
    labels={"n1": "v1"})
r = metadata.Run(
    workspace=ws1,
    name="run-" + datetime.utcnow().isoformat("T") ,
    description="a run in ws_1",
)
exec = metadata.Execution(
    name = "execution" + datetime.utcnow().isoformat("T") ,
    workspace=ws1,
    run=r,
    description="execution example",
)<jupyter_output><empty_output><jupyter_text>Log data set, model and its evaluation<jupyter_code>data_set = exec.log_input(
        metadata.DataSet(
            description="an example data",
            name="mytable-dump",
            owner="<EMAIL>",
            uri="file://path/to/dataset",
            version="v1.0.0",
            query="SELECT * FROM mytable"))
model = exec.log_output(
    metadata.Model(
            name="MNIST",
            description="model to recognize handwritten digits",
            owner="<EMAIL>",
            uri="gcs://my-bucket/mnist",
            model_type="neural network",
            training_framework={
                "name": "tensorflow",
                "version": "v1.0"
            },
            hyperparameters={
                "learning_rate": 0.5,
                "layers": [10, 3, 1],
                "early_stop": True
            },
            version="v0.0.1",
            labels={"mylabel": "l1"}))
metrics = exec.log_output(
    metadata.Metrics(
            name="MNIST-evaluation",
            description="validating the MNIST model to recognize handwritten digits",
            owner="<EMAIL>",
            uri="gcs://my-bucket/mnist-eval.csv",
            data_set_id=data_set.id,
            model_id=model.id,
            metrics_type=metadata.Metrics.VALIDATION,
            values={"accuracy": 0.95},
            labels={"mylabel": "l1"}))<jupyter_output><empty_output><jupyter_text>List all the models in the workspace<jupyter_code>pandas.DataFrame.from_dict(ws1.list(metadata.Model.ARTIFACT_TYPE_NAME))<jupyter_output><empty_output><jupyter_text>Get basic lineage<jupyter_code>print("model id is %s\n" % model.id)<jupyter_output>model id is 8<jupyter_text>Find the execution that produces this model.<jupyter_code>output_events = ws1.client.list_events2(model.id).events
assert len(output_events) == 1
execution_id = output_events[0].execution_id
print(execution_id)<jupyter_output>3<jupyter_text>Find all events related to that execution.<jupyter_code>all_events = ws1.client.list_events(execution_id).events
assert len(all_events) == 3
print("\nAll events related to this model:")
pandas.DataFrame.from_dict([e.to_dict() for e in all_events])<jupyter_output>All events related to this model:
{'fraction_non_alphanumeric': 0.1314541622760801, 'fraction_numerical': 0.02370916754478398, 'mean_word_length': 3.3744239631336406, 'pattern_counts': {'":': 9, '<': 28, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '115863', 'license': 'Apache-2.0', 'n_tokens_mistral': 1277, 'n_tokens_neox': 1219, 'n_words': 244, 'path': 'ch06/Metadata.ipynb', 'repo_name': 'gabrielclimb/intro-to-ml-with-kubeflow-examples', 'size': '3835'}