<jupyter_start><jupyter_text>TFX on KubeFlow Pipelines ExampleThis notebook should be run inside a KF Pipelines cluster. Install TFX and KFP packages<jupyter_code>!pip3 install https://storage.googleapis.com/ml-pipeline/tfx/tfx-0.12.0rc0-py2.py3-none-any.whl 
!pip3 install kfp --upgrade<jupyter_output><empty_output><jupyter_text>Enable DataFlow API for your GKE cluster  Get the TFX repo with sample pipeline<jupyter_code>!git clone https://github.com/tensorflow/tfx
# copy the trainer code to a storage bucket as the TFX pipeline will need that code file in GCS
from tensorflow import gfile
gfile.Copy('tfx/examples/chicago_taxi_pipeline/taxi_utils.py', 'gs://<my bucket>/<path>/taxi_utils.py')<jupyter_output><empty_output><jupyter_text>Configure the TFX pipeline exampleReload this cell by running the load command to get the pipeline configuration file```%load tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow.py```Configure:- Set `_input_bucket` to the GCS directory where you've copied taxi_utils.py. I.e. gs:////- Set `_output_bucket` to the GCS directory where you've want the results to be written- Set GCP project ID (replace my-gcp-project). Note that it should be project ID, not project name.The dataset in BigQuery has 100M rows, you can change the query parameters in WHERE clause to limit the number of rows used.<jupyter_code>%load tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow.py<jupyter_output><empty_output><jupyter_text>Compile the pipeline and submit a run to the Kubeflow cluster<jupyter_code># Get or create a new experiment
import kfp
client = kfp.Client()
experiment_name='TFX Examples'
try:
    experiment_id = client.get_experiment(experiment_name=experiment_name).id
except:
    experiment_id = client.create_experiment(experiment_name).id

pipeline_filename = 'chicago_taxi_pipeline_kubeflow.tar.gz'

#Submit a pipeline run
run_name = 'Run 1'
run_result = client.run_pipeline(experiment_id, run_name, pipeline_filename, {})<jupyter_output><empty_output><jupyter_text>Connect to the ML Metadata Store<jupyter_code>!pip3 install ml_metadata
from ml_metadata.metadata_store import metadata_store
from ml_metadata.proto import metadata_store_pb2
import os

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.mysql.host = os.getenv('MYSQL_SERVICE_HOST')
connection_config.mysql.port = int(os.getenv('MYSQL_SERVICE_PORT'))
connection_config.mysql.database = 'mlmetadata'
connection_config.mysql.user = 'root'
store = metadata_store.MetadataStore(connection_config)
# Get all output artifacts
store.get_artifacts()
# Get a specific artifact type

# TFX types 
# types = ['ModelExportPath', 'ExamplesPath', 'ModelBlessingPath', 'ModelPushPath', 'TransformPath', 'SchemaPath']

store.get_artifacts_by_type('ExamplesPath')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08794326241134752, 'fraction_numerical': 0.005673758865248227, 'mean_word_length': 6.924157303370786, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '19182', 'license': 'Apache-2.0', 'n_tokens_mistral': 914, 'n_tokens_neox': 907, 'n_words': 240, 'path': 'samples/tfx-oss/TFX Example.ipynb', 'repo_name': 'adrian555/pipelines', 'size': '2820'}