<jupyter_start><jupyter_text>Step 3: Data ValidationUse the code below to run TensorFlow Data Validation on your pipeline.  Start by importing and opening the metadata store.<jupyter_code>!pip install -q papermill
!pip install -q matplotlib
!pip install -q networkx
from __future__ import print_function


import os
import tfx_utils
%matplotlib notebook

def _make_default_sqlite_uri(pipeline_name):
    return os.path.join(os.environ['HOME'], 'airflow/tfx/metadata', pipeline_name, 'airflow.db')

def get_metadata_store(pipeline_name):
    return tfx_utils.TFXReadonlyMetadataStore.from_sqlite_db(_make_default_sqlite_uri(pipeline_name))

pipeline_name = 'taxi'

pipeline_db_path = _make_default_sqlite_uri(pipeline_name)
print('Pipeline DB:\n{}'.format(pipeline_db_path))

store = get_metadata_store(pipeline_name)<jupyter_output>Pipeline DB:
/Users/dileepholla/airflow/tfx/metadata/taxi/metadata.db<jupyter_text>Now print out the data artifacts:<jupyter_code># Visualize properties of example artifacts
print (store.get_artifacts_of_type_df(tfx_utils.TFXArtifactTypes.EXAMPLES))<jupyter_output>Empty DataFrame
Columns: []
Index: []<jupyter_text>Now visualize the dataset features.Hint: try ID 2 or 3<jupyter_code># Visualize stats for data
store.display_stats_for_examples(3)<jupyter_output><empty_output><jupyter_text>Now plot the artifact lineage:<jupyter_code># Try different IDs here. Click stop in the plot when changing IDs.
%matplotlib notebook
store.plot_artifact_lineage(2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08514851485148515, 'fraction_numerical': 0.0033003300330033004, 'mean_word_length': 8.187878787878788, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '260615', 'license': 'MIT', 'n_tokens_mistral': 524, 'n_tokens_neox': 509, 'n_words': 114, 'path': 'Labs/Lab 7 - airflow tfx/notebooks/step3.ipynb', 'repo_name': 'akashmdubey/CSYE-Big-Data-Intelligent-Analytics-Raw', 'size': '1515'}