<jupyter_start><jupyter_text>Chicago taxi example using TFX schema curation custom componentThis example demonstrate the use of schema curation custom component. User defined function `schema_fn` defined in `module_file.py` is used to change schema feature `tips` from required to optional using schema curation component.base code taken from: https://github.com/tensorflow/tfx/blob/master/docs/tutorials/tfx/components_keras.ipynb  Setup  Install TFX**Note: In Google Colab, because of package updates, the first time you run this cell you must restart the runtime (Runtime > Restart runtime ...).**<jupyter_code>!pip install -U tfx
x = !pwd

if 'schemacomponent' not in str(x):
  !git clone https://github.com/rcrowe-google/schemacomponent
  %cd schemacomponent/example<jupyter_output>fatal: destination path 'schemacomponent' already exists and is not an empty directory.
/content/schemacomponent/example<jupyter_text>Chicago taxi example pipeline<jupyter_code>import os
import pprint
import tempfile
import urllib

import absl
import tensorflow as tf
import tensorflow_model_analysis as tfma
tf.get_logger().propagate = False
pp = pprint.PrettyPrinter()

from tfx import v1 as tfx
from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext

%load_ext tfx.orchestration.experimental.interactive.notebook_extensions.skip
from schemacomponent.component import component<jupyter_output><empty_output><jupyter_text>Set up pipeline paths<jupyter_code># This is the root directory for your TFX pip package installation.
_tfx_root = tfx.__path__[0]<jupyter_output><empty_output><jupyter_text>Download example dataWe download the example dataset for use in our TFX pipeline.The dataset we're using is the [Taxi Trips dataset](https://data.cityofchicago.org/Transportation/Taxi-Trips/wrvz-psew) released by the City of Chicago. The columns in this dataset are:pickup_community_areafaretrip_start_monthtrip_start_hourtrip_start_daytrip_start_timestamppickup_latitudepickup_longitudedropoff_latitudedropoff_longitudetrip_milespickup_census_tractdropoff_census_tractpayment_typecompanytrip_secondsdropoff_community_areatipsWith this dataset, we will build a model that predicts the `tips` of a trip.<jupyter_code>_data_root = tempfile.mkdtemp(prefix='tfx-data')
DATA_PATH = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/chicago_taxi_pipeline/data/simple/data.csv'
_data_filepath = os.path.join(_data_root, "data.csv")
urllib.request.urlretrieve(DATA_PATH, _data_filepath)<jupyter_output><empty_output><jupyter_text>Run TFX components In the cells that follow, we create TFX components one-by-one and generates `schema` using `schemaGen` component.<jupyter_code>context = InteractiveContext()

#create and run exampleGen component
example_gen = tfx.components.CsvExampleGen(input_base=_data_root)
context.run(example_gen)

#create and run statisticsGen component
statistics_gen = tfx.components.StatisticsGen(
    examples=example_gen.outputs['examples'])
context.run(statistics_gen)

#create and run schemaGen component
schema_gen = tfx.components.SchemaGen(
    statistics=statistics_gen.outputs['statistics'],
    infer_feature_shape=False)
context.run(schema_gen)<jupyter_output>WARNING:absl:InteractiveContext pipeline_root argument not provided: using temporary directory /tmp/tfx-interactive-2021-06-30T11_52_08.566472-s8ndb5at as root for pipeline outputs.
WARNING:absl:InteractiveContext metadata_connection_config not provided: using SQLite ML Metadata database at /tmp/tfx-interactive-2021-06-30T11_52_08.566472-s8ndb5at/metadata.sqlite.
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.<jupyter_text>Schema curation custom componentUsing schema curation component `tips` is changed into `optional` featureCode for modifying schema is in user supplied `schema_fn` in `module_file.py`  Display infered schemaIn the infered schema, `tips` feature is shown as a `required` feature:      tips | FLOAT |	required	| single<jupyter_code>#display infered schema
context.show(schema_gen.outputs['schema'])<jupyter_output><empty_output><jupyter_text>Modifying schema<jupyter_code>#schema curation component
schema_curation = component.SchemaCuration(schema=schema_gen.outputs['schema'],
        module_file='module_file.py')
context.run(schema_curation)<jupyter_output>ERROR:absl:udf_utils.get_fn {'module_file': 'module_file.py', 'module_path': None, 'schema_fn': None} 'schema_fn'<jupyter_text>Display modified schemafeature `tips` is now `optional` in the modified schema<jupyter_code>context.show(schema_curation.outputs['custom_schema'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07873042044517725, 'fraction_numerical': 0.009480626545754328, 'mean_word_length': 7.775768535262206, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '496718', 'license': 'Apache-2.0', 'n_tokens_mistral': 1513, 'n_tokens_neox': 1469, 'n_words': 369, 'path': 'tfx_addons/schema_curation/example/taxi_example_colab.ipynb', 'repo_name': 'vulkomilev/tfx-addons', 'size': '4852'}