<commit_before># TFX Generic Container-based Component Proposal

Status        | Proposed
:------------ | :-------
**RFC #**     | https://github.com/tensorflow/community/pull/146
**Author(s)** | <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>), <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>), <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>)
**Sponsor**   | <NAME> (<EMAIL>)
**Updated**   | 2019-09-04

## Objective

This document proposes a design to enable users to attach an arbitrary
containerized program as a component to a pipeline authored using the TFX DSL,
in a way that inter-operates with other components.

This RFC assumes some clarification on the TFX’s use of artifacts and metadata
as explained in [this section of TFX user guide](https://www.tensorflow.org/tfx/guide#artifacts).

## Motivation

A key value proposition provided by
[Kubeflow Pipelines (KFP)](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/)
is letting users orchestrate arbitrary containers as part of a Machine Learning
(ML) pipeline. Many users already have custom ML applications written in
languages other than Python (e.g. in R, Java, C++ etc), and the ability to chain
existing containerized application programs with other pipeline steps through a
Python DSL is valuable. As of 2019/09
([tfx 0.14](https://github.com/tensorflow/tfx/blob/0.14.0rc1//RELEASE.md#version-0140),
even though TFX supports `KubeflowDagRunner` as an orchestrator, the
[TFX DSL](https://github.com/tensorflow/tfx/tree/0.14.0rc1/tfx) does not provide
a mechanism to accomplish this.

## User Benefit

This document proposes a way to define a proper component based on any
containerized program as a solution to address this problem. The proposed
**container-based component** is realized by a simple DSL extension for custom
containers to the TFX DSL. It enables users to easily declare input and output
artifacts of a pipeline step implemented as a custom container-based program, as
well as a prescription for how to invoke the container’s entrypoint while
passing in the relevant artifact metadata and execution parameters.

In doing so, we will not only enable custom containerized application programs
to be used in TFX, but also augment KFP-based pipelines with the following
capabilities:

*   **Metadata-centric interface**: The proposed container-based component
    provides a framework to clearly declare input- and output- signatures of the
    container-based component in terms of
    [strongly typed](https://github.com/google/ml-metadata/blob/ba69ae039bd2205ec2d7b982b3bfdda4718bf8df/ml_metadata/proto/metadata_store.proto#L55)
    [artifact metadata](https://github.com/google/ml-metadata/blob/ba69ae039bd2205ec2d7b982b3bfdda4718bf8df/ml_metadata/proto/metadata_store.proto#L29).
    This is a key value proposition of TFX DSL.

    *   As of [kfp 0.1.26](https://github.com/kubeflow/pipelines/tree/0.1.26),
        the
        [KFP DSL](https://github.com/kubeflow/pipelines/tree/0.1.26/sdk/python/kfp/dsl)
        has type system to input- and output- of a component. However, it is in
        a way that it doesn’t attach semantic meaning to the input- and output-
        variables in a way compatible to ML Metadata, and the way how TFX
        uitilizes it to realize its functionalies.

*   **Metadata-driven component execution**: Metadata-centric interface of
    components enables TFX Driver and Publisher logic to be applied
    consistently, thus enabling caching of component execution as well
    component-specific (meta)data-driven execution decisions

    *   *Example*: ModelValidator can validate models trained with *any*
        component, so long as the produced artifact is of the *type* Model.

*   **Inter-Component communication via ML Metadata store**: This enables higher
    order component-specific driver logic that depends on another component’s
    behavior.

    *   *Example*: Pusher can choose to proceed or halt pushing depending on
        output artifact from ModelValidator.

*   **Ability to share and reuse a wrapped component as drop-in replacement with
    another**: As a result of the artifact centric, strongly typed input- and
    output- signatures, it enables robust sharing and drop-in replacement of
    components, so long as signatures (the list of input- and output- artifact
    *types*) are the same.

Additionally, the proposed container-based component will enable the following
new features in TFX DSL, which already exist In Kubeflow Pipelines:

*   **Ability to use any containerized application program in a pipeline**: The
    proposed container-based component does not preclude any containerized
    application programs from being used as a pipeline step.

*   **Ability to have fine-grained control on the underlying k8s execution
    environment**: The proposed container-based component preserves the user’s
    ability to control underlying Kubernetes runtime configuration for
    containerized program execution.

## Design Proposal

As stated previously, custom containers may be written in arbitrary languages.
The input interface to containers is restricted to the command-line used to
invoke the application, while the output interface is through file I/O (either
STDOUT, STDOUT, or container-local files).

Since container applications may not have access to TFX libraries, they are not
able to (or don’t even wish to) serialize/de-serialize
[ML Metadata](https://github.com/google/ml-metadata/blob/master/g3doc/get_started.md)
representing input- and output- artifacts, which is what defines the interface
to all TFX components[^1]. Hence, this design proposes DSL extensions that allow
users to declare the types of the input and output artifacts, and directly
reference the URIs of these artifacts when invoking their containers, while
retaining the ability to invoke the containerized program in exactly the way it
expects with regards to flags, arguments and environmental variables. The TFX
framework itself will generate the necessary driver-level code so that metadata
is logged and used for caching executions.

[^1]: ML Metadata is also what enables inter-component communication to realize
    artifact driven component-specific behavior (such as ExampleValidator and
    ModelValidator)

In order to make this proposal concrete, let’s consider a motivating example of
creating a simple 2-step pipeline. The first step generates examples, and the
second trains a model based on the previously produced examples. The following
shows some example code for these two steps:

*   `my_example_gen.py`

```python
import argparse import pandas as pd

def convert_and_save(df, file):
  ...

if __name__ == '__main__':

    parser = argparse.ArgumentParser()

    # URI location of input CSV file
    parser.add_argument('--input_csv_file', ...)
    # URI location of output CSV file
    parser.add_argument('--output_csv_file', ...)
    # Number of CSV columns in the input file
    parser.add_argument('--num_csv_columns', ...)

    args = parser.parse_args()
    arguments = args.__dict__

    df = pd.read_csv(arguments['input_csv_file'],
                     usecols=range(0, int(arguments['num_csv_columns'])))

    # implementation of business logic to ingest dataset
    convert_and_save(df, arguments['output_csv_file'])

```

*   `my_trainer.py`

```python


import argparse
import pandas as pd
import sklearn.linear_model import Logisticregression

def load_dataset(file):
  ...

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    # URI location of input file with pre-processed examples.
    parser.add_argument('--input_examples', ...)
    # URI location to output the trained model
    parser.add_argument('--output_model', ...)
    # Number of training steps to use.
    parser.add_argument('--train_steps', ...)

    args = parser.parse_args()
    arguments = args.__dict__

    x_train, y_train, x_eval, y_eval = load_datset(arguments['input_examples'])

    model = LogisticRegression()
    model.fit(x_train, y_train, args.train_steps)

    #
    # ... perform grid search, etc, ...
    #
<commit_msg>Style fix for code block<commit_after># TFX Generic Container-based Component Proposal

Status        | Proposed
:------------ | :-------
**RFC #**     | https://github.com/tensorflow/community/pull/146
**Author(s)** | <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>), <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>), <NAME> (<a href="mailto:<EMAIL>"><EMAIL></a>)
**Sponsor**   | <NAME> (<EMAIL>)
**Updated**   | 2019-09-04

## Objective

This document proposes a design to enable users to attach an arbitrary
containerized program as a component to a pipeline authored using the TFX DSL,
in a way that inter-operates with other components.

This RFC assumes some clarification on the TFX’s use of artifacts and metadata
as explained in [this section of TFX user guide](https://www.tensorflow.org/tfx/guide#artifacts).

## Motivation

A key value proposition provided by
[Kubeflow Pipelines (KFP)](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/)
is letting users orchestrate arbitrary containers as part of a Machine Learning
(ML) pipeline. Many users already have custom ML applications written in
languages other than Python (e.g. in R, Java, C++ etc), and the ability to chain
existing containerized application programs with other pipeline steps through a
Python DSL is valuable. As of 2019/09
([tfx 0.14](https://github.com/tensorflow/tfx/blob/0.14.0rc1//RELEASE.md#version-0140),
even though TFX supports `KubeflowDagRunner` as an orchestrator, the
[TFX DSL](https://github.com/tensorflow/tfx/tree/0.14.0rc1/tfx) does not provide
a mechanism to accomplish this.

## User Benefit

This document proposes a way to define a proper component based on any
containerized program as a solution to address this problem. The proposed
**container-based component** is realized by a simple DSL extension for custom
containers to the TFX DSL. It enables users to easily declare input and output
artifacts of a pipeline step implemented as a custom container-based program, as
well as a prescription for how to invoke the container’s entrypoint while
passing in the relevant artifact metadata and execution parameters.

In doing so, we will not only enable custom containerized application programs
to be used in TFX, but also augment KFP-based pipelines with the following
capabilities:

*   **Metadata-centric interface**: The proposed container-based component
    provides a framework to clearly declare input- and output- signatures of the
    container-based component in terms of
    [strongly typed](https://github.com/google/ml-metadata/blob/ba69ae039bd2205ec2d7b982b3bfdda4718bf8df/ml_metadata/proto/metadata_store.proto#L55)
    [artifact metadata](https://github.com/google/ml-metadata/blob/ba69ae039bd2205ec2d7b982b3bfdda4718bf8df/ml_metadata/proto/metadata_store.proto#L29).
    This is a key value proposition of TFX DSL.

    *   As of [kfp 0.1.26](https://github.com/kubeflow/pipelines/tree/0.1.26),
        the
        [KFP DSL](https://github.com/kubeflow/pipelines/tree/0.1.26/sdk/python/kfp/dsl)
        has type system to input- and output- of a component. However, it is in
        a way that it doesn’t attach semantic meaning to the input- and output-
        variables in a way compatible to ML Metadata, and the way how TFX
        uitilizes it to realize its functionalies.

*   **Metadata-driven component execution**: Metadata-centric interface of
    components enables TFX Driver and Publisher logic to be applied
    consistently, thus enabling caching of component execution as well
    component-specific (meta)data-driven execution decisions

    *   *Example*: ModelValidator can validate models trained with *any*
        component, so long as the produced artifact is of the *type* Model.

*   **Inter-Component communication via ML Metadata store**: This enables higher
    order component-specific driver logic that depends on another component’s
    behavior.

    *   *Example*: Pusher can choose to proceed or halt pushing depending on
        output artifact from ModelValidator.

*   **Ability to share and reuse a wrapped component as drop-in replacement with
    another**: As a result of the artifact centric, strongly typed input- and
    output- signatures, it enables robust sharing and drop-in replacement of
    components, so long as signatures (the list of input- and output- artifact
    *types*) are the same.

Additionally, the proposed container-based component will enable the following
new features in TFX DSL, which already exist In Kubeflow Pipelines:

*   **Ability to use any containerized application program in a pipeline**: The
    proposed container-based component does not preclude any containerized
    application programs from being used as a pipeline step.

*   **Ability to have fine-grained control on the underlying k8s execution
    environment**: The proposed container-based component preserves the user’s
    ability to control underlying Kubernetes runtime configuration for
    containerized program execution.

## Design Proposal

As stated previously, custom containers may be written in arbitrary languages.
The input interface to containers is restricted to the command-line used to
invoke the application, while the output interface is through file I/O (either
STDOUT, STDOUT, or container-local files).

Since container applications may not have access to TFX libraries, they are not
able to (or don’t even wish to) serialize/de-serialize
[ML Metadata](https://github.com/google/ml-metadata/blob/master/g3doc/get_started.md)
representing input- and output- artifacts, which is what defines the interface
to all TFX components[^1]. Hence, this design proposes DSL extensions that allow
users to declare the types of the input and output artifacts, and directly
reference the URIs of these artifacts when invoking their containers, while
retaining the ability to invoke the containerized program in exactly the way it
expects with regards to flags, arguments and environmental variables. The TFX
framework itself will generate the necessary driver-level code so that metadata
is logged and used for caching executions.

[^1]: ML Metadata is also what enables inter-component communication to realize
    artifact driven component-specific behavior (such as ExampleValidator and
    ModelValidator)

In order to make this proposal concrete, let’s consider a motivating example of
creating a simple 2-step pipeline. The first step generates examples, and the
second trains a model based on the previously produced examples. The following
shows some example code for these two steps:

*   `my_example_gen.py`

```python
import argparse
import pandas as pd

def convert_and_save(df, file):
  ...

if __name__ == '__main__':

    parser = argparse.ArgumentParser()

    # URI location of input CSV file
    parser.add_argument('--input_csv_file', ...)
    # URI location of output CSV file
    parser.add_argument('--output_csv_file', ...)
    # Number of CSV columns in the input file
    parser.add_argument('--num_csv_columns', ...)

    args = parser.parse_args()
    arguments = args.__dict__

    df = pd.read_csv(arguments['input_csv_file'],
                     usecols=range(0, int(arguments['num_csv_columns'])))

    # implementation of business logic to ingest dataset
    convert_and_save(df, arguments['output_csv_file'])

```

*   `my_trainer.py`

```python
import argparse
import pandas as pd
import sklearn.linear_model import Logisticregression

def load_dataset(file):
  ...

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    # URI location of input file with pre-processed examples.
    parser.add_argument('--input_examples', ...)
    # URI location to output the trained model
    parser.add_argument('--output_model', ...)
    # Number of training steps to use.
    parser.add_argument('--train_steps', ...)

    args = parser.parse_args()
    arguments = args.__dict__

    x_train, y_train, x_eval, y_eval = load_datset(arguments['input_examples'])

    model = LogisticRegression()
    model.fit(x_train, y_train, args.train_steps)

    #
    # ... perform grid search, etc, ...
    #

{'fraction_non_alphanumeric': 0.08138890599027274, 'fraction_numerical': 0.011820476512959428, 'mean_word_length': 4.055711173358232, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 20, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2491681', 'n_tokens_mistral': 4556, 'n_tokens_neox': 4282, 'n_words': 1828}