<commit_before># KFP-Tekton Compiler Features

This document describes the features supported by the _KFP-Tekton_ compiler.
With the current implementation, the _KFP-Tekton_ compiler is capable of
[compiling 80%](/sdk/python/tests/README.md) of approximately 90 sample
and test pipelines found in the KFP repository.

<!-- # Table of Contents -->

- [Pipeline DSL Features with Native Tekton Implementation](#pipeline-dsl-features-with-native-tekton-implementation)
    - [pod_annotations and pod_labels](#pod_annotations-and-pod_labels)
    - [Retries](#retries)
    - [Volumes](#volumes)
    - [Timeout for Tasks and Pipelines](#timeout-for-tasks-and-pipelines)
    - [RunAfter](#runafter)
    - [Input Parameters](#input-parameters)
    - [ContainerOp](#containerop)
    - [Affinity, Node Selector, and Tolerations](#affinity-node-selector-and-tolerations)
    - [ImagePullSecrets](#imagepullsecrets)
    - [Exit Handler](#exit-handler)
    - [Pipeline Loops](#pipeline-loops)
    - [Any Sequencer](#any-sequencer)
    - [Tekton Pipeline Variables](#tekton-pipeline-variables)
- [Pipeline DSL Features with a Custom Tekton Implementation](#pipeline-dsl-features-with-a-custom-tekton-implementation)
  - [Features with the Same Behavior as Argo](#features-with-the-same-behavior-as-argo)
    - [InitContainers](#initcontainers)
    - [Conditions](#conditions)
    - [ResourceOp, VolumeOp, and VolumeSnapshotOp](#resourceop-volumeop-and-volumesnapshotop)
    - [Output Parameters](#output-parameters)
    - [Input Artifacts](#input-artifacts)
    - [Output Artifacts](#output-artifacts)
  - [Features with Limitations](#features-with-limitations)
    - [Variable Substitutions](#variable-substitutions)
  - [Features with a Different Behavior than Argo](#features-with-a-different-behavior-than-argo)
    - [Sidecars](#sidecars)


# Pipeline DSL Features with Native Tekton Implementation

Below are the features using Tekton's native support without any custom workaround.

### pod_annotations and pod_labels

`pod_annotations` and `pod_labels` are for assigning custom annotations or labels to a pipeline component. They are implemented with
Tekton's [task metadata](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#configuring-a-task) field under Tekton
Task. The [pipeline transformers](/sdk/python/tests/compiler/testdata/pipeline_transformers.py) example shows how to
 apply
custom annotations and labels to one or more components in the pipeline.

### Retries

Retry feature allows users to specify the number of times a particular component should retry its execution when it fails. It is
implemented with Tekton's [task spec](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-retries-parameter)
under Tekton Pipeline. The [retry](/sdk/python/tests/compiler/testdata/retry.py) python test is an example of how to use
this feature.

### Volumes

Volumes are for mounting existing Kubernetes resources onto the components. It is implemented with Tekton's
[task volumes](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#specifying-volumes) feature under Tekton Task. The
[volume](/sdk/python/tests/compiler/testdata/volume.py) python test is an example of how to use this feature.

### Timeout for Tasks and Pipelines

Timeout can be used for setting the amount of time allowed on executing a component within the Pipeline or setting the amount of
time allowed on executing the whole pipeline. By default, the generated pipeline won't be timeout to simulate the same behavior
as Argo, but users can explicitly assign a timeout period on the task or pipeline level. The task timeout is implemented with Tekton's
[task failure timeout](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#configuring-the-failure-timeout) under
Tekton Pipeline, and pipeline timeout is implemented with Tekton's
[pipeline failure timeout](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#configuring-a-failure-timeout)
under Tekton PipelineRun. The [timeout](/sdk/python/tests/compiler/testdata/timeout.py) python test is an example of
how to use this feature.

If you want to use the
[Tekton global default timeout value](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#configuring-a-failure-timeout)
for the generated pipeline, you can run `export TEKTON_GLOBAL_DEFAULT_TIMEOUT=true` to enable this feature.


### RunAfter

RunAfter is for indicating that a component must execute after one or more other components. It is implemented with Tekton's
[runAfter](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-runafter-parameter) feature under Tekton
Pipeline. The [sequential](/sdk/python/tests/compiler/testdata/sequential.py) python test is an example of how to use this
feature.

### Input Parameters

Input Parameters are for passing pipeline parameters or other component outputs to the next running component. It is implemented
with Tekton's [parameters](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#specifying-parameters) features under Tekton
task. The [parallel_join](/sdk/python/tests/compiler/testdata/parallel_join.py) python test is an example of how to use this
feature.

### ContainerOp

ContainerOp defines the container spec for a pipeline component. It is implemented with Tekton's
[steps](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#defining-steps) features under Tekton task. The generated
Tekton task name is the same as the containerOp name whereas the step name is always called "main". The
[sequential](/sdk/python/tests/compiler/testdata/sequential.py) python test is an example of how to use this feature.

### Affinity, Node Selector, and Tolerations

Affinity, Node Selector, and Tolerations are Kubernetes spec for selecting which node should run the component based on
user-defined constraints. They are implemented with Tekton's [PipelineRunTaskSpec](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#specifying-task-run-specs)
features under Tekton PipelineRun.
The [affinity](/sdk/python/tests/compiler/testdata/affinity.py),
[node_selector](/sdk/python/tests/compiler/testdata/node_selector.py), and
[tolerations](/sdk/python/tests/compiler/testdata/tolerations.py) Python tests are examples of how to use these features.

This feature has been implemented in Tekton version `0.13.0`.

### ImagePullSecrets

ImagePullSecret is a feature for the components to know which secret to use when pulling container images from private registries. It is implemented
with Tekton's [podTemplate](https://github.com/tektoncd/pipeline/blob/master/docs/podtemplates.md) field under Tekton
PipelineRun. The [imagepullsecrets](/sdk/python/tests/compiler/testdata/imagepullsecrets.py) Python test is an example of how to use this
feature.

This feature has been available since Tekton version `0.13.0`.

### Exit Handler

An _exit handler_ is a component that always executes, irrespective of success or failure,
at the end of the pipeline. It is implemented using Tekton's
[finally](https://github.com/tektoncd/pipeline/blob/v0.16.0/docs/pipelines.md#adding-finally-to-the-pipeline)
section under the Pipeline `spec`. An example of how to use an _exit handler_ can be found in
the [exit_handler](/sdk/python/tests/compiler/testdata/exit_handler.py) compiler test.

The `finally` syntax is supported since Tekton version `0.14.0`.

### Pipeline Loops

PipelineLoops is a feature for running the same component tasks multiple times. right now Tekton support the loop pipeline/tasks via custom tasks. The controller we named as "PipelineLoop", about the usage of the controller, you can refer to the examples [here](/tekton-catalog/pipeline-loops/examples).

To use this feature, please ensure Tekton version >= v0.19, and "data.enable-custom-tasks" is "true" in feature-flags configmap:
`kubectl edit cm feature-flags -n tekton-pipelines`

For the python sdk to use this feature, refer to the examples:
[loop_static](/sdk/python/tests/compiler/testdata/loop_static.py), [withparam_global](/sdk/python/tests/compiler/testdata/withparam_global.py), [withitem_nested](/sdk/python/tests/compiler/testdata/withitem_nested.py)

### Any Sequencer

When any one of the task dependencies complete successfully, the dependent task will be started. Order of execution of the dependencies doesn’t matter, and the pipeline doesn't wait for all the task dependencies to complete before moving to the next step. Please follow the details of the implementation in the [design doc](https://docs.google.com/document/d/1oXOdiItI4GbEe_qzyBmMAqfLBjfYX1nM94WHY3EPa94/edit#heading=h.dt8bhna4spym). 

For example:

```
from kfp_tekton.compiler.any_sequencer import after_any

dsl.ContainerOp(
  ...
).apply(after_any(([containerOps]))
```

Please note that the service account of the `Any Sequencer` needs 'get' premission to watch the status of the sepecified `taskRun`.

### Tekton Pipeline Variables

Some [Tekton Pipeline Variables](https://github.com/tektoncd/pipeline/blob/master/docs/variables.md#variables-available-in-a-pipeline) that cannot be used in ContainerOps directly, and must be passed by `params`. The _KFP-Tekton_ compiler supports user using the variables in ContainerOps, the relative params will be added automatically when compiling. See the [example](python/tests/compiler/testdata/tekton_pipeline_variables.py).
Supported variables list:
```
$(context.pipeline.name)
$(context.pipelineRun.name)
$(context.pipelineRun.namespace)
$(context.pipelineRun.uid)
```
<commit_msg>fix references and syntax for Loops (#445)<commit_after># KFP-Tekton Compiler Features

This document describes the features supported by the _KFP-Tekton_ compiler.
With the current implementation, the _KFP-Tekton_ compiler is capable of
[compiling 80%](/sdk/python/tests/README.md) of approximately 90 sample
and test pipelines found in the KFP repository.

<!-- # Table of Contents -->

- [Pipeline DSL Features with Native Tekton Implementation](#pipeline-dsl-features-with-native-tekton-implementation)
    - [pod_annotations and pod_labels](#pod_annotations-and-pod_labels)
    - [Retries](#retries)
    - [Volumes](#volumes)
    - [Timeout for Tasks and Pipelines](#timeout-for-tasks-and-pipelines)
    - [RunAfter](#runafter)
    - [Input Parameters](#input-parameters)
    - [ContainerOp](#containerop)
    - [Affinity, Node Selector, and Tolerations](#affinity-node-selector-and-tolerations)
    - [ImagePullSecrets](#imagepullsecrets)
    - [Exit Handler](#exit-handler)
    - [Pipeline Loops](#pipeline-loops)
    - [Any Sequencer](#any-sequencer)
    - [Tekton Pipeline Variables](#tekton-pipeline-variables)
- [Pipeline DSL Features with a Custom Tekton Implementation](#pipeline-dsl-features-with-a-custom-tekton-implementation)
  - [Features with the Same Behavior as Argo](#features-with-the-same-behavior-as-argo)
    - [InitContainers](#initcontainers)
    - [Conditions](#conditions)
    - [ResourceOp, VolumeOp, and VolumeSnapshotOp](#resourceop-volumeop-and-volumesnapshotop)
    - [Output Parameters](#output-parameters)
    - [Input Artifacts](#input-artifacts)
    - [Output Artifacts](#output-artifacts)
  - [Features with Limitations](#features-with-limitations)
    - [Variable Substitutions](#variable-substitutions)
  - [Features with a Different Behavior than Argo](#features-with-a-different-behavior-than-argo)
    - [Sidecars](#sidecars)


# Pipeline DSL Features with Native Tekton Implementation

Below are the features using Tekton's native support without any custom workaround.

### pod_annotations and pod_labels

`pod_annotations` and `pod_labels` are for assigning custom annotations or labels to a pipeline component. They are implemented with
Tekton's [task metadata](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#configuring-a-task) field under Tekton
Task. The [pipeline transformers](/sdk/python/tests/compiler/testdata/pipeline_transformers.py) example shows how to
 apply
custom annotations and labels to one or more components in the pipeline.

### Retries

Retry feature allows users to specify the number of times a particular component should retry its execution when it fails. It is
implemented with Tekton's [task spec](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-retries-parameter)
under Tekton Pipeline. The [retry](/sdk/python/tests/compiler/testdata/retry.py) python test is an example of how to use
this feature.

### Volumes

Volumes are for mounting existing Kubernetes resources onto the components. It is implemented with Tekton's
[task volumes](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#specifying-volumes) feature under Tekton Task. The
[volume](/sdk/python/tests/compiler/testdata/volume.py) python test is an example of how to use this feature.

### Timeout for Tasks and Pipelines

Timeout can be used for setting the amount of time allowed on executing a component within the Pipeline or setting the amount of
time allowed on executing the whole pipeline. By default, the generated pipeline won't be timeout to simulate the same behavior
as Argo, but users can explicitly assign a timeout period on the task or pipeline level. The task timeout is implemented with Tekton's
[task failure timeout](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#configuring-the-failure-timeout) under
Tekton Pipeline, and pipeline timeout is implemented with Tekton's
[pipeline failure timeout](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#configuring-a-failure-timeout)
under Tekton PipelineRun. The [timeout](/sdk/python/tests/compiler/testdata/timeout.py) python test is an example of
how to use this feature.

If you want to use the
[Tekton global default timeout value](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#configuring-a-failure-timeout)
for the generated pipeline, you can run `export TEKTON_GLOBAL_DEFAULT_TIMEOUT=true` to enable this feature.


### RunAfter

RunAfter is for indicating that a component must execute after one or more other components. It is implemented with Tekton's
[runAfter](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-runafter-parameter) feature under Tekton
Pipeline. The [sequential](/sdk/python/tests/compiler/testdata/sequential.py) python test is an example of how to use this
feature.

### Input Parameters

Input Parameters are for passing pipeline parameters or other component outputs to the next running component. It is implemented
with Tekton's [parameters](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#specifying-parameters) features under Tekton
task. The [parallel_join](/sdk/python/tests/compiler/testdata/parallel_join.py) python test is an example of how to use this
feature.

### ContainerOp

ContainerOp defines the container spec for a pipeline component. It is implemented with Tekton's
[steps](https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#defining-steps) features under Tekton task. The generated
Tekton task name is the same as the containerOp name whereas the step name is always called "main". The
[sequential](/sdk/python/tests/compiler/testdata/sequential.py) python test is an example of how to use this feature.

### Affinity, Node Selector, and Tolerations

Affinity, Node Selector, and Tolerations are Kubernetes spec for selecting which node should run the component based on
user-defined constraints. They are implemented with Tekton's [PipelineRunTaskSpec](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#specifying-task-run-specs)
features under Tekton PipelineRun.
The [affinity](/sdk/python/tests/compiler/testdata/affinity.py),
[node_selector](/sdk/python/tests/compiler/testdata/node_selector.py), and
[tolerations](/sdk/python/tests/compiler/testdata/tolerations.py) Python tests are examples of how to use these features.

This feature has been implemented in Tekton version `0.13.0`.

### ImagePullSecrets

ImagePullSecret is a feature for the components to know which secret to use when pulling container images from private registries. It is implemented
with Tekton's [podTemplate](https://github.com/tektoncd/pipeline/blob/master/docs/podtemplates.md) field under Tekton
PipelineRun. The [imagepullsecrets](/sdk/python/tests/compiler/testdata/imagepullsecrets.py) Python test is an example of how to use this
feature.

This feature has been available since Tekton version `0.13.0`.

### Exit Handler

An _exit handler_ is a component that always executes, irrespective of success or failure,
at the end of the pipeline. It is implemented using Tekton's
[finally](https://github.com/tektoncd/pipeline/blob/v0.16.0/docs/pipelines.md#adding-finally-to-the-pipeline)
section under the Pipeline `spec`. An example of how to use an _exit handler_ can be found in
the [exit_handler](/sdk/python/tests/compiler/testdata/exit_handler.py) compiler test.

The `finally` syntax is supported since Tekton version `0.14.0`.

### Pipeline Loops

PipelineLoops is a feature for running a component or a set of component tasks multiple times in a loop. Right now Tekton supports loop pipeline/tasks via an implementation of [Tekton Custom Tasks](https://github.com/tektoncd/community/blob/master/teps/0002-custom-tasks.md) controller named as "PipelineLoop". Please refer to the examples [here](/tekton-catalog/pipeline-loops/examples) to understand more details about the usage of loops. 

To use this feature, please ensure Tekton version >= v0.19, and "data.enable-custom-tasks" is "true" in feature-flags configmap:
`kubectl edit cm feature-flags -n tekton-pipelines`

To see how the Python SDK provides this feature, refer to the examples below:
- [loop_static](/sdk/python/tests/compiler/testdata/loop_static.py)
- [withparam_global](/sdk/python/tests/compiler/testdata/withparam_global.py)
- [withitem_nested](/sdk/python/tests/compiler/testdata/withitem_nested.py)

### Any Sequencer

When any one of the task dependencies complete successfully, the dependent task will be started. Order of execution of the dependencies doesn’t matter, and the pipeline doesn't wait for all the task dependencies to complete before moving to the next step. Please follow the details of the implementation in the [design doc](https://docs.google.com/document/d/1oXOdiItI4GbEe_qzyBmMAqfLBjfYX1nM94WHY3EPa94/edit#heading=h.dt8bhna4spym). 

For example:

```
from kfp_tekton.compiler.any_sequencer import after_any

dsl.ContainerOp(
  ...
).apply(after_any(([containerOps]))
```

Please note that the service account of the `Any Sequencer` needs 'get' premission to watch the status of the sepecified `taskRun`.

### Tekton Pipeline Variables

Some [Tekton Pipeline Variables](https://github.com/tektoncd/pipeline/blob/master/docs/variables.md#variables-available-in-a-pipeline) that cannot be used in ContainerOps directly, and must be passed by `params`. The _KFP-Tekton_ compiler supports user using the variables in ContainerOps, the relative params will be added automatically when compiling. See the [example](python/tests/compiler/testdata/tekton_pipeline_variables.py).
Supported variables list:
```
$(context.pipeline.name)
$(context.pipelineRun.name)
$(context.pipelineRun.namespace)
$(context.pipelineRun.uid)
```

{'fraction_non_alphanumeric': 0.0949007033512619, 'fraction_numerical': 0.003775341332230037, 'mean_word_length': 5.170070197830249, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 29, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2045212', 'n_tokens_mistral': 5660, 'n_tokens_neox': 5473, 'n_words': 1832}