<commit_before>## Pipeline 

- [Introduction](#introduction)
- [Pipeline format](#format)
- [Pipeline process state](#state)
- [Pipeline Lifecycle](#lifecycle)
- [Shared workflow](#shared)

<a name="introduction"></a>
### Introduction

**[Pipeline](../../model/pipeline.go)** an abstraction defining a simple sequence of tasks.

A task can either execute a [workflow](../workflow) or [service](../service) action.

For instance the following pipeline will execute SSH command (service: exec, action: run).

```bash
endly -r=run
```

@run.yaml
```yaml
pipeline:
  action: exec:run
  target:
    url:  ssh://127.0.0.1/
    credentials: ${env.HOME}/.secret/localhost.json
  commands:
    - mkdir -p /tmp/app/build 
    - chown ${os.user} /tmp/app/build 
```


<name a="format"></a>
### Pipeline format
Pipeline run request can use either JSON or YAML.


The general pipeline syntax: 

@xxx.yaml
```yaml
params:
  k1:v1
init: var/@init
defaults:
  d1:v1

pipeline:
  node1:
     action: serviceID:action
     requestField1: val1
     requestFieldN: valN
           
  nodeX:
    subNodeX:
      workflow: workflowSelector
      tasks: task selector
      paramsKey1: val1
      paramKeyN: valN
      
    subNodeZ:
       action: serviceID:action
       requestField1: val1
       requestFieldN: valN
      

post: 
  - age = $response.user.age

```
Pipeline execution node is determined by presence of either action or workflow attribute, otherwise
any sub node organization is allowed i.e

```yaml
pipeline:
  service:
    mysql:<commit_msg>updated doc<commit_after>## Pipeline 

- [Introduction](#introduction)
- [Pipeline format](#format)
- [Pipeline process state](#state)
- [Pipeline Lifecycle](#lifecycle)
- [Shared workflow](#shared)

<a name="introduction"></a>
### Introduction

**[Pipeline](../../model/pipeline.go)** an abstraction defining a simple sequence of tasks.

A task can either execute a [workflow](../workflow) or [service](../service) action.

For instance the following pipeline will execute SSH command (service: exec, action: run).

```bash
endly -r=run
```

@run.yaml
```yaml
pipeline:
  action: exec:run
  target:
    url:  ssh://127.0.0.1/
    credentials: ${env.HOME}/.secret/localhost.json
  commands:
    - mkdir -p /tmp/app/build 
    - chown ${os.user} /tmp/app/build 
```


<name a="format"></a>
### Pipeline format
Pipeline run request can use either JSON or YAML.


The general pipeline syntax: 

@xxx.yaml
```yaml
params:
  k1:v1
init: var/@init
defaults:
  d1:v1

pipeline:
  node1:
     action: serviceID:action
     requestField1: val1
     requestFieldN: valN
           
  nodeX:
    subNodeX:
      workflow: workflowSelector
      tasks: task selector
      paramsKey1: val1
      paramKeyN: valN
      
    subNodeZ:
       action: serviceID:action
       requestField1: val1
       requestFieldN: valN
      

post: 
  - age = $response.user.age

```
Pipeline execution node is determined by presence of either **_action_** or **_workflow_** attribute, otherwise
any sub node organization is allowed i.e

```yaml
pipeline:
  service:
    mysql:
{'fraction_non_alphanumeric': 0.12823834196891193, 'fraction_numerical': 0.011010362694300517, 'mean_word_length': 3.1970108695652173, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1320472', 'n_tokens_mistral': 1018, 'n_tokens_neox': 962, 'n_words': 292}