<commit_before># Cyclone Design

Cyclone is a powerful workflow engine and end-to-end pipeline solution implemented with native Kubernetes resources. 
Cyclone is architectured with a low-level workflow engine that is application agnostic, offering capabilities like
workflow DAG scheduling, resource lifecycle management.

## Architecture

![Cyclone arch](images/cyclone-arch.png)

### Components

* Cyclone Web: UI of Cyclone helps users to easily use Cyclone.
* Cyclone Server: API server of Cyclone provides a set of RESTful APIs to operate Cyclone resources.
* Workflow Controller: Controller of Cyclone workflow engine, controls the execution of stages in workflows.

### Kubernetes CRDs

* Stage: Minimum executable unit for workflow, it does some work on input resources and output results.
* Resource: Resource is the data used by stages as input or output, such as git repository or docker images.
* Workflow: Executable DAG graph composed of stages.
* WorkflowRun: Running record of a workflow.
* WorkflowTrigger: Auto-trigger policy for workflow.

## Implementation

### Workflow Engine

Workflow controller is the main component of the workflow engine. It watches various kinds of Cyclone CRDs, and controls the execution of workflows.

Workflow controller creates a pod for each stage to execute, and there are 3 kinds of containers in the pod:

* Init Containers: Init containers will prepare input resources for the stage before workload containers start.
* Workload Containers: Workload containers are functional containers, which do the main work for stages.<commit_msg>docs(design): add functional modules (#793)<commit_after># Cyclone Design

Cyclone is a powerful workflow engine and end-to-end pipeline solution implemented with native Kubernetes resources. 
Cyclone is architectured with a low-level workflow engine that is application agnostic, offering capabilities like
workflow DAG scheduling, resource lifecycle management.

## Architecture

![Cyclone arch](images/cyclone-arch.png)

### Components

* Cyclone Web: UI of Cyclone helps users to easily use Cyclone.
* Cyclone Server: API server of Cyclone provides a set of RESTful APIs to operate Cyclone resources.
* Workflow Controller: Controller of Cyclone workflow engine, controls the execution of stages in workflows.

### Kubernetes CRDs

* Stage: Minimum executable unit for workflow, it does some work on input resources and output results.
* Resource: Resource is the data used by stages as input or output, such as git repository or docker images.
* Workflow: Executable DAG graph composed of stages.
* WorkflowRun: Running record of a workflow.
* WorkflowTrigger: Auto-trigger policy for workflow.

## Functional Modules

There are 3 main functional modules in Cyclone:

* Integration Center: Integrates external systems like SCM, docker registry, S3.
* Projects: Manages a group of workflows and their shared configs. 
* Stage Templates: Manages built-in stage templates and customized templates.

## Implementation

### Workflow Engine

Workflow controller is the main component of the workflow engine. It watches various kinds of Cyclone CRDs, and controls the execution of workflows.

Workflow controller creates a pod for each stage to execute, and there are 3 kinds of containers in the pod:

* Init Containers: Init containers will prepare input resources for the stage before workload containers start.
* Workload Containers: Workload containers are functional containers, which do the main work for stages.
{'fraction_non_alphanumeric': 0.04708904109589041, 'fraction_numerical': 0.001997716894977169, 'mean_word_length': 5.043103448275862, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6149376', 'n_tokens_mistral': 864, 'n_tokens_neox': 804, 'n_words': 470}