<jupyter_start><jupyter_text>How to define and execute multiple workflows in a single SoS script * **Difficulty level**: easy* **Time need to lean**: 10 minutes or less* **Key points**:  * A SoS script can contain multiple workflows  * A SoS section can serve in multiple workflows  * Multiple workflows can be concatenated to be executed together     Workflow Specification   Forward-style workflows  A SoS forward-style workflow has a name and one or more numbered steps. The workflows are defined from sections in a SoS script. For example, the following sections specify a workflow with four steps `5`, `10`, `20`, and `100`. The workflow steps can be specified in any order and do not have to be consecutive.<jupyter_code>[5]
[20]
[10]
[100]<jupyter_output><empty_output><jupyter_text>A workflow specified in this way is the **`default`** workflow and is actually called `default` in SoS output. You can specify a workflow with name and give each step a short description as follows:<jupyter_code>[mapping_5 (get data)]
[mapping_20 (align)]
[mapping_10 (quality control)]
[mapping_100 (generate report)]<jupyter_output><empty_output><jupyter_text>A SoS script can define multiple workflows. For example, the following sections of SoS script defines two workflows named ``mouse`` and ``human``.<jupyter_code>[mouse_10]
[mouse_20]
[mouse_30]
[human_10]
[human_20]
[human_30]<jupyter_output><empty_output><jupyter_text>In this case, a command line option is needed to specify workflow name. This can be done by magic `%run` in Jupyter notebook, or a positional argument from the command line, e.g.```    % sos run myscript mouse``` Note that the workflow argument is not needed if a `default` workflow is defined in the script like the following example<jupyter_code>[10]
[20]
[30]
[test_10]
[test_20]
[test_30]<jupyter_output><empty_output><jupyter_text>Multiple steps can share a single step as follows<jupyter_code>[mouse_10,human_10]
[mouse_20]
[human_20]
[mouse_30,human_30]<jupyter_output><empty_output><jupyter_text>and wildcard steps can be used to define a step for multiple workflows:<jupyter_code>%run mouse
[*_10]
[mouse_20]
[human_20]
[*_30]<jupyter_output><empty_output><jupyter_text>If the steps defined in a shared section is similar but not identical, it can use step variable (discussed elsewhere) `step_name` to behave differently in different workflows. In the following example, the variable `step_name` will be `mouse_20` or `human_20` depending on the workflow being executed, and is used to determine the correct reference genome for different workflows.<jupyter_code>[mouse_20,human_20]
reference = "/path/to/mouse/reference" if \
  step_name.startswith('mouse') else "/path/to/human/reference"

print(f"Reference genome {reference} is used")<jupyter_output><empty_output><jupyter_text>Sub- and combined workflows  Although workflows are defined separately with all their steps, they do not have to be executed in their entirety. A `subworkflow` refers to a workflow that is defined from one or more steps of an existing workflows. It is specified using syntax `workflow:[from-to]` where `from-to` can be `n` (step `n`), `-n` (up to `n`), `n-m` (step `n` to `m`) and `m-` (from `m`). For example ```pythonA               complete workflow AA:5-10          step 5 to 10 of AA:50-           step 50 upA:-10           up to step 10 of AA:10            step 10 of workflow A``` In practice, the `-n` format is frequently used to execute part of the workflow for debudding purposes, for example:<jupyter_code>%run default:-20
[10]
print(step_name)
[20]
print(step_name)
[30]
print(step_name)<jupyter_output>default_10
default_20<jupyter_text>You can also combine subworkflows to execute multiple workflows one after another. For example, ```pythonA + B           workflow A, followed by BA:0 + B         step 0 of A, followed by BA:-50 + B + C   up to step 50 of workflow A, followed by B, and C``` This syntax can be used from the command line, e.g.```bashsos-runner myscript align+call```or from the `%run` magic of Jupyter notebook<jupyter_code>%preview -n multi.dot
%run check+align+call -d multi.dot
[check_10]
print(step_name)
[align_10]
print(step_name)
[align_20]
print(step_name)
[call_10]
print(step_name)
[call_20]
print(step_name)<jupyter_output>check_10
align_10
align_20
call_10
call_20
{'fraction_non_alphanumeric': 0.09324386857936141, 'fraction_numerical': 0.031004164738546967, 'mean_word_length': 5.310948905109489, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '627526', 'license': 'MIT', 'n_tokens_mistral': 1455, 'n_tokens_neox': 1328, 'n_words': 536, 'path': 'src/user_guide/multi_workflow.ipynb', 'repo_name': 'andrewrocks/sos-docs', 'size': '4322'}