<jupyter_start><jupyter_text>Makefile-style pattern-matching rules * **Difficulty level**: intermediate* **Time need to lean**: 20 minutes or less* **Key points**:  * Option `provides` extends the "data-flow" style workflow that allows steps to generate different outputs.  * Steps with `provides` section option has default `step_output`. `input` can be derived from pattern-matched variables.  Auxiliary steps Auxiliary steps are special steps that are executed to provide [targets](sos_actions.html) that are required by others.For example, when the following step is executed with an input file `bamfile` (with extension `.bam`), it checks the existence of input file (`bamfile`), and a dependent index file (with extension `.bam.bai`).```sos[100 (call variant)]input:   bamfiledepends: bamfile + '.bai'run:     commands to call variants from      input bam file```Because the step depends on an index file, SoS will look in the script for a step that provides such a target, which would be similar to```sos[index_bam : provides='{sample}.bam.bai']input: f"{sample}.bam"run: expand=True     samtools index {_input}```Such a step is characterized by a **`provides`** option (or a step with simple `output` statement and is called an auxiliary step. In this particular case, if `bamfile="AS123.bam"`, the requested dependent file would be `AS123.bam.bai`. Through the matching mechanism of option `provides`, the `index_bam` step would be executed with variables `sample="AS123"` and `step_output="AS123.bam.bai"`. Unless option `-T` (see [tracing dependency](trace_dependency.html) for details) is specified, **SoS will not check if an depdendent target can be generated by an auxiliary step if it already exists**. In an extreme case, `sos run -t filename` will quit directly if `filename` already exists.  An auxiliary step can trigger other auxiliary steps and form a DAG (Directed Acyclic Graph). Acutually, you can write workflows in a make-file style with all auxiliary steps and execute workflows defined by targets. If you are familiar with Makefile, especially [snakemake](https://bitbucket.org/johanneskoester/snakemake), it can be natural for you to implement your workflow in this style. The advantage of SoS is that **you can use either or both forward-style and makefile-style steps to define your workflow** and take advantages of both approaches.  Step option `provides` An auxiliary step is defined by the `provides` option in section head, in the format of```sos[step_name : provides=target]```where `target` can be* A filename or file pattern such as `"{sample}.bam.idx"`* Other types of targets such as `executable("ms")`* A list (sequence) of one or more file patterns and targets.  File Pattern A file pattern is a filename with optional patterns with variable names enbraced in `{ }`. SoS matches filenames with the patterns and, if successful, assign variables with matched parts of the names. For example,```[compress: provides = '{filename}.bam']```would be triggered with target `sample_A.bam` and `sample_B.bam`. When the step is triggered by `sample_A.bam`, it defines variable `filename` as `sample_A` and sets the output of the step as `sample_A.bam`.The following example removes all local `*.bam` and `*.bam.bi` file before it executes three workflows defined by `targets`. We use magic `%run` to execute it, which is equivalent to executing it from command line using commands such as```bash    sos run myscript -t TS1.bam``` Let us create a workflow with two auxiliary steps `compress` and `index`. The `compress` step generates a `.bam` file (no input here for simplicity) and `index` step creates a `.bam.bai` file from the `.bam` file.<jupyter_code>%save test_provides.sos -f

[compress: provides = '{filename}.bam']
print(f"> {step_name} input to {_output}")
sh: expand=True
    touch {_output}

[index: provides = '{filename}.bam.bai']
input: f"{filename}.bam"
print(f"> {step_name} {_input} to {_output}")
sh: expand=True
    touch {_output}<jupyter_output><empty_output><jupyter_text>If we only want to generate a `bam` file (with option `-t TS1.bam`), the `compress` step is executed<jupyter_code>!rm -f TS1.bam TS1.bam.bai 

%runfile test_provides -t TS1.bam -v1<jupyter_output>> compress input to TS1.bam<jupyter_text>If we would like to generate both `.bam` and `.bam.bai` files (with option `-t TS2.bam.bai`), both steps are executed.<jupyter_code>!rm -f TS2.bam TS2.bam.bai

%runfile test_provides -t TS2.bam.bai -v1<jupyter_output>> compress input to TS2.bam
> index TS2.bam to TS2.bam.bai<jupyter_text>As you can see from the output, when the first workflow is executed with target `TS1.bam`, step `compress` is executed to produce it. In the run, both steps are executed to generate `TS2.bam` and then `TS2.bam.bai`.  Non-file targets In addition to output files, an auxiliary step can provide targets of other types. A most widely used target is `sos_variable`, which provides variables that can be accessed by later steps. For example,<jupyter_code>%run -v1

# this step provides variable `numNotebooks`
[count: provides=sos_variable('numNotebooks')]
import glob
numNotebooks = len(glob.glob('*.ipynb'))

[default]
depends: sos_variable('numNotebooks')
print(f"There are {numNotebooks} notebooks in this directory")<jupyter_output>There are 94 notebooks in this directory<jupyter_text>However, for this particular example, it is more straightforward to return the variable with [option `shared`](shared_variables.html)  as follows:<jupyter_code>%run -v1

# this step provides variable `numNotebooks`
[count: shared='numNotebooks']
import glob
numNotebooks = len(glob.glob('*.ipynb'))

[default]
depends: sos_variable('numNotebooks')
print(f"There are {numNotebooks} notebooks in this directory")<jupyter_output>There are 94 notebooks in this directory<jupyter_text>Multiple targets You can specify multiple targets to the `provides` option. A step would be triggered if any of the targets matches. For example, the `temp` step is triggered twice in the following example, first time by target `text.bak` and the second time by target `text.tmp`.<jupyter_code>!rm -f text.bak text.tmp

%run -v1
[temp: provides = ['{filename}.bak', '{filename}.tmp']]
print(f"Touch {_output}")
sh: expand=True
    touch {_output}

[default]
depends: 'text.bak', 'text.tmp'<jupyter_output>Touch text.tmp
Touch text.bak<jupyter_text>However, depending on what the auxiliary step is designed, it might be generating multiple output at the same time and it would be wasteful to execute the step multiple times. In this case, you can define an `output` statement and let SoS know that the execution of the step generates multiple targets.<jupyter_code>!rm -f text.bak text.tmp

%run -v1
[temp: provides = ['{filename}.bak', '{filename}.tmp']]
output: f'{filename}.bak', f'{filename}.tmp'

print(f"Touch {_output}")
sh: expand=True
    touch {_output}

[default]
depends: 'text.bak', 'text.tmp'<jupyter_output>Touch text.bak text.tmp
{'fraction_non_alphanumeric': 0.10193826274228285, 'fraction_numerical': 0.006173725771715721, 'mean_word_length': 5.309782608695652, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '284941', 'license': 'MIT', 'n_tokens_mistral': 2149, 'n_tokens_neox': 2075, 'n_words': 884, 'path': 'src/user_guide/auxiliary_steps.ipynb', 'repo_name': 'sosdocs/sos-docs', 'size': '6965'}