<jupyter_start><jupyter_text>Execution of SoS workflows The content of this chapter is largely applicable to batch mode but we represent it in Jupyter notebook for easy reproduction of results. If you are not familiar with Jupyter notebook, please refer to chapter [Notebook Interface](../documentation/Notebook_Interface.html) for details. For the impatience, the magics* `%sandgox` execute the cell in a temporary directory* `!cmd` execute shell command `cmd`* `%run` run the cell as if in command line with specified options* `%rerun` rerun the last executed script (cell content without magics)* `%set` set options that would be included in each `$run` command  logging level SoS uses a logging system to output all sorts of information during the execution of workflows. The amount of output can be controlled by logging level, which can be `error` (0), `warning` (1), `info` (2), `debug` (3), and `trace` (4). The default logging level for SoS is `info` in batch mode and `warning` in interactive mode.For example, logging at `info` level would produce message indicating the steps executed and input output files, but nothing but warning and errors at the `warning` level.<jupyter_code>%sandbox
!touch a.txt

%run -v3
[10]
[20]
input: 'a.txt'
[30]
[40]
%sandbox
!touch a.txt

%run -v1
[10]
[20]
input: 'a.txt'
[30]
[40]<jupyter_output><empty_output><jupyter_text>`dryrun` mode The `dryrun` mode is used to check for errors of SoS workflows without actually executing any of the actions and tasks. It can be specified with option `-n` to command `run` (`sos run -n`, `sos-runner -n`) or with subcommand `dryrun` (`sos dryrun`).  Checking syntax error The first use of dryrun mode is to check syntax error of scripts. For example, the following script yields an error because the header of the step is wrong (should use `:` instead of `,`).<jupyter_code>%sandbox --expect-error
%run -n
[10, skip=False]
sh:
   echo "I am command echo"<jupyter_output>File contains parsing errors: 
	[line  2]: [10, skip=False]
sh:
   echo "I am command echo"
Invalid statements: SyntaxError('invalid syntax', ('<string>', 1, 10, '[10, skip=False]\n'))<jupyter_text>Print instead of execute scripts Script-executing actions such as `run`, `sh`, and `python` does not execute the scripts in dryrun mode. In stead, they will print the script with the command to execute them. This usage allows you to check if the scripts are interpolated correctly, or even executing them outside of SoS.For example, the following script will print the expanded version of the scripts to be executed:<jupyter_code>!rm -f test*.txt
%run -n

[10]
input: for_each={'i': (1, 2)}
output: f'test_{i}.txt'
sh: expand=True
  sleep {100*i}
  echo "Generating {_output}"
  touch {_output}<jupyter_output><empty_output><jupyter_text>The output files does not exist after the execution of the workflow.<jupyter_code>!ls test*.txt<jupyter_output>ls: test*.txt: No such file or directory<jupyter_text>It is worth noting, however, that sos executes Python statements in steps as usual so your workflow will appear to be executed normally if you do not use actions.For example, if you implement your steps in Python and include them directly, the step will be executed<jupyter_code>!rm -f test*.txt
%run -n

[10]
input: for_each={'i': (1, 2)}
output: f'test_{i}.txt'
print("Generating {_output}")
_output.touch()<jupyter_output>Generating {_output}
Generating {_output}<jupyter_text>and the output files will be generated:<jupyter_code>!ls test*.txt<jupyter_output>test_1.txt test_2.txt<jupyter_text>Generation of placeholder files Another consequence of running workflows in dryrun mode is that SoS generates empty placeholder files during dryrun and remove them afterwards. This allows the workflows to be executed "normally" because the execution of the next steps might depend on the existence of the output files.For example, when running in dryrun mode, the following script will be executed "normally". The `_input` files (which are the output of step 10) could be opened at step 20, but the content would be empty:<jupyter_code>!rm -f test*.txt
%run -n

[10]
input: for_each={'i': (1, 2)}
output: f'test_{i}.txt'
sh: expand=True
  echo "Generating {_output}" > {_output}
  
[20]
input: group_by=1
output: f'{_input}.bak'
print(f'Content of {_input}:')
with open(_input) as ifile:
    print(ifile.read())
sh: expand=True
  cp {_input} {_output}<jupyter_output><empty_output><jupyter_text>`test_1.txt` and `test_2.txt` would be generated normally in run mode:<jupyter_code>%rerun<jupyter_output>Content of test_1.txt:
Generating test_1.txt

Content of test_2.txt:
Generating test_2.txt<jupyter_text>The placeholder files are removed automatically after the execution of the workflow in dryrun mode. However, if you interrupt a script running in dryrun mode, some placeholder files might remain and interfere with the execution of workflows. In this case, you can use command<jupyter_code>!sos remove --placeholders<jupyter_output>INFO: No remaining placeholder file exists.<jupyter_text>to remove such files. A shorter form of this command is `sos remove -p`.  External tasks External tasks are executed in `dryrun` mode as follows:1. Only task from the first substep will be generated and submitted.2. The task will be submitted to remote hosts as usual, with input files synchronized to remote host if needed. This allows you to test if the remote hosts are accessible.3. The task will be executed in `dryrun` mode, and will print instead of execute the scripts defined in actions.For example, although step `10` of the following workflow has 100 substeps, only the first task will be generated and executed in dryrun mode.<jupyter_code>%run -n

[10]
input: for_each={'i': range(100)}
output: f'test_{i}.txt'
task:
sh: expand=True
  echo "Generating {_output}" > {_output}<jupyter_output><empty_output><jupyter_text>Docker actions When an action is executed in docker in dryrun mode, SoS would download the docker image if needed but will not execute the script. Instead, SoS will print the interpolated script and the docker command needs to execute the script. For example, running an `docker_build` action in dryrun mode yields the following output:<jupyter_code>%run -n
[0]
docker_build:  tag='test/docker_build1', label='label with space', compress=True, memory='2G'
#
# Super simple example of a Dockerfile
#
FROM ubuntu:latest
MAINTAINER <NAME> "<EMAIL>"

WORKDIR /home<jupyter_output><empty_output><jupyter_text>and running a script in docker in dryrun mode:<jupyter_code>%run -n
[0]
run: docker_image='ubuntu'
echo 'Echo'<jupyter_output><empty_output><jupyter_text>Change system $PATH There are cases where you would like to use a specific version of programs for your workflow but do not want to change the system `$PATH` because of its global effect. In this case you can prepend pathes to these executables to `$PATH` using option `-b`.The following example first cretes a executable `ls` in `tmp` with an `echo` command. Using the option `-b tmp`, the `tmp` directory is prefixed to the system `$PATH` before the workflow is executed. The consequence is that this fake `ls` supersedes the system `ls` when `ls` is called in `step_10` of the workflow.<jupyter_code>%sandbox
!mkdir tmp
!echo "#!/bin/bash" > tmp/ls
!echo "echo This is fake ls" >> tmp/ls
!chmod +x tmp/ls

%run -b tmp
[10]
sh:
    ls<jupyter_output>This is fake ls<jupyter_text>The `-b` option has a default value `~/.sos/bin`, so any command under `~/.sos/bin` would be executed (before system command with the same name) even if the executables are not under system `$PATH`. This feature allows you to create commands that would only be used inside SoS-scripts, and more interestingly, allows you to create executable or install programs on-the-fly.For example, step 20 of the following workflow depends on an executable `lls` that is not a system executable.<jupyter_code>%sandbox
!rm -f ~/.sos/bin/lls

[install_lls: provides=executable('lls')]
run:
    echo "#!/bin/bash" > ~/.sos/bin/lls
    echo "echo This is lls" >> ~/.sos/bin/lls
    chmod +x ~/.sos/bin/lls

[20]
depends: executable('lls')
run:
    lls<jupyter_output><empty_output><jupyter_text>because `lls` is created under `~/.sos/bin`, it would be immediately available to SoS after the `install_lls` step. This works for any program as long as you can create a symbolic link under `~/.sos/bin` after its installation. You can always disable this behavior by setting option `-b` without value.  Runtime signature One of the most annonying problems with the development and execution of workflows is that it can take very long times to execute then. What makes things worse is that we frequently need to re-run the workflow with different paremeters and even different tools -- it can be really time-consuming to re-execute the whole workflow repeatedly, but it is also very error-prone to repeat selected steps of a workflow. SoS addresses this problem by using runtime signatures to keep track of execution units, namely the input, output, and dependent targets, and related SoS variables of a piece of workflow. SoS tracks execution of statements at the step level for each [input group](../documentation/SoS_Step.html) and saves runtime signatures at a folder called `.sos` under the project directory. The runtime signatures are used to1. Avoid repeated execution of identical units, and2. Keep track of workflow related files for project managementThis tutorial focuses on the first usage. The second one would be described in detail in [Project Management](Project_Management.html).  `ignore` mode SoS workflows can be executed in batch mode and in interactive mode using the SoS kernel in Jupyter notebook or qtconsole. Because the SoS kernel is mostly used to execute short statements in SOS and other kernels, runtime signatures are by default set to `ignore` in interactive mode (and to `default` in batch mode). Let us create a temporary directory and execute a workflow that take a bit of time to execute. This is done in the default `ignore` signature mode of the Jupyter notebook<jupyter_code>%sandbox --dir tmp

!rm -rf .sos/.runtime
![ -d temp ] || mkdir temp
%sandbox --dir tmp
%run
parameter: size=1000
[10]
output:  "temp/result.txt"
sh: expand=True
    dd if=/dev/urandom of={_output} count={size}

[20]
output:  'temp/size.txt'
with open(_output[0], 'w') as sz:
    sz.write(f"{_input}: {os.path.getsize(_input[0])}\n")<jupyter_output>1000+0 records in
1000+0 records out
512000 bytes transferred in 0.041920 secs (12213687 bytes/sec)<jupyter_text>Now, if we re-run the last script, nothing changes and it takes a bit of time to execute the script.<jupyter_code>%sandbox --dir tmp
%rerun<jupyter_output>INFO: [32mdefault_10[0m (index=0) is [32mignored[0m due to saved signature
INFO: [32mdefault_20[0m (index=0) is [32mignored[0m due to saved signature<jupyter_text>`default` mode Now let us switch to `default` mode of signature by running the script with option `-s default`. When you run the script for the first time, it would execute normally and save runtime signature of the steps.<jupyter_code>%sandbox --dir tmp
%rerun -s default<jupyter_output>INFO: [32mdefault_10[0m (index=0) is [32mignored[0m due to saved signature
INFO: [32mdefault_20[0m (index=0) is [32mignored[0m due to saved signature<jupyter_text>but both steps would be ignored. Here we use `-v2` to show the `ignored` message. This time we use magic `%set` to make option `-s default` persistent so that we do not have to specify it each time.<jupyter_code>%sandbox --dir tmp
%set -s default
%rerun -v2<jupyter_output>Set sos options to "-s default"<jupyter_text>However, if you use a different parameter (not the default `size=1000`), the steps would be rerun<jupyter_code>%sandbox --dir tmp
%rerun -v2 --size 2000<jupyter_output>2000+0 records in
2000+0 records out
1024000 bytes transferred in 0.085565 secs (11967497 bytes/sec)<jupyter_text>The signature is at the step level so if you change the second step of the script, the first step would still be skipped. Note that the step is independent of the script executed so a step would be skipped even if its signature was saved by the execution of another workflow. The signature is clever enough to allow minor changes such as addition of spaces and comments.<jupyter_code>%sandbox --dir tmp
%run --size 2000 -v2
parameter: size=1000
[10]
output:  "temp/result.txt"
# added comment
sh: expand=True
    dd if=/dev/urandom of={_output} count={size}

[20]
output:  'temp/size.txt'
with open(_output[0], 'w') as sz:
    sz.write(f"Modified {_input}: {os.path.getsize(_input[0])}\n")<jupyter_output>INFO: [32mdefault_10[0m (index=0) is [32mignored[0m due to saved signature<jupyter_text>`assert` mode The `assert` mode is used to detect if anything has been changed after the execution of a workflow. For example,<jupyter_code>%sandbox --dir tmp
%set -s assert
%rerun --size 2000 -v2<jupyter_output>Set sos options to "-s assert"<jupyter_text>Now if you change one of the output files, sos would fail with an error message.<jupyter_code>%sandbox --expect-error --dir tmp
!echo "aaa" >> temp/result.txt
%rerun --size 2000 -v2<jupyter_output>Failed to process step output: "temp/result.txt" (Signature mismatch: File has changed temp/result.txt)<jupyter_text>`force` mode The `force` signature mode ignores existing signatures and re-run the workflow. This is needed when you would like to forcefully re-run all the steps to generate another set of output if outcome of some steps is random, or to re-run the workflow because of changes that is not tracked by SoS, for example after you have installed a new version of a program.<jupyter_code>%sandbox --dir tmp
%set
%rerun --size 2000 -s force<jupyter_output>Reset sos options from "-s assert" to ""
2000+0 records in
2000+0 records out
1024000 bytes transferred in 0.088850 secs (11525039 bytes/sec)<jupyter_text>`build` mode The `build` mode is somewhat opposite to the `force` mode in that it creates (or overwrite existing signature if exists) with existing output files. It is useful, for example, if you are adding a step to a workflow that you have tested outside of SoS (without signature) but do not want to rerun it, or if for some reason you have lost your signature files and would like to reconstruct them from existing outputs.<jupyter_code>%sandbox --dir tmp
%rerun --size 2000 -s build -v2<jupyter_output>INFO: Step [32mdefault_10[0m (index=0) is [32mignored[0m with signature constructed
INFO: Step [32mdefault_20[0m (index=0) is [32mignored[0m with signature constructed<jupyter_text>This mode can introduce erraneous files to the signatures because it does not check the validity of the incorporated files. For example, SoS would not complain if you change parameter and replace `temp/result.txt` with something else.<jupyter_code>%sandbox --dir tmp
!echo "something else" > temp/result.txt
%rerun -s build -v2
# cleanup
!rm -rf tmp<jupyter_output><empty_output><jupyter_text>Removal of large intermediate files SoS keep tracks of all intermediate files and will rerun steps only if any of the tracked files are removed or changed. However, it is often desired to remove some of the large non-essential intemediate files to reduce diskspace used by completed workflows, while allowing the workflow to be re-executed without these files. SoS provides a command```sos remove files --zap```to zap specified file, or for example```sos remove . --size +5G --zap```to zap all files larger than 5G. This command removes specified files but keeps a special `{file}.zapped` file with essential information (e.g. md5 signature, and size). SoS would consider a file exist when a `.zapped` file is present and will only regenerate the file if the actual file is needed for a later step. For example, let us execute a workflow with output `temp/result.txt`, and `temp/size.txt`.<jupyter_code>%run -s force
[10]
output:  "temp/result.txt"
# added comment
sh: expand=True
    dd if=/dev/urandom of={_output} count=2000

[20]
output:  'temp/size.txt'
with open(_output[0], 'w') as sz:
    sz.write(f"Modified {_input}: {os.path.getsize(_input[0])}\n")<jupyter_output>2000+0 records in
2000+0 records out
1024000 bytes transferred in 0.082577 secs (12400549 bytes/sec)<jupyter_text>and let us zap the intermediate file `temp/result.txt`,<jupyter_code>!sos remove temp/result.txt --zap -y
!ls temp<jupyter_output>INFO: No tracked file from 0 run are identified.
INFO: 0 file zapped
result.txt size.txt<jupyter_text>As you can see, `temp/result.txt` is replaced with `temp/result.txt.zapped`. Now if you rerun the workflow<jupyter_code>%rerun -s default -v2<jupyter_output>INFO: [32mdefault_10[0m (index=0) is [32mignored[0m due to saved signature
INFO: [32mdefault_20[0m (index=0) is [32mignored[0m due to saved signature<jupyter_text>Summary Report Command `sos run` accepts an option `-p` (rePort) for generating a summary report after the completion of the workflow. This option accepts a name for the generated output but an name containing a timestamp would be generated automatically if it is the parameter is given without value.Using the same example but converting one of the steps as a task, a report named "test.html" would be generated with option `-p test.html`, showing summary of steps and tasks.<jupyter_code>%run -p test.html -s force
parameter: size=1000
[10]
output:  "temp/result.txt"
task:
sh: expand=True
    dd if=/dev/urandom of={_output} count={size}

[20]
output:  'temp/size.txt'
with open(_output[0], 'w') as sz:
    sz.write(f"{_input}: {os.path.getsize(_input[0])}\n")<jupyter_output><empty_output><jupyter_text>A DAG would be generated and included in the report if option `-d` is specified.<jupyter_code>%rerun -p test_dag.html -d -s force<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08948696383515559, 'fraction_numerical': 0.021306419960751333, 'mean_word_length': 5.129209621993128, 'pattern_counts': {'":': 0, '<': 104, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '378090', 'license': 'MIT', 'n_tokens_mistral': 5671, 'n_tokens_neox': 5317, 'n_words': 2410, 'path': 'src/tutorials/Execution_of_Workflow.ipynb', 'repo_name': 'andrewrocks/sos-docs', 'size': '17855'}