<jupyter_start><jupyter_text>The `task` statement and option `-q` * **Difficulty level**: intermediate* **Time need to lean**: 30 minutes or less* **Key points**:   * Tasks are defined by `task` statement as part of substeps   * Tasks are executed independently of SoS process and can be executed on remote hosts, even if the host does not share file systems with local host  The concept of task     Tasks are part of SoS substeps that are encapsulated and executed independently of the main workflow, possibly on remote hosts and cluster systems.  The idea behind tasks is that a workflow can contain a variety of substeps (scripts etc). Some of them are light weight and can be executed together with the workflow (on a local laptop or desktop), and some can be computational or resource intensive and have to be executed on the cluster, or have to be executed on specific remote hosts due to unavailablity of software or data.Whereas traditional workflow systems require all pieces of workflows to be executed on the same host, SoS allows you to send pieces of workflow to remote hosts, collect their results, and continue. This is achieved through a task mechanism.  The `task` statement Technically speaking, a **substep** consists of everything after the `input` statement. It can be repeated with subsets of input files or parameters defined by input options `group_by` or `for_each`. For example, if `bam_files` is a list of bam files,```sos[10]input: bam_files, group_by=1output: f"{_input}.bai"print(f'Indexing {_input}')run: expand=True    samtools index {_input}```execute a shell script to process each bam file. This is done sequentially for each input file, and is performed by SoS. You can easily specify part or all of a step process as **tasks**, by prepending the statements with a `task` keyword:```sos[10]input: bam_files, group_by=1output: f"{_input}.bai"print(f'Indexing {_input}')task:run: expand=True    samtools index {_input}```This statement declares the rest of the step process as a `task`. For each input file, a task will be created with an ID determined from task content and context (input and output files, variables etc).   Specification of task queue     A queue is needed to execute a task    The task statement will be ignored if no task queue is specified, either from command line with option -q or option queue of the task statement.    `task` statement is ignored by default Tasks need to be submitted to a **task queue** to be executed. If you do not specify queue, the task statement will be ignored.<jupyter_code>%run -v1

input: for_each=dict(i=range(2))
task:
sh: expand=True
  echo "I am task {i}"<jupyter_output>I am task 0
I am task 1<jupyter_text>Executing tasks in a `localhost` queue If you do not have anything defined in `~/.sos/hosts.yml` (see [host configuration](host_setup.html) for details), a `localhost` task queue will be available to you. Technically speaking this is a `process` queue with a maximum of 10 concurrent jobs. To submit the task to this queue, you can specify it as the `queue` option of `task` statement.<jupyter_code>%run -v1

input: for_each=dict(i=range(2))
task: queue='localhost'
sh: expand=True
echo "I am task {i}"<jupyter_output><empty_output><jupyter_text>As you can see, two tasks are generated from two substeps. The tasks have different IDs (because they have different value for variable `i`) and are executed successfully on `localhost`. A more convenient way to specify task queue is to use the `-q` option of command `sos run` (or magic `%run`,  `%sosrun`, or `%runfile`) as follows:<jupyter_code>%run -q localhost -v1

input: for_each=dict(i=range(2))
task:
sh: expand=True
echo "I am task {i}"<jupyter_output><empty_output><jupyter_text>The differences between these two methods are that1. Task queue specified by option `-q` is applied to all tasks defined in the workflow2. If both `-q` and option `queue` are specified, task queue specified by option `quque` will be used for specific tasks.That is to say, you should generally use `-q` to send tasks to different task queue, and use `queue` for tasks that are designed for specified queues. Under the hood, two task files are created under `~/.sos/tasks` (e.g. `4fe780b5245b9e7b.task`) and SoS executes commands such as `sos execute 4fe780b5245b9e7b` to execute them, and collect result from the tasks. This is why you cannot see the output from the `echo "I am task {i}"` command executed by the tasks.  Executing tasks on remote servers    A `process` queue, like a localhost, execute the task directly, subject to the number of concurrent tasks. If you have a properly configured remote host (namely, have public-key authentication, have SoS installed, specified its `address`, `paths` etc in `hosts` configuration), you can send tasks to the remote host for execution. More specifically, with the following example, SoS1. Creates a task with an unique ID (`c600532cd99c02c2`) determined by the content of the task.2. Copy the task file to remote host, translating `input` and `output` paths if two systems have different file systems.3. Copy input files to remote host if needed.4. Execute the task by executing something like  ```  ssh host "bash --login -c sos execute c600532cd99c02c2"  ```5. Monitor the status of the task and update the status in the notebook.6. Copy the output file to local host if needed. So with a host definition similar to the following```yamlhosts:  bcb:    address: bcbdesktop.mdanderson.edu    max_running_jobs: 100    paths:        home: /Users/{user_name}/scratch```I can submit a task with a `R` script to be executed on `bcb`<jupyter_code>%run -q bcb

output: 'test1.png'
task:
R:
  set.seed(1)
  x <- 1:100
  y <- -0.05*x + rnorm(100)
  png("test1.png", height=400, width=600)
  plot(x, y, pch=19, col=rgb(0.5, 0.5, 0.5, 0.5), cex=1.5)
  abline(lm(y ~ x))
  dev.off()<jupyter_output>INFO: No matching tasks are identified. Use option -a to check all tasks.
INFO: c600532cd99c02c2 [32mstarted[0m<jupyter_text>The output file is available locally after the completion of the task.<jupyter_code>%preview test1.png<jupyter_output><empty_output><jupyter_text>Difference between options -q and -r    If you have seen a similar example in remote execution of workflows, you will find that option -r and -q are similar in that they send the R script to remote host for execution. There are however a few key differences:  Option -r execute the entire workflows on remote host while option -q only part of substeps. In particular, option -r starts one `sos run` process on remote host and option -q starts two sos execute processes on the remote host. Option -r uses input files on remote host and leaves output files on remote host. Option -q copies input files from local to remote host, and copies output files from remote to local host after the tasks are completed.          Executing tasks on a cluster system    A server can also be a `pbs` queue, which represent a batch job management system such as Slurm, IBM LSF, or Torque. With proper configuration (again, see [host configuration](host_setup.html) for details), you will be able to submit your tasks to a cluster system as cluster jobs. The cluster system can be a localhost if you are submitting tasks on the headnode of a cluster system or a remote host (if you are submitting tasks remotely.For example, with a host configuration similar to```yamlhosts:     htc:        address: htc_cluster.mdanderson.edu        description: HTC cluster (PBS)        queue_type: pbs        status_check_interval: 60        submit_cmd: qsub {job_file}        status_cmd: qstat {job_id}        kill_cmd: qdel {job_id}                nodes: 1        cores: 4        walltime: 01:00:00        mem: 4G        task_template: |            !/bin/bash            PBS -N {job_name}            PBS -l nodes={nodes}:ppn={cores}            PBS -l walltime={walltime}            PBS -l mem={mem//10**9}GB            PBS -o ~/.sos/tasks/{task}.out            PBS -e ~/.sos/tasks/{task}.err            PBS -m n            module load R            {command}``` You can submit your task to the cluster system<jupyter_code>%run -q htc -s force

output: 'test.png'
task: walltime='1h', mem='2G', nodes=1, cores=1

R:
  set.seed(1)
  x <- 1:100
  y <- -0.03*x + rnorm(50)
  png("test.png", height=400, width=600)
  plot(x, y, pch=19, col=rgb(0.5, 0.5, 0.5, 0.5), cex=1.5)
  abline(lm(y ~ x))
  dev.off()<jupyter_output>INFO: No matching tasks are identified. Use option -a to check all tasks.<jupyter_text>Again, the output of the task will be synchronized to local host<jupyter_code>%preview test.png<jupyter_output><empty_output><jupyter_text>Although the usage of the workflow is largely the same as the previous example, the `task` statement has a few options to specify the resources used by the task, as required by the cluster system. The value of these options will be used to expand a template specified in the host specification file to generate a shell script that will be created and copied to remote host. The generated shell script could be found in the `-v4` output of the `sos status` or magic `%task status`<jupyter_code>%task status 1fafb4489a7b4b47 -q htc -v4<jupyter_output><empty_output><jupyter_text>Then, instead of command```ssh host "bash --login -c sos execute c600532cd99c02c2"```as in the last example, a command like```ssh host "bash --login -c qsub ~/.sos/tasks/1fafb4489a7b4b47.sh"```was used to submit the job to the cluster system. The entire workflow for executing tasks on a remote cluster system contains five steps and can be depicted as follows. The process would be a lot simpler if you are allowed to execute workflows and submit tasks on the headnode of a cluster system since no file synchronization would be needed.     Task Options The `task` statement accepts the following options| Option | Usage | reference | | -- | -- | --  | | `shared` | Return variable to the substep | this tutorial || `env` | Customized environment | this tutorial || `prepend_path` | Customized `$PATH` | this tutorial || `walltime` | Total `walltime` of task | [task template](task_template.html) || `mem` | Memory required for task | [task template](task_template.html)  || `nodes` | Number of computing nodes | [task template](task_template.html)  || `cores` | Number of cores per node | [task template](task_template.html)  || `workdir` |  | [path translation and file synchronization ](task_files.html) || `to_host` | | [path translation and file synchronization ](task_files.html) || `from_host` | | [path translation and file synchronization ](task_files.html) || `map_vars` | | [path translation and file synchronization ](task_files.html) || ANY | Any template variable | [task template](task_template.html)  |We will discuss some options in this tutorial, and leave the rest to specific tutorials.  Option `shared` SoS tasks are executed externally and by default does not return any value. Similar to the `shared` step option (that passes step variables to the workflow), you can use `shared` option to pass task variables to the step in which the task is defined.For example, the following script perform some simulations in 10 tasks and return the result by variable `rng`, which is then shared to the workflow by step option `shared` so that it can be available to the next step.<jupyter_code>%run -q localhost
[10 (simulate): shared='rng']
task: shared='rng'
import random
rng = random.randint(1, 1000)

[20]
print(rng)<jupyter_output>INFO: 62f388838fadbc2f [32mstarted[0m
898
{'fraction_non_alphanumeric': 0.08222742648972158, 'fraction_numerical': 0.019082314164281376, 'mean_word_length': 4.2865657955066485, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '397867', 'license': 'MIT', 'n_tokens_mistral': 3396, 'n_tokens_neox': 3201, 'n_words': 1608, 'path': 'src/user_guide/task_statement.ipynb', 'repo_name': 'sosdocs/sos-docs', 'size': '11529'}