<commit_before>---
layout: post
title: Anvi'o snakemake workflows
modified: 2018-02-15
excerpt: "Bringing the magic of anvi'o together with the wonders of snakemake."
comments: true
authors: [alon]
categories: [anvio]
---

{% capture images %}{{site.url}}/images/anvio/2018-02-15-anvio-snakemake-workflows{% endcapture %}

{% include _toc.html %}

{:.notice}
This post will only work for anvi'o `v5` or later, AND snakemake `v4` or later.

[Snakemake](https://snakemake.readthedocs.io/en/stable/) is a very useful and robust language for creating
computational workflows. Recently we have been using it extensivley and it has been wonderful.

In order to let others enjoy anvi'o together with the wonders of snakemake, we embarked on an effort to make
"easy-to-use" (well, not *too* easy .. after all this is 'science') snakefiles for some of the commonly used
workflows involving anvi'o.

The purpose of this post is to demonstrate through mock datasets the main use cases of these anvi'o snakemake workflows. We will begin with a general introduction, and continue with examples of each workflow using mock data. You can download the data package for this tutorial [in this TBD link](), and repeat all the steps described here to reproduce all the results.

# A general introduction to the anvi'o snakemake workflows

Running workflows (such as pangenomics or assembly-based metagenomics) requires many steps, especially for a project
with numerous samples. The snakemake workflows are here to help you streamline the repetative steps. To learn more
about snakemake, please refer to the [snakemake docummentation](https://snakemake.readthedocs.io/en/stable/);
If you don't wish to dig into the docummentation of snakemake right now, that's fine, the only essential detail right now is to know that each step of the analysis (for example running `anvi-gen-contigs-databse`) corresponds to a
"rule" in the snakemake workflow.

## Introducing: `anvi-run-workflow`

In order to make it easier to use these workflows for anvi'o users who are less familiar with snakemake,
we added a new program, `anvi-run-workflow`.

```
$ anvi-run-workflow -h

WARNING
===============================================
If you publish results from this workflow, please do not forget to cite
snakemake (doi:10.1038/nmeth.3176)

usage: anvi-run-workflow [-h] [-w WORKFLOW]
                         [--get-default-config OUTPUT_FILENAME]
                         [--list-workflows] [--list-dependencies]
                         [-c CONFIG_FILE] [--dry-run] [--save-workflow-graph]
                         [-A ...]

optional arguments:
  -h, --help            show this help message and exit

ESSENTIAL INPUTS:
  Things you must provide or this won't work

  -w WORKFLOW, --workflow WORKFLOW
                        You must specify a workflow name. To see a list of
                        available workflows run --list-workflows.

ADDITIONAL STUFF:
  additional stuff

  --get-default-config OUTPUT_FILENAME
                        Store a json formatted config file with all the
                        default settings of the workflow. This is a good draft
                        you could use in order to write your own config file.
                        This config file contains all parameters that could be
                        configured for this workflow. NOTICE: the config file
                        is provided with default values only for parameters
                        that are set by us in the workflow. The values for the
                        rest of the parameters are determined by the relevant
                        program.
  --list-workflows      Print a list of available snakemake workflows
  --list-dependencies   Print a list of the dependencies of this workflow. You
                        must provide a workflow name and a config file.
                        snakemake will figure out which rules need to be run
                        according to your config file, and according to the
                        files available on your disk. According to the rules
                        that need to be run, we will let you know which
                        programs are going to be used, so that you can make
                        sure you have all of them installed and loaded.
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        TBD
  --dry-run             Don't do anything real. Test everything, and stop
                        right before wherever the developer said 'well, this
                        is enough testing', and decided to print out results.
  --save-workflow-graph
                        Save a graph representation of the workflow. If you
                        are using this flag and if your system is unable to
                        generate such graph outputs, you will hear anvi'o
                        complaining (still, totally worth trying).
  -A ..., --additional-params ...
                        Additional snakemake parameters to add when running
                        snakemake. NOTICE: --additional-params HAS TO BE THE
                        LAST ARGUMENT THAT IS PASSED TO anvi-run-workflow,
                        ANYTHING THAT FOLLOWS WILL BE CONSIDERED AS PART OF
                        THE ADDITIONAL PARAMETERS THAT ARE PASSED TO
                        SNAKEMAKE. Any parameter that is accepted by snakemake
                        should be fair game here, but it is your
                        responsibility to make sure that whatever you added
                        makes sense. To see what parameters are available
                        please refer to the snakemake documentation. For
                        example, you could use this to set up cluster
                        submission using --additional-params --cluster "YOUR-
                        CLUSTER-SUBMISSION-CMD"
```

This program is meant to help you prepare the config file (which we will promptly learn about) and then run
the workflow.

If you want to see which workflows are available run:

```
$ anvi-run-workflow --list-workflows
Available workflows ..........................: contigs, metagenomics, pangenomics
```

We will do our best to keep this page up-to-date when we add or change things.

## The config.json file

In order to allow flexibility, the user can configure many parameters for the workflow using a config file.
Even if you want to use all the default parameters you still have to supply a config file, but don't worry, we got you covered,
you can always start by running:

```
anvi-run-workflow -w WORKFLOW-NAME --get-default-config OUTPUT_FILENAME
```

This will generate a config file with all the parameters that could be configured. For each parameter that you don't plan
to change, you can just keep it as is, or even remove it from your config file (to make your config file shorter and cleaner).

The config file contains configurations of three types:

1. general parameters of the workflow (e.g. a name for you project, which mode of the workflow to use (see metagenomics
workflow, `reference_mode`), etc.).

2. rule specific parameters (e.g. `min_contig_length` for `anvi-profile`). We tried as much as possible to allow the user to
change any parameter that is configurable in the underlying software that are used in the workflow.

3. names for the output directories of the workflows.

<div class="extra-info" markdown="1">

<span class="extra-info-header">A note regarding rule specific parameters</span>
To make things consistent, we decided that the way that parameters appear in the config file
will be identicle to their name in the underlying program. If there are multiple ways to use an argument
then we chose the longer one. For example, while `anvi-run-hmms` allows you to configure the path to the
HMM profiles directory using either `-H` or `--hmm-profile-dir`, we only allow you to use `--hmm-profile-dir` in the config file. But don't worry, if you use the wrong name, there will be an helpful message to let you know 
what are the correct arguments you can use.
</div>

Below, you can find example config files for each one of the workflows.

# Metagenomics workflow

The majority of the steps used in this workflow are extensively described in the
[anvi'o user tutorial for metagenomic workflow](http://merenlab.org/2016/06/22/anvio-tutorial-v2/).
However, in contrast to that tuturial, which begins with the FASTA files of contigs and BAM files, this
workflow includes steps to get there, including quality filtering, assembly, and mapping steps.

The default entering point to the metagenomics workflow are the unprocessed raw reads from one or more metagenomes.
The default output of the pipline is an anvi'o merged profile database ready for refinement of bins
(or whatever it is that you want to do with it), along with an annotated anvi'o contigs database.

The pipline includes the following steps:

1. QC of the metagenomes using [illumina-utils](https://github.com/merenlab/illumina-utils/). Including
generating a report of the results of the QC.
2. (Co-)Assembly of metagenomes using either [megahit](https://github.com/voutcn/megahit) or [idba_ud](https://github.com/loneknightpy/idba).
3. Generating an anvi'o CONTIGS database using [anvi-gen-contigs-database](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-gen-contigs-database).
4. Mapping short reads from metagenomes to the contigs using [bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml).
5. Profiling individual BAM files using [anvi-profile](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-profile).
6. Merging resulting anvi'o profile databases using [anvi-merge](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-merge).

This workflow also includes all steps that are included in the `contigs` workflow which is described below. In short,
these are all the steps to annotate your contigs database(s) with functions, HMMs, and taxonomy.

## Setting the stage

TBD (basically instructions to download a tar with all the files needed for this tutorial).

All the files that are currently needed are here: `/users/ashaiber/sandbox/snakemake_tutorial/metagenomics/WORKFLOW_TUTORIAL_DATA.tar.gz`. I just didn't get the chance to write a nice `wget`, etc. lines for it.

## Standard usage


As mentioned above, the standard usage of this workflow is meant to go through all the steps from raw reads to having a merged profile database (or databases) ready for binning.

All you need is a bunch of FASTQ files, and a `samples.txt` file. Here, we will go through a mock example with three small metagenomes. These metagenomes were made by choosing a small number of reads from three [HMP](https://www.hmpdacc.org/) metagenomes (these reads were not chosen randomly, for more details, [ask Alon](mailto:<EMAIL>)). In your working directory you have the following `samples.txt` file, let's have a look:

```bash
$ cat samples.txt

sample	group	r1	r2<commit_msg>Setting the stage for workflow tutorial<commit_after>---
layout: post
title: Anvi'o snakemake workflows
modified: 2018-02-15
excerpt: "Bringing the magic of anvi'o together with the wonders of snakemake."
comments: true
authors: [alon]
categories: [anvio]
---

{% capture images %}{{site.url}}/images/anvio/2018-02-15-anvio-snakemake-workflows{% endcapture %}

{% include _toc.html %}

{:.notice}
This post will only work for anvi'o `v5` or later, AND snakemake `v4` or later.

[Snakemake](https://snakemake.readthedocs.io/en/stable/) is a very useful and robust language for creating
computational workflows. Recently we have been using it extensivley and it has been wonderful.

In order to let others enjoy anvi'o together with the wonders of snakemake, we embarked on an effort to make
"easy-to-use" (well, not *too* easy .. after all this is 'science') snakefiles for some of the commonly used
workflows involving anvi'o.

The purpose of this post is to demonstrate through mock datasets the main use cases of these anvi'o snakemake workflows. We will begin with a general introduction, and continue with examples of each workflow using mock data. You can download the data package for this tutorial [in this TBD link](), and repeat all the steps described here to reproduce all the results.

# A general introduction to the anvi'o snakemake workflows

Running workflows (such as pangenomics or assembly-based metagenomics) requires many steps, especially for a project
with numerous samples. The snakemake workflows are here to help you streamline the repetative steps. To learn more
about snakemake, please refer to the [snakemake docummentation](https://snakemake.readthedocs.io/en/stable/);
If you don't wish to dig into the docummentation of snakemake right now, that's fine, the only essential detail right now is to know that each step of the analysis (for example running `anvi-gen-contigs-databse`) corresponds to a
"rule" in the snakemake workflow.

## Introducing: `anvi-run-workflow`

In order to make it easier to use these workflows for anvi'o users who are less familiar with snakemake,
we added a new program, `anvi-run-workflow`.

```
$ anvi-run-workflow -h

WARNING
===============================================
If you publish results from this workflow, please do not forget to cite
snakemake (doi:10.1038/nmeth.3176)

usage: anvi-run-workflow [-h] [-w WORKFLOW]
                         [--get-default-config OUTPUT_FILENAME]
                         [--list-workflows] [--list-dependencies]
                         [-c CONFIG_FILE] [--dry-run] [--save-workflow-graph]
                         [-A ...]

optional arguments:
  -h, --help            show this help message and exit

ESSENTIAL INPUTS:
  Things you must provide or this won't work

  -w WORKFLOW, --workflow WORKFLOW
                        You must specify a workflow name. To see a list of
                        available workflows run --list-workflows.

ADDITIONAL STUFF:
  additional stuff

  --get-default-config OUTPUT_FILENAME
                        Store a json formatted config file with all the
                        default settings of the workflow. This is a good draft
                        you could use in order to write your own config file.
                        This config file contains all parameters that could be
                        configured for this workflow. NOTICE: the config file
                        is provided with default values only for parameters
                        that are set by us in the workflow. The values for the
                        rest of the parameters are determined by the relevant
                        program.
  --list-workflows      Print a list of available snakemake workflows
  --list-dependencies   Print a list of the dependencies of this workflow. You
                        must provide a workflow name and a config file.
                        snakemake will figure out which rules need to be run
                        according to your config file, and according to the
                        files available on your disk. According to the rules
                        that need to be run, we will let you know which
                        programs are going to be used, so that you can make
                        sure you have all of them installed and loaded.
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        TBD
  --dry-run             Don't do anything real. Test everything, and stop
                        right before wherever the developer said 'well, this
                        is enough testing', and decided to print out results.
  --save-workflow-graph
                        Save a graph representation of the workflow. If you
                        are using this flag and if your system is unable to
                        generate such graph outputs, you will hear anvi'o
                        complaining (still, totally worth trying).
  -A ..., --additional-params ...
                        Additional snakemake parameters to add when running
                        snakemake. NOTICE: --additional-params HAS TO BE THE
                        LAST ARGUMENT THAT IS PASSED TO anvi-run-workflow,
                        ANYTHING THAT FOLLOWS WILL BE CONSIDERED AS PART OF
                        THE ADDITIONAL PARAMETERS THAT ARE PASSED TO
                        SNAKEMAKE. Any parameter that is accepted by snakemake
                        should be fair game here, but it is your
                        responsibility to make sure that whatever you added
                        makes sense. To see what parameters are available
                        please refer to the snakemake documentation. For
                        example, you could use this to set up cluster
                        submission using --additional-params --cluster "YOUR-
                        CLUSTER-SUBMISSION-CMD"
```

This program is meant to help you prepare the config file (which we will promptly learn about) and then run
the workflow.

If you want to see which workflows are available run:

```
$ anvi-run-workflow --list-workflows
Available workflows ..........................: contigs, metagenomics, pangenomics
```

We will do our best to keep this page up-to-date when we add or change things.

## The config.json file

In order to allow flexibility, the user can configure many parameters for the workflow using a config file.
Even if you want to use all the default parameters you still have to supply a config file, but don't worry, we got you covered,
you can always start by running:

```
anvi-run-workflow -w WORKFLOW-NAME --get-default-config OUTPUT_FILENAME
```

This will generate a config file with all the parameters that could be configured. For each parameter that you don't plan
to change, you can just keep it as is, or even remove it from your config file (to make your config file shorter and cleaner).

The config file contains configurations of three types:

1. general parameters of the workflow (e.g. a name for you project, which mode of the workflow to use (see metagenomics
workflow, `reference_mode`), etc.).

2. rule specific parameters (e.g. `min_contig_length` for `anvi-profile`). We tried as much as possible to allow the user to
change any parameter that is configurable in the underlying software that are used in the workflow.

3. names for the output directories of the workflows.

<div class="extra-info" markdown="1">

<span class="extra-info-header">A note regarding rule specific parameters</span>
To make things consistent, we decided that the way that parameters appear in the config file
will be identicle to their name in the underlying program. If there are multiple ways to use an argument
then we chose the longer one. For example, while `anvi-run-hmms` allows you to configure the path to the
HMM profiles directory using either `-H` or `--hmm-profile-dir`, we only allow you to use `--hmm-profile-dir` in the config file. But don't worry, if you use the wrong name, there will be an helpful message to let you know 
what are the correct arguments you can use.
</div>

Below, you can find example config files for each one of the workflows.

# Metagenomics workflow

The majority of the steps used in this workflow are extensively described in the
[anvi'o user tutorial for metagenomic workflow](http://merenlab.org/2016/06/22/anvio-tutorial-v2/).
However, in contrast to that tuturial, which begins with the FASTA files of contigs and BAM files, this
workflow includes steps to get there, including quality filtering, assembly, and mapping steps.

The default entering point to the metagenomics workflow are the unprocessed raw reads from one or more metagenomes.
The default output of the pipline is an anvi'o merged profile database ready for refinement of bins
(or whatever it is that you want to do with it), along with an annotated anvi'o contigs database.

The pipline includes the following steps:

1. QC of the metagenomes using [illumina-utils](https://github.com/merenlab/illumina-utils/). Including
generating a report of the results of the QC.
2. (Co-)Assembly of metagenomes using either [megahit](https://github.com/voutcn/megahit) or [idba_ud](https://github.com/loneknightpy/idba).
3. Generating an anvi'o CONTIGS database using [anvi-gen-contigs-database](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-gen-contigs-database).
4. Mapping short reads from metagenomes to the contigs using [bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml).
5. Profiling individual BAM files using [anvi-profile](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-profile).
6. Merging resulting anvi'o profile databases using [anvi-merge](http://merenlab.org/2016/06/22/anvio-tutorial-v2/#anvi-merge).

This workflow also includes all steps that are included in the `contigs` workflow which is described below. In short,
these are all the steps to annotate your contigs database(s) with functions, HMMs, and taxonomy.

## Setting the stage

We prepared some files to make it easy for you to go through this tutorial. Here are the steps to get things started on your computer:

```bash
wget http://merenlab.org/files/WORKFLOW_TUTORIAL_DATA.tar.gz

tar -xvzf WORKFLOW_TUTORIAL_DATA.tar.gz

cd WORKFLOW_TUTORIAL_DATA
```

Now you can follow the steps in this tutorial and run everything on your machine (assuming you have everything installed).

## Standard usage


As mentioned above, the standard usage of this workflow is meant to go through all the steps from raw reads to having a merged profile database (or databases) ready for binning.

All you need is a bunch of FASTQ files, and a `samples.txt` file. Here, we will go through a mock example with three small metagenomes. These metagenomes were made by choosing a small number of reads from three [HMP](https://www.hmpdacc.org/) metagenomes (these reads were not chosen randomly, for more details, [ask Alon](mailto:<EMAIL>)). In your working directory you have the following `samples.txt` file, let's have a look:

```bash
$ cat samples.txt

sample	group	r1	r2
{'fraction_non_alphanumeric': 0.0688361581920904, 'fraction_numerical': 0.007050847457627118, 'mean_word_length': 2.7938957475994513, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3744194', 'n_tokens_mistral': 6099, 'n_tokens_neox': 5656, 'n_words': 2826}