<commit_before>---
layout: post
title: "An anvi'o workflow for microbial pangenomics"
excerpt: "The user-friendly interface anvi'o provides to work with pangenomes."
modified: 2015-10-14
tags: []
categories: [anvio]
comments: true
authors: [tom, meren]
---

{% include _toc.html %}

Cultivation of closely related microorganisms, and the subsequent recovery of their genomes, revealed that not all isolates share the same functional traits for a given population. In other words, individual isolates do not fully echo the functional complexity of naturally occurring microbial populations, and reaching to a plateau of functional recovery is only possible through large genomic collections. Overlapping and differing functions among the genomes of closely related organisms led to the introduction of a new concept, "pangenomics" [(Tettelin et al., 2005)](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1216834/), where genes across genomes are segregated into distinct groups: the core-genome, genes detected in multiple yet not all genomes, and finally isolate-specific genes. Pangenomic investigations are now widely used to dissect the functional traits of microorganisms, and to uncover environmentally- and clinically-important patterns of functioning.

Although [multiple bioinformatics software](http://omictools.com/pangenomics-c1590-p1.html) are available to generate and/or visualize pangenomes, these solutions do not necessary offer flexible work environments, and hence limit the user's ability to interact with their data. 

That's why we are happy to demonstrate how [anvi'o](https://peerj.com/articles/1319/) can be used to process, visualize, and manipulate pangenomic data in its user-friendly environment. Although we are still [working](http://github.com/meren/anvio) on new modules for a fully automatized pangenomic workflow, the current anvi'o interface already provide its user with the opportunity to analyze pangenomes combined with a variety of contextual data, and to generate high-quality, publication-ready figures.

Our goal in this article is to describe original pangenomic analyses of publically available genomic data to demonstrate the anvi'o pangenomic workflow. Please don't hesitate to use the comments section for your suggestions, our [discussion group]({% post_url anvio/2015-10-14-anvio-discussion-group %}) for your technical questions. You can also keep an eye on our [public code repository](http://github.com/meren/anvio) for new releases, or to report issues.

Please note that you can find every file mentioned in this article, including run scripts to invoke anvi'o interactive interface here: [http://dx.doi.org/10.6084/m9.figshare.1603512](http://dx.doi.org/10.6084/m9.figshare.1603512).

## Step 1: Creating input files

You will find all files metnioned in this section in `01-ECOLI-PANGENOME/` directory of our figshare archive.

### Generating protein clusters for the data matrix

For our analysis of ten *Escherichia coli* (EC) genomes, we downloaded genbank files from [ftp://ftp.ncbi.nlm.nih.gov/genomes/Bacteria/](ftp://ftp.ncbi.nlm.nih.gov/genomes/Bacteria/).

{:.notice}
If you would like to repeat this analysis on your collection of genomes, you can use any gene finder that reports proper genbank files for open reading frames identified, and use those genbank files instead (we thought prodigal would work, but the genbank files prodigal generates does not seem to be compatible with genbank file format ITEP expects. Why? Because <a href="https://twitter.com/merenbey/status/662373313826693120">reasons</a>).

We clustered amino acid sequences for 47,415 open reading frames in these genomes using [ITEP](https://price.systemsbiology.org/research/itep/) ("Integrated Toolkit for Exploration of Pan-genomes", a useful tool we were introduced by our colleague [<NAME>](https://twitter.com/RikaEAnderson)). ITEP identified a total of 7,795 protein clusters in our collection, where each cluster contained 1 to 102 proteins. 

The main purpose of this step is to create a basic matrix to connect genomes and protein clusters. Although we used ITEP, any other solution to cluster protein sequences could have been used to acquire this essential input file.

We stored this information as a matrix of protein clusters and their occurrences across 10 genomes (see `data.txt` in our figshare archive) which should look like the following table, where each cell shows the count of a given protein cluster (first column) in each genome (first row):

|contig|O157_H7-EDL933|CFT073|K-12_W3110|O157-H7-Sakai|SE11|SE15|BL21-DE3|K-12_MG1655|O103-H2-12009|O111-H-11128|
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|1|18|8|0|18|0|0|0|0|23|35|
|2|19|9|0|18|0|0|0|0|23|33|
|3|2|1|6|1|1|4|56|6|1|1|
|4|10|2|2|11|2|1|3|1|10|12|
|5|9|3|2|11|2|1|1|2|10|10|
|6|0|0|7|1|1|3|29|7|1|2|
|7|5|6|4|4|7|5|3|4|6|5|
|8|14|6|0|13|0|5|0|0|6|3|
|9|9|3|0|10|4|1|1|0|9|9|
|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|
|7795|0|0|0|0|0|0|0|0|0|1|

If you are using ITEP, you should be creating this file from the output file ITEP reports and stores in `flatclusters/` directory. This may be a bit challening, and if you like, you can use this script here to convert the ITEP output into our `data.txt`: `anvi-script-itep-to-data-txt`. This script may already be available to you with your latest anvi'o installation (just copy paste it in your terminal to see whether it is the case). If not, download it from here: [anvi-script-itep-to-data-txt](https://github.com/meren/anvio/blob/master/sandbox/anvi-script-itep-to-data-txt).

When you have this script, you can run this command to get the `data.txt` (if you didn't have the script and ended up downloading it, put "python" at the beginning of the following command):

{% highlight bash %}
$ anvi-script-itep-to-data-txt flatclusters/all_I_2.0_c_0.5_m_maxbit
Num genomes ..................................: 10
Num protein clusters .........................: 7,795
Data file is generated .......................: data.txt
{% endhighlight %}

### Creating a tree of protein clusters

We created a tree based on the distribution of protein clusters across genomes (`tree.newick`). This tree will be used as the centerpiece in the interactive interface you will see very soon:<commit_msg>better<commit_after>---
layout: post
title: "An anvi'o workflow for microbial pangenomics"
excerpt: "The user-friendly interface anvi'o provides to work with pangenomes."
modified: 2015-10-14
tags: []
categories: [anvio]
comments: true
authors: [tom, meren]
---

{% include _toc.html %}

Cultivation of closely related microorganisms, and the subsequent recovery of their genomes, revealed that not all isolates share the same functional traits for a given population. In other words, individual isolates do not fully echo the functional complexity of naturally occurring microbial populations, and reaching to a plateau of functional recovery is only possible through large genomic collections. Overlapping and differing functions among the genomes of closely related organisms led to the introduction of a new concept, "pangenomics" [(Tettelin et al., 2005)](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1216834/), where genes across genomes are segregated into distinct groups: the core-genome, genes detected in multiple yet not all genomes, and finally isolate-specific genes. Pangenomic investigations are now widely used to dissect the functional traits of microorganisms, and to uncover environmentally- and clinically-important patterns of functioning.

Although [multiple bioinformatics software](http://omictools.com/pangenomics-c1590-p1.html) are available to generate and/or visualize pangenomes, these solutions do not necessary offer flexible work environments, and hence limit the user's ability to interact with their data. 

That's why we are happy to demonstrate how [anvi'o](https://peerj.com/articles/1319/) can be used to process, visualize, and manipulate pangenomic data in its user-friendly environment. Although we are still [working](http://github.com/meren/anvio) on new modules for a fully automatized pangenomic workflow, the current anvi'o interface already provide its user with the opportunity to analyze pangenomes combined with a variety of contextual data, and to generate high-quality, publication-ready figures.

Our goal in this article is to describe original pangenomic analyses of publically available genomic data to demonstrate the anvi'o pangenomic workflow. Please don't hesitate to use the comments section for your suggestions, our [discussion group]({% post_url anvio/2015-10-14-anvio-discussion-group %}) for your technical questions. You can also keep an eye on our [public code repository](http://github.com/meren/anvio) for new releases, or to report issues.

Please note that you can find every file mentioned in this article, including run scripts to invoke anvi'o interactive interface here: [http://dx.doi.org/10.6084/m9.figshare.1603512](http://dx.doi.org/10.6084/m9.figshare.1603512).

## Step 1: Creating input files

You will find all files metnioned in this section in `01-ECOLI-PANGENOME/` directory of our figshare archive.

### Generating protein clusters for the data matrix

For our analysis of ten *Escherichia coli* (EC) genomes, we downloaded genbank files from [ftp://ftp.ncbi.nlm.nih.gov/genomes/Bacteria/](ftp://ftp.ncbi.nlm.nih.gov/genomes/Bacteria/).

{:.notice}
If you would like to repeat this analysis on your collection of genomes, you can use any gene finder that reports proper genbank files for open reading frames identified, and use those genbank files instead (we thought prodigal would work, but the genbank files prodigal generates does not seem to be compatible with genbank file format ITEP expects. Why? Because <a href="https://twitter.com/merenbey/status/662373313826693120">reasons</a>).

We clustered amino acid sequences for 47,415 open reading frames in these genomes using [ITEP](https://price.systemsbiology.org/research/itep/) ("Integrated Toolkit for Exploration of Pan-genomes", a useful tool we were introduced by our colleague [<NAME>](https://twitter.com/RikaEAnderson)). ITEP identified a total of 7,795 protein clusters in our collection, where each cluster contained 1 to 102 proteins. 

The main purpose of this step is to create a basic matrix to connect genomes and protein clusters. Although we used ITEP, any other solution to cluster protein sequences could have been used to acquire this essential input file.

We stored this information as a matrix of protein clusters and their occurrences across 10 genomes (see `data.txt` in our figshare archive) which should look like the following table, where each cell shows the count of a given protein cluster (first column) in each genome (first row):

|contig|O157_H7-EDL933|CFT073|K-12_W3110|O157-H7-Sakai|SE11|SE15|BL21-DE3|K-12_MG1655|O103-H2-12009|O111-H-11128|
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|1|18|8|0|18|0|0|0|0|23|35|
|2|19|9|0|18|0|0|0|0|23|33|
|3|2|1|6|1|1|4|56|6|1|1|
|4|10|2|2|11|2|1|3|1|10|12|
|5|9|3|2|11|2|1|1|2|10|10|
|6|0|0|7|1|1|3|29|7|1|2|
|7|5|6|4|4|7|5|3|4|6|5|
|8|14|6|0|13|0|5|0|0|6|3|
|9|9|3|0|10|4|1|1|0|9|9|
|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|(...)|
|7795|0|0|0|0|0|0|0|0|0|1|

If you are using ITEP, you should be creating this file from the output file ITEP reports and stores in `flatclusters/` directory. This may be a bit challening, and if you like, you can use this script here to convert the ITEP output into a proper `data.txt` file: [`anvi-script-itep-to-data-txt`](https://github.com/meren/anvio/blob/master/sandbox/anvi-script-itep-to-data-txt)

{:.notice}
`anvi-script-itep-to-data-txt` may already be available to you with your latest anvi'o installation (just copy-paste the name in your terminal to see whether it is the case). If not, follow [this link](https://github.com/meren/anvio/blob/master/sandbox/anvi-script-itep-to-data-txt) to download it somewhere on your computer.

Run the following command to get the `data.txt` from your ITEP result (if you didn't have the script and ended up downloading it, don't forget to put "python" at the beginning of the following command):

{% highlight bash %}
$ anvi-script-itep-to-data-txt flatclusters/all_I_2.0_c_0.5_m_maxbit
Num genomes ..................................: 10
Num protein clusters .........................: 7,795
Data file is generated .......................: data.txt
{% endhighlight %}

### Creating a tree of protein clusters

We created a tree based on the distribution of protein clusters across genomes (`tree.newick`). This tree will be used as the centerpiece in the interactive interface you will see very soon:
{'fraction_non_alphanumeric': 0.11872791519434629, 'fraction_numerical': 0.04601491951315273, 'mean_word_length': 4.603167619885614, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1771729', 'n_tokens_mistral': 4193, 'n_tokens_neox': 3722, 'n_words': 1523}