<commit_before># FindFungi-v0.23.3

A pipeline for the identification of fungi in public metagenomics datasets.
The FindFungi pipeline uses the metagenomics read-classifier Kraken with 32 custom fungal 
databases to generate 32 taxon predictions for a single read. These 32 predictions are 
combined to generate a consensus prediction. All reads are then BLASTed against their
predicted genomes to generate read distribution skewness scores to select for the 
most likely true positives.

FindFungi-v0.23.3 corrects an error in the LowestCommonAncestor_V4.sh script where the 
Python path and script path were incorrect.

FindFungi v0.23.2 corrects an error where the KrakenReduction.py script requested a file 
that was generated in FindFungi v0.22. This step has been removed.

FindFungi v0.23.1 corrects an error in v0.23, which did not correctly calculate Pearson’s 
skewness scores. A multiplication factor of 3 was omitted in v0.23 
(3(mean-median)/standard deviation). Skewness scores calculated by v0.23.1 are therefore 
3-times higher than with v0.23 (i.e. cut-offs should be expressed as -0.6 to +0.6, and 
not -0.2 to +0.2 as in v0.23). This makes no difference to the interpretation of any of 
the results or the data analysis. 

FindFungi-0.23 was built on an IBM platform load-sharing facility with 32 worker nodes.
Similar architecture is required to set up the pipeline due to the memory requirements of 
the Kraken databases.

## Quickstart

Download the pipeline, databases, associated scripts, prerequisites and other tools. 
Run the pipeline:

```
./FindFungi-0.23.3.sh /path/to/FASTQ-file.fastq Dataset-name
```

## Getting Started

These instructions will hopefully allow you to get a copy of FindFungi up and running 
on your own compute-cluster/server for development or your own analyses. If using a 
non-IBM LSF compute cluster, change the 'bsub' commands to reflect your architecture.
If using a single server, remove the 'bsub' commands.

### Prerequisites

FindFungi v0.23 was built using the following:

* gcc version 4.4.4 20100726 (Red Hat 4.4.4-13)
* coreutils 8.27
* python 2.7.13 (modules: sys, os, NCBITaxa, Bio, math, argparse, itertools, collections, 
re)
* skewer 0.2.2
* kraken 0.10.5-beta
* ncbi blast 2.2.30
* Rscript 3.3.3 (packages: wordcloud)
* graphviz 2.40.1

### Installing

* Download all of the scripts from GitHub/GiantSpaceRobot and move to a directory 
(/your/directory/scripts). You may need to give these scripts more permissions (e.g. chmod 755 *).
* In the FindFungi-v0.23.3 script, change the absolute paths of skewer, kraken, blast, the shell 
and python scripts to reflect your environment, or add these tools and scripts to you $PATH. 
You will also need to edit the LowestCommonAncestor.sh script to include the path to the downloaded scripts.
* NOTE: It may be necessary for you to include the absolute paths for all of the scripts
and tools within the FindFungi-0.23.sh master script, depending on the cluster node
preferences (e.g. executing 'python' actually calls the node's version of python, not yours).
* Download the Kraken and BLAST databases from this website (http://bioinformatics.czc.hokudai.ac.jp/findfungi/).<commit_msg>Added note about avoiding conda, using pip instead<commit_after># FindFungi-v0.23.3

A pipeline for the identification of fungi in public metagenomics datasets.
The FindFungi pipeline uses the metagenomics read-classifier Kraken with 32 custom fungal 
databases to generate 32 taxon predictions for a single read. These 32 predictions are 
combined to generate a consensus prediction. All reads are then BLASTed against their
predicted genomes to generate read distribution skewness scores to select for the 
most likely true positives.

FindFungi-v0.23.3 corrects an error in the LowestCommonAncestor_V4.sh script where the 
Python path and script path were incorrect.

FindFungi v0.23.2 corrects an error where the KrakenReduction.py script requested a file 
that was generated in FindFungi v0.22. This step has been removed.

FindFungi v0.23.1 corrects an error in v0.23, which did not correctly calculate Pearson’s 
skewness scores. A multiplication factor of 3 was omitted in v0.23 
(3(mean-median)/standard deviation). Skewness scores calculated by v0.23.1 are therefore 
3-times higher than with v0.23 (i.e. cut-offs should be expressed as -0.6 to +0.6, and 
not -0.2 to +0.2 as in v0.23). This makes no difference to the interpretation of any of 
the results or the data analysis. 

FindFungi-0.23 was built on an IBM platform load-sharing facility with 32 worker nodes.
Similar architecture is required to set up the pipeline due to the memory requirements of 
the Kraken databases.

## Quickstart

Download the pipeline, databases, associated scripts, prerequisites and other tools. 
Run the pipeline:

```
./FindFungi-0.23.3.sh /path/to/FASTQ-file.fastq Dataset-name
```

## Getting Started

These instructions will hopefully allow you to get a copy of FindFungi up and running 
on your own compute-cluster/server for development or your own analyses. If using a 
non-IBM LSF compute cluster, change the 'bsub' commands to reflect your architecture.
If using a single server, remove the 'bsub' commands.

### Prerequisites

FindFungi v0.23 was built using the following:

* gcc version 4.4.4 20100726 (Red Hat 4.4.4-13)
* coreutils 8.27
* python 2.7.13 (modules: sys, os, ete3, Bio, math, argparse, itertools, collections, 
re)
  * Don't use conda to install these modules, please use pip
* skewer 0.2.2
* kraken 0.10.5-beta
* ncbi blast 2.2.30
* Rscript 3.3.3 (packages: wordcloud)
* graphviz 2.40.1

### Installing

* Download all of the scripts from GitHub/GiantSpaceRobot and move to a directory 
(/your/directory/scripts). You may need to give these scripts more permissions (e.g. chmod 755 *).
* In the FindFungi-v0.23.3 script, change the absolute paths of skewer, kraken, blast, the shell 
and python scripts to reflect your environment, or add these tools and scripts to you $PATH. 
You will also need to edit the LowestCommonAncestor.sh script to include the path to the downloaded scripts.
* NOTE: It may be necessary for you to include the absolute paths for all of the scripts
and tools within the FindFungi-0.23.sh master script, depending on the cluster node
preferences (e.g. executing 'python' actually calls the node's version of python, not yours).
* Download the Kraken and BLAST databases from this website (http://bioinformatics.czc.hokudai.ac.jp/findfungi/).
{'fraction_non_alphanumeric': 0.06471494607087827, 'fraction_numerical': 0.035901386748844376, 'mean_word_length': 4.205292702485966, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2382015', 'n_tokens_mistral': 2068, 'n_tokens_neox': 1906, 'n_words': 906}