<commit_before>TF ChIP-Seq Pipeline
===

TF ChIP-Seq pipeline is based on https://docs.google.com/document/d/1lG_Rd7fnYgRpSIqrIfuVlAz2dW1VaSQThzk836Db99c/edit#.

Dependencies: BigDataScript (BDS)


### Configuration file

Modify $CONF_FILE (by default: conf_tf_chipseq.txt) to have your own settings.

```
* PREFIX 			: Prefix for all output files
* OUTPUT_DIR 		: Output directory (both relative and absolute paths work)
* TMP_DIR 			: Temporary folder for intermediate files during bwa alignment
* USE_BGZIP			: Index BED type files (for visualization in a genome browser). Make sure bgzip and tabix installed add their path to MODULE_*.

* WALLTIME 			: default walltime for all jobs (in seconds)
* NTHREADS 			: default # of threads for all jobs
* MEMORY			: default max. memory for all jobs (in bytes)

* PRE_IDR 			: Set it true if you just want to compute QC score and stop before peak calling (default: false)
* NUM_REP			: # of replicates you want to test (default:2)

* NTHREADS_BWA 		: # of threads for bwa aligment
* BWA_INDEX_NAME	: Prefix of bwa index files (eg. if you have bwa index files including hg19_Male.bwt, BWA_INDEX_NAME=hg19_Male)
* BWA_PARAM			: Parameters for bwa alignment (default: -q 5 -l 32 -k 2)

* MARKDUP 			: Dupe marker path
* MAPQ_THRESH		: MAPQ_THRESH

* NTHREADS_R		: # of threads for peak calling (spp)
* DUPE_REMOVED		: If true, use run_spp.nodups.R instead of run_spp.R
* NREADS 			: NREADS (default: 15000000)
* NPEAK 			: -npeak NPEAK in run_spp.R (default: 300000)

* IDR_THRESH	 	: Threshold for IDR (default=0.02)

* CREATE_WIG  		: Create wig file from .tagAlign.gz
* CREATE_BEDGRAPH 	: Create bedGraph file from .tagAlign.gz
* CONVERT_TO_BIGWIG : Convert bedGraph to bigwig
* CHROM_SIZES 		: Location of chrom.sizes file for your .fa
* UMAP_DIR 			: Location of umap (for hg19, globalmap_k20tok54)
* SEQ_DIR 			: Location of sequence .fa files (for hg19, chr?.fa)

* MODULE_* 			: Freely name suffix and specify RHS, then BDS will run "module add RHS"
* EXPORT_* 			: Freely name suffix and specify RHS, then BDS will add env. variable to bash shell
```

### How to specify bioinformatics software version?

You can specify software versions with MODULE_* and EXPORT_* in the above configuration file. You can freely name suffix of MODULE_ or EXPORT_ then BDS will read all keys starting with MODULE_ or EXPORT_. Use ; for new line.

For example, to use bwa 0.7.7 and bedtools 2.x.x and samtools 1.2

```
MODULE_BWA= bwa/0.7.7 
MODULE_ANY_SUFFIX_SHOULD_BE_OKAY= bedtools/2.x.x; samtools/1.2
```

The above lines in the configuration file will execute the following:

```
module add bwa/0.7.7; module add bedtools/2.x.x; module add samtools/1.2

```

To use a software unmoduled, For example "sw_example" is on /usr/bin/example/, then add software path to environment variable PATH.

```
EXPORT_ETC= export PATH="${PATH}:/usr/bin/example"
EXPORT_TEST= TEST_PATH="/usr/lib/example"; export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TEST_PATH}"
EXPORT_YOU_CAN_DEFINE_ANY_CUSTOM_VAR= CUST_VAR=200
```

The above lines in the configuration file will execute the following:

```
export PATH="${PATH}:/usr/bin/example"; TEST_PATH="/usr/lib/example"; export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TEST_PATH}"; CUST_VAR=200
```

IMPORTANT!! There is a bug in BDS code. YOU NEED TO WRAP ANY ENVIRONMENT VARS IN RHS WITH CURVED BRACKETS ${} !!!!!

```
EXPORT_BASH= export PATH="${PATH}:/bin:/usr/bin"
```

IMPORTANT!! Do not remove the above line in the conf. file. Common linux commands like rm will not work without it.



### Usage 

Firstly, copy bds cluster settings (bds.config.scg3) in $PIPELINE_DIR (path for tf_chipseq.bds) to BDS directory. 
```
cp $PIPELINE_DIR/bds.config.scg3 $HOME/.bds/bds.config
vi $HOME/.bds/bds.config
```

Modify the following line in $HOME/.bds/bds.config. Due to BDS bug, you cannot use $HOME. Instead specify absolute path to your BDS directory (/home/your_account_name/.bds by default).

```
clusterRunAdditionalArgs = -v PATH=/home/leepc12/.bds
```

You can also specify your own cluster configuration file but I already provided one for SCG3.
```
bds -c your_own_bds.config any_script.bds
```

It is recommend to separate your working directory $WORK_DIR from $PIPELINE_DIR because BDS produces a lot of intermediate dir/files on $WORK_DIR. 

If you want to locally run jobs on your machine with both configuration files not specifed,

```
cd $WORK_DIR
bds $PIPELINE_DIR/tf_chipseq.bds
```

You can specify your own configuration file (conf_tf_chipseq.txt by default) with -c.

```
bds $PIPELINE_DIR/tf_chipseq.bds -c $CONF_FILE
```

If you want to run your jobs on scg3 cluster, use -s sge.
```
bds -s sge $PIPELINE_DIR/tf_chipseq.bds
```

If you run BDS script with -dryRun, it does not actually run the job, it compiles the script and list jobs to be executed.
```
bds -dryRun -s sge $PIPELINE_DIR/tf_chipseq.bds
```

For advanced users, both $BDS_CONFIG and $CONF_FILE (chipseq configuration file) can be specified like the following. Make sure you don't change the order of arguments.

```
bds -c $BDS_CONFIG -dryRun -s sge $PIPELINE_DIR/tf_chipseq.bds -c $CONF_FILE
```

### Installation insctruction for BigDataScript (BDS)

Official homepage for BDS is at <a href="http://pcingola.github.io/BigDataScript/download.html">http://pcingola.github.io/BigDataScript/download.html</a> and the git repo for BDS is at <a href="https://github.com/pcingola/BigDataScript.git">https://github.com/pcingola/BigDataScript.git</a>.

```
cd $HOME
wget -O bds_Linux.tgz https://github.com/pcingola/BigDataScript/releases/download/v0.999h/bds_Linux.tgz
tar zxvf bds_Linux.tgz
```

BDS is installed at $HOME/.bds/. You should add it to your PATH. Edit your $HOME/.bashrc and add the following:
```
export PATH=$PATH:$HOME/.bds/
```

### Local installation instruction for R-2.15.1

```
cd $HOME
wget http://cran.r-project.org/src/base/R-2/R-2.15.1.tar.gz
tar zxvf R-2.15.1.tar.gz
cd R-2.15.1
./configure --prefix=$HOME/R --with-readline=no --with-x=no --enable-R-static-lib
make
make install
cd ..
wget http://compbio.med.harvard.edu/Supplements/ChIP-seq/spp_1.10.tar.gz
$HOME/R/bin/R
	install.packages("snow")
	install.packages("snowfall")
	install.packages("bitops")
	install.packages("caTools")
q()
mkdir $HOME/RLib
$HOME/R/bin/R CMD INSTALL -l ~/RLib spp_1.10.tar.gz
```
Add the following line to your $HOME/.bashrc and to your configuration file.
```
export R_LIBS=$HOME/RLib
```

### Local installation instruction for SPP (run_spp.R: Anshul's phantompeakqualtool)
```
cd $HOME
wget https://phantompeakqualtools.googlecode.com/files/ccQualityControl.v.1.1.tar.gz
tar zxvf ccQualityControl.v.1.1.tar.gz
```
Make sure that you change mod for *.R in SPP to allow linux which finds them
```
chmod 755 phantompeakqualtools/*
```
Add the following line to your $HOME/.bashrc and to your configuration file.
```
export PATH=${PATH}:$HOME/phantompeakqualtools
```

### Local installation instruction for Python3 and packages
```
cd $HOME

echo Python3
wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz
tar zxvf Python-3.4.3.tgz
cd Python-3.4.3
./configure --prefix=$HOME/usr/local
make altinstall prefix=$HOME/usr/local exec-prefix=$HOME/usr/local

echo Cython 
wget http://cython.org/release/Cython-0.22.tar.gz
cd Cython-0.22
$HOME/usr/local/bin/python3.4 setup.py install --prefix=$HOME/local/

echo Packages
cd $HOME
$HOME/usr/local/bin/pip3.4 install --install-option="--prefix=$HOME/local" numpy
$HOME/usr/local/bin/pip3.4 install --install-option="--prefix=$HOME/local" matplotlib
```<commit_msg>bugfix<commit_after>TF ChIP-Seq Pipeline
===

TF ChIP-Seq pipeline is based on https://docs.google.com/document/d/1lG_Rd7fnYgRpSIqrIfuVlAz2dW1VaSQThzk836Db99c/edit#.

Dependencies: BigDataScript (BDS)


### Configuration file

Modify $CONF_FILE (by default: conf_tf_chipseq.txt) to have your own settings.

```
* PREFIX 			: Prefix for all output files
* OUTPUT_DIR 		: Output directory (both relative and absolute paths work)
* TMP_DIR 			: Temporary folder for intermediate files during bwa alignment
* USE_BGZIP			: Index BED type files (for visualization in a genome browser). Make sure bgzip and tabix installed add their path to MODULE_*.

* WALLTIME 			: default walltime for all jobs (in seconds)
* NTHREADS 			: default # of threads for all jobs
* MEMORY			: default max. memory for all jobs (in bytes)

* PRE_IDR 			: Set it true if you just want to compute QC score and stop before peak calling (default: false)
* NUM_REP			: # of replicates you want to test (default:2)

* NTHREADS_BWA 		: # of threads for bwa aligment
* BWA_INDEX_NAME	: Prefix of bwa index files (eg. if you have bwa index files including hg19_Male.bwt, BWA_INDEX_NAME=hg19_Male)
* BWA_PARAM			: Parameters for bwa alignment (default: -q 5 -l 32 -k 2)

* MARKDUP 			: Dupe marker path
* MAPQ_THRESH		: MAPQ_THRESH

* NTHREADS_R		: # of threads for peak calling (spp)
* DUPE_REMOVED		: If true, use run_spp.nodups.R instead of run_spp.R
* NREADS 			: NREADS (default: 15000000)
* NPEAK 			: -npeak NPEAK in run_spp.R (default: 300000)

* IDR_THRESH	 	: Threshold for IDR (default=0.02)

* CREATE_WIG  		: Create wig file from .tagAlign.gz
* CREATE_BEDGRAPH 	: Create bedGraph file from .tagAlign.gz
* CONVERT_TO_BIGWIG : Convert bedGraph to bigwig
* CHROM_SIZES 		: Location of chrom.sizes file for your .fa
* UMAP_DIR 			: Location of umap (for hg19, globalmap_k20tok54)
* SEQ_DIR 			: Location of sequence .fa files (for hg19, chr?.fa)

* MODULE_* 			: Freely name suffix and specify RHS, then BDS will run "module add RHS"
* EXPORT_* 			: Freely name suffix and specify RHS, then BDS will add env. variable to bash shell
```

### How to specify bioinformatics software version?

You can specify software versions with MODULE_* and EXPORT_* in the above configuration file. You can freely name suffix of MODULE_ or EXPORT_ then BDS will read all keys starting with MODULE_ or EXPORT_. Use ; for new line.

For example, to use bwa 0.7.7 and bedtools 2.x.x and samtools 1.2

```
MODULE_BWA= bwa/0.7.7 
MODULE_ANY_SUFFIX_SHOULD_BE_OKAY= bedtools/2.x.x; samtools/1.2
```

The above lines in the configuration file will execute the following:

```
module add bwa/0.7.7; module add bedtools/2.x.x; module add samtools/1.2

```

To use a software unmoduled, For example "sw_example" is on /usr/bin/example/, then add software path to environment variable PATH.

```
EXPORT_ETC= export PATH="${PATH}:/usr/bin/example"
EXPORT_TEST= TEST_PATH="/usr/lib/example"; export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TEST_PATH}"
EXPORT_YOU_CAN_DEFINE_ANY_CUSTOM_VAR= CUST_VAR=200
```

The above lines in the configuration file will execute the following:

```
export PATH="${PATH}:/usr/bin/example"; TEST_PATH="/usr/lib/example"; export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TEST_PATH}"; CUST_VAR=200
```

IMPORTANT!! There is a bug in BDS code. YOU NEED TO WRAP ANY ENVIRONMENT VARS IN RHS WITH CURVED BRACKETS ${} !!!!!

```
EXPORT_BASH= export PATH="${PATH}:/bin:/usr/bin"
```

IMPORTANT!! Do not remove the above line in the conf. file. Common linux commands like rm will not work without it.



### Usage 

Firstly, copy bds cluster settings (bds.config.scg3) in $PIPELINE_DIR (path for tf_chipseq.bds) to BDS directory. 
```
cp $PIPELINE_DIR/bds.config.scg3 $HOME/.bds/bds.config
vi $HOME/.bds/bds.config
```

Modify the following line in $HOME/.bds/bds.config. Due to BDS bug, you cannot use $HOME. Instead specify absolute path to your BDS directory (/home/your_account_name/.bds by default).

```
clusterRunAdditionalArgs = -v PATH=/home/leepc12/.bds
```

You can also specify your own cluster configuration file but I already provided one for SCG3.
```
bds -c your_own_bds.config any_script.bds
```

It is recommend to separate your working directory $WORK_DIR from $PIPELINE_DIR because BDS produces a lot of intermediate dir/files on $WORK_DIR. 

If you want to locally run jobs on your machine with both configuration files not specifed,

```
cd $WORK_DIR
bds $PIPELINE_DIR/tf_chipseq.bds
```

You can specify your own configuration file (conf_tf_chipseq.txt by default) with -c.

```
bds $PIPELINE_DIR/tf_chipseq.bds -c $CONF_FILE
```

If you want to run your jobs on scg3 cluster, use -s sge.
```
bds -s sge $PIPELINE_DIR/tf_chipseq.bds
```

If you run BDS script with -dryRun, it does not actually run the job, it compiles the script and list jobs to be executed.
```
bds -dryRun -s sge $PIPELINE_DIR/tf_chipseq.bds
```

For advanced users, both $BDS_CONFIG and $CONF_FILE (chipseq configuration file) can be specified like the following. Make sure you don't change the order of arguments.

```
bds -c $BDS_CONFIG -dryRun -s sge $PIPELINE_DIR/tf_chipseq.bds -c $CONF_FILE
```

### Installation insctruction for BigDataScript (BDS)

Official homepage for BDS is at <a href="http://pcingola.github.io/BigDataScript/download.html">http://pcingola.github.io/BigDataScript/download.html</a> and the git repo for BDS is at <a href="https://github.com/pcingola/BigDataScript.git">https://github.com/pcingola/BigDataScript.git</a>.

```
cd $HOME
wget -O bds_Linux.tgz https://github.com/pcingola/BigDataScript/releases/download/v0.999h/bds_Linux.tgz
tar zxvf bds_Linux.tgz
```

BDS is installed at $HOME/.bds/. You should add it to your PATH. Edit your $HOME/.bashrc and add the following:
```
export PATH=$PATH:$HOME/.bds/
```

### Local installation instruction for R-2.15.1

```
cd $HOME
wget http://cran.r-project.org/src/base/R-2/R-2.15.1.tar.gz
tar zxvf R-2.15.1.tar.gz
cd R-2.15.1
./configure --prefix=$HOME/R --with-readline=no --with-x=no --enable-R-static-lib
make
make install
cd ..
wget http://compbio.med.harvard.edu/Supplements/ChIP-seq/spp_1.10.tar.gz
$HOME/R/bin/R
	install.packages("snow")
	install.packages("snowfall")
	install.packages("bitops")
	install.packages("caTools")
q()
mkdir $HOME/RLib
$HOME/R/bin/R CMD INSTALL -l ~/RLib spp_1.10.tar.gz
```
Add the following line to your $HOME/.bashrc and to your configuration file.
```
export R_LIBS=${HOME}/RLib
```

### Local installation instruction for SPP (run_spp.R: Anshul's phantompeakqualtool)
```
cd $HOME
wget https://phantompeakqualtools.googlecode.com/files/ccQualityControl.v.1.1.tar.gz
tar zxvf ccQualityControl.v.1.1.tar.gz
```
Make sure that you change mod for *.R in SPP to allow linux which finds them
```
chmod 755 phantompeakqualtools/*
```
Add the following line to your $HOME/.bashrc and to your configuration file.
```
export PATH=${PATH}:${HOME}/phantompeakqualtools
```

### Local installation instruction for Python3 and packages
```
cd $HOME

echo Python3
wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz
tar zxvf Python-3.4.3.tgz
cd Python-3.4.3
./configure --prefix=$HOME/usr/local
make altinstall prefix=$HOME/usr/local exec-prefix=$HOME/usr/local

echo Cython 
wget http://cython.org/release/Cython-0.22.tar.gz
cd Cython-0.22
$HOME/usr/local/bin/python3.4 setup.py install --prefix=$HOME/local/

echo Packages
cd $HOME
$HOME/usr/local/bin/pip3.4 install --install-option="--prefix=$HOME/local" numpy
$HOME/usr/local/bin/pip3.4 install --install-option="--prefix=$HOME/local" matplotlib
```
{'fraction_non_alphanumeric': 0.1158137999339716, 'fraction_numerical': 0.017563552327500825, 'mean_word_length': 4.217361350327248, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1426184', 'n_tokens_mistral': 5750, 'n_tokens_neox': 5347, 'n_words': 1680}