<commit_before>Benchmarking with GIAB
======================

This workflow is used in our project to benchmark the sites identified in our call sets for sample NA12878 with the sites identified by the Genome in a Bottle consortium [https://github.com/genome-in-a-bottle]. 
This consortium was created to produce reference data sets that can be used for genome comparison and benchmarking and the high-confidence sites identified by this consortium for sample NA12878 are the result 
of an effort to integrate the data generated by 13 different sequencing technologies and analysis methods.

Dependencies
------------

* Nextflow
 This pipeline uses a workflow management system named Nextflow. This software can be downloaded from:

 https://www.nextflow.io/

* Bgzip and Tabix
 Bgzip and Tabix are part of the HTSlib project, which can be downloaded from:

 https://github.com/samtools/htslib

* BCFTools
 Downloadable from:

 http://www.htslib.org/download/

* IGSR-analysis code base
 The scripts needed to run this workflow can be downloaded by cloning the IGSR-analysis github repo from:

 https://github.com/igsr/igsr_analysis.git

Files required
--------------

* GIAB call set downloaded from::

  ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/latest/GRCh38/

* BED file with the high-confidence regions defined by GIAB and downloaded from::

  ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/latest/GRCh38/

* BED file with the regions that IGSR excludes from the comparison (file named exclude_nonvalid.bed) containing the GRCh38 coordinates 
  for the centromeres, non PAR regions for chrX and the entire chrY and chrM


How to run the pipeline
-----------------------

* First, you need to create a ``nexflow.config`` file that can be used by Nextflow to set the required variables. Here goes an example of one of these files::

	params.bcftools_folder='/homes/ernesto/bin/bcftools-1.6/' // folder containin the bcftools binary
	params.tabix='/nfs/production/reseq-info/work/ernesto/bin/anaconda3/bin/tabix' // path to tabix binary
	params.bgzip='/nfs/production/reseq-info/work/ernesto/bin/anaconda3/bin/bgzip' // path to bgzip
	params.non_valid_regions='/nfs/production/reseq-info/work/ernesto/isgr/SUPPORTING/REFERENCE/exclude_nonvalid.bed' // path to BED format file containing the regions that will be excluded from the comparison
	params.high_conf_regions='/nfs/production/reseq-info/work/ernesto/isgr/SUPPORTING/REFERENCE/GIAB/chr20DIR/HIGH_CONF_REGIONS/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.chr20.bed' // path to high-confidence regions as defined by GIAB

* Then, you can start your pipeline by doing::

	nextflow -c nextflow.config run $IGSR_CODEBASE/scripts/VCF/QC/compare_with_giab.nf --vcf VCF --chros chr20 --vt snps --giab giab.chr20.vcf.gz

 Where:
  * ``-c`` option allows you to specify the path to the ``nextflow.config`` file
  * ``$IGSR_CODEBASE`` is the folder containing the igsr codebase downloaded from ``https://github.com/igsr/igsr_analysis.git``
  * ``--vcf`` is the VCF that will be benchmarked against GIAB. Notice that you will need to create a tabix index of this file before running this pipeline
  * ``--chros`` is the chromosome or comma-separated list of chromosomes that will be analyzed. i.e.: chr20 or chr1,chr2,chr3 and so on...
  * ``--vt`` is the parameter used to set the type of variants that will be analyzed
  * ``--giab`` is the path to the GIAB VCF for the chromosome/s specified with option ``--chros``

Pipeline output
---------------
<commit_msg>changes in giab doc<commit_after>Benchmarking with GIAB
======================

This workflow is used in our project to benchmark the sites identified in our call sets for sample NA12878 with the sites identified by the Genome in a Bottle consortium [https://github.com/genome-in-a-bottle]. 
This consortium was created to produce reference data sets that can be used for genome comparison and benchmarking and the high-confidence sites identified by this consortium for sample NA12878 are the result 
of an effort to integrate the data generated by 13 different sequencing technologies and analysis methods.

Dependencies
------------

* Nextflow
 This pipeline uses a workflow management system named Nextflow. This software can be downloaded from:

 https://www.nextflow.io/

* Bgzip and Tabix
 Bgzip and Tabix are part of the HTSlib project, which can be downloaded from:

 https://github.com/samtools/htslib

* BCFTools
 Downloadable from:

 http://www.htslib.org/download/

* IGSR-analysis code base
 The scripts needed to run this workflow can be downloaded by cloning the IGSR-analysis github repo from:

 https://github.com/igsr/igsr_analysis.git

Files required
--------------

* GIAB call set downloaded from::

  ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/latest/GRCh38/

* BED file with the high-confidence regions defined by GIAB and downloaded from::

  ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/latest/GRCh38/

* BED file with the regions that IGSR excludes from the comparison (file named exclude_nonvalid.bed) containing the GRCh38 coordinates 
  for the centromeres, non PAR regions for chrX and the entire chrY and chrM


How to run the pipeline
-----------------------

* First, you need to create a ``nexflow.config`` file that can be used by Nextflow to set the required variables. Here goes an example of one of these files::

	params.bcftools_folder='/homes/ernesto/bin/bcftools-1.6/' // folder containin the bcftools binary
	params.tabix='/nfs/production/reseq-info/work/ernesto/bin/anaconda3/bin/tabix' // path to tabix binary
	params.bgzip='/nfs/production/reseq-info/work/ernesto/bin/anaconda3/bin/bgzip' // path to bgzip
	params.non_valid_regions='/nfs/production/reseq-info/work/ernesto/isgr/SUPPORTING/REFERENCE/exclude_nonvalid.bed' // path to BED format file containing the regions that will be excluded from the comparison
	params.high_conf_regions='/nfs/production/reseq-info/work/ernesto/isgr/SUPPORTING/REFERENCE/GIAB/chr20DIR/HIGH_CONF_REGIONS/HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.chr20.bed' // path to high-confidence regions as defined by GIAB for chromosome/s defined by the ``--chros`` option

* Then, you can start your pipeline by doing::

	nextflow -c nextflow.config run $IGSR_CODEBASE/scripts/VCF/QC/compare_with_giab.nf --vcf VCF --chros chr20 --vt snps --giab giab.chr20.vcf.gz

 Where:
  * ``-c`` option allows you to specify the path to the ``nextflow.config`` file
  * ``$IGSR_CODEBASE`` is the folder containing the igsr codebase downloaded from ``https://github.com/igsr/igsr_analysis.git``
  * ``--vcf`` is the VCF that will be benchmarked against GIAB. Notice that you will need to create a tabix index of this file before running this pipeline
  * ``--chros`` is the chromosome or comma-separated list of chromosomes that will be analyzed. i.e.: chr20 or chr1,chr2,chr3 and so on...
  * ``--vt`` is the parameter used to set the type of variants that will be analyzed
  * ``--giab`` is the path to the GIAB VCF for the chromosome/s specified with option ``--chros``

Pipeline output
---------------

{'fraction_non_alphanumeric': 0.10602112193114799, 'fraction_numerical': 0.017281580030174187, 'mean_word_length': 3.9336941813261164, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5517312', 'n_tokens_mistral': 2396, 'n_tokens_neox': 2169, 'n_words': 796}