<commit_before>Real data
---------

The `qsim` manuscript demonstrates how `qsim`-predicted mapping qualities improve performance when analyzing real data.

### Human

We use real data to demonstrate `qsim` performance.  Primarily we use two
human samples from the [1000 Genomes project](http://www.1000genomes.org):

* [ERR050082](http://www.ebi.ac.uk/ena/data/view/ERR050082)
    * 42,245,074 paired-end 100 x 100 nt reads
    * NA06985, CEPH, female
* [ERR050083](http://www.ebi.ac.uk/ena/data/view/ERR050083)
    * 66,391,067 paired-end 100 x 100 nt reads
    * NA11881, CEPH, male

Would be nice to know the sex here.

### Maize

(Future work)

Might also experiment with real data from a more repetitive genome, like maize.  If so, the links below might be relevant.

* http://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR1592709
* ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR159/009/SRR1592709/SRR1592709_1.fastq.gz
* ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR159/009/SRR1592709/SRR1592709_2.fastq.gz

### Steps

Steps:

* Download FASTQ reads (`get_real_reads.sh`)
* Align FASTQ reads with various tools, producing SAM files (`sbatch_align.sh` / `Makefile`)
* Sort and compare groups of SAM files, compiling CSV correct/incorrect information & ROC tables (`sbatch_multialign.sh` / `Makefile`)
* Generate table of `qsim` overhead measurements (`tabulate.py`)

From directory containing the `qsim` and `qsim-experiments` repo clones:

```
pushd qsim-experiments/experiments/real_data
sh get_real_reads.sh  # might want to submit to your DRM
sh sbatch_align.sh
# copy and paste all the alignment jobs to submit them
# ...when those are done, proceed
sh sbatch_multialign.sh
# copy and paste all the alignment jobs to submit them
python tabulate.py
popd
```<commit_msg>qsim->qtip and updating tabulate scripts<commit_after>Real data
---------

The `qtip` manuscript demonstrates how `qtip`-predicted mapping qualities improve performance when analyzing real data.

### Human

We use real data to demonstrate `qtip` performance.  Primarily we use two
human samples from the [1000 Genomes project](http://www.1000genomes.org):

* [ERR050082](http://www.ebi.ac.uk/ena/data/view/ERR050082)
    * 42,245,074 paired-end 100 x 100 nt reads
    * NA06985, CEPH, female
* [ERR050083](http://www.ebi.ac.uk/ena/data/view/ERR050083)
    * 66,391,067 paired-end 100 x 100 nt reads
    * NA11881, CEPH, male

Would be nice to know the sex here.

### Maize

(Future work)

Might also experiment with real data from a more repetitive genome, like maize.  If so, the links below might be relevant.

* http://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR1592709
* ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR159/009/SRR1592709/SRR1592709_1.fastq.gz
* ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR159/009/SRR1592709/SRR1592709_2.fastq.gz

### Steps

Steps:

* Download FASTQ reads (`get_real_reads.sh`)
* Align FASTQ reads with various tools, producing SAM files (`sbatch_align.sh` / `Makefile`)
* Sort and compare groups of SAM files, compiling CSV correct/incorrect information & ROC tables (`sbatch_multialign.sh` / `Makefile`)
* Generate table of `qtip` overhead measurements (`perf_tabulate.py > perf.csv`)
* Generate table of `qtip` accuracy measurements (`overall_tabulate.py`)

From directory containing the `qtip` and `qtip-experiments` repo clones:

```
pushd qtip-experiments/experiments/real_data
sh get_real_reads.sh  # might want to submit to your DRM
sh sbatch_align.sh
# copy and paste all the alignment jobs to submit them
# ...when those are done, proceed
sh sbatch_multialign.sh
# copy and paste all the alignment jobs to submit them
python perf_tabulate.py > perf.csv
python overall_tabulate.py
popd
```
{'fraction_non_alphanumeric': 0.11280239195433542, 'fraction_numerical': 0.06577874422397391, 'mean_word_length': 4.287356321839081, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6695548', 'n_tokens_mistral': 1459, 'n_tokens_neox': 1243, 'n_words': 383}