<commit_before># fastq-profiler

__fastq-profiler__ is a command line utility for keeping track of fastqs and process and store information associated with them. `fastq-profiler` generates summary statistics from processed fastq files and stores the data using the files md5sum as an identifier in [Google Datastore](https://cloud.google.com/datastore/).

I chose Google Datastore because it is centralized - allowing you to profile fastqs locally, or within cluster environments and elsewhere in parallel without having to track/combine files. `fastq-profiler` also integrates with [FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and can be used to output fastq statistics for analysis. Importantly, when duplicates are identified, fastq-profiler keeps track of both locations, allowing you to identify manage duplicate files if necessary.

### Installation

```
pip install https://github.com/danielecook/fastq-profiler/archive/v0.0.1.tar.gz
```

### Setup

* Setup an account with google cloud.
* Authorize Google Cloud using the [gcloud SDK](https://cloud.google.com/sdk/):

```
gcloud auth login
```

* Set your project and default Google Datastore "kind" using:

```
fq set <project> <kind>
```

### Data

__fastq-profiler__ generates a hash of every fastq submitted and uses it to track and store data about fastqs. The end result looks like this within the browsable Google Datastore interface:

![fqprofile-datastore](datastore.png)

The following pieces of information about fastqs (properties) are stored:

__Array Elements__

The three properties below are stored as arrays. Elements within those arrays correspond with one another (e.g. [1,2] ~ [A,B])

* `hostname` - Hostname
* `basename` Fastq basename
* `filename` - Absolute path of Fastq (in every location identified)

__Singlular properties__

* `key` Hash of the file
* `locations_count` - Count of identified locations.
* `date_created` Earliest identified date created
* `flowcell_lane` Flowcell lane
* `filesize` - Filesize in bytes
* `hfilesize` - Filesize in human readable form
* `total_reads` - Read count
* `[ATCGN]_count` - Base counts
* `GC_content` - GC content
* `min_length` - Minimum read length
* `avg_length` - Average read length
* `max_length` - maximum read length

__Additional fields included if applicable__

* `instrument` Instrument name if available
* `flowcell_lane` 
* `flowcell_number`
* `run_id`
* `pair` 1/2 for paired end sequencing.
* `barcode` Index/barcode of read for pooled sequencing
* `control_bits` 

__Illumina Filename__

If the filename follows the [Illumina filename conventions](http://support.illumina.com/content/dam/illumina-support/help/BaseSpaceHelp_v2/Content/Vault/Informatics/Sequencing_Analysis/BS/swSEQ_mBS_FASTQFiles.htm), these items will be parsed out as well:

* `illumina_filename_sample`
* `illumina_filename_barcode_sequence` OR `illumina_filename_sample_number`
* `illumina_filename_lane`
* `illumina_filename_read`
* `illumina_filename_set_number`

_For example:_

`EA-CFB-2-421_S1_L001_R1_001.fastq.gz` would be parsed into:

* `illumina_filename_sample` = EA-CFB-2-421U
* `illumina_filename_sample_number` = S1
* `illumina_filename_lane` = L001
* `illumina_filename_read` =  R1
* `illumina_filename_set_number` 1

`fq profile` creates a .checksum file in every directory containing fastqs that it is run on. The `.checksum` file is used as a cache of file hashes to make retrieval of data easier and help with file tracking.

__FastQC Stats__
<commit_msg>Fix typo<commit_after># fastq-profiler

__fastq-profiler__ is a command line utility for keeping track of fastqs and process and store information associated with them. `fastq-profiler` generates summary statistics from processed fastq files and stores the data using the files md5sum as an identifier in [Google Datastore](https://cloud.google.com/datastore/).

I chose Google Datastore because it is centralized - allowing you to profile fastqs locally, or within cluster environments and elsewhere in parallel without having to track/combine files. `fastq-profiler` also integrates with [FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and can be used to output fastq statistics for analysis. Importantly, when duplicates are identified, fastq-profiler keeps track of both locations, allowing you to identify manage duplicate files if necessary.

### Installation

```
pip install https://github.com/danielecook/fastq-profiler/archive/v0.0.1.tar.gz
```

### Setup

* Setup an account with google cloud.
* Authorize Google Cloud using the [gcloud SDK](https://cloud.google.com/sdk/):

```
gcloud auth login
```

* Set your project and default Google Datastore "kind" using:

```
fq set <project> <kind>
```

### Data

__fastq-profiler__ generates a hash of every fastq submitted and uses it to track and store data about fastqs. The end result looks like this within the browsable Google Datastore interface:

![fqprofile-datastore](datastore.png)

The following pieces of information about fastqs (properties) are stored:

__Array Elements__

The three properties below are stored as arrays. Elements within those arrays correspond with one another (e.g. [1,2] ~ [A,B])

* `hostname` - Hostname
* `basename` Fastq basename
* `filename` - Absolute path of Fastq (in every location identified)

__Singlular properties__

* `key` Hash of the file
* `locations_count` - Count of identified locations.
* `date_created` Earliest identified date created
* `flowcell_lane` Flowcell lane
* `filesize` - Filesize in bytes
* `hfilesize` - Filesize in human readable form
* `total_reads` - Read count
* `[ATCGN]_count` - Base counts
* `GC_content` - GC content
* `min_length` - Minimum read length
* `avg_length` - Average read length
* `max_length` - maximum read length

__Additional fields included if applicable__

* `instrument` Instrument name if available
* `flowcell_lane` 
* `flowcell_number`
* `run_id`
* `pair` 1/2 for paired end sequencing.
* `barcode` Index/barcode of read for pooled sequencing
* `control_bits` 

__Illumina Filename__

If the filename follows the [Illumina filename conventions](http://support.illumina.com/content/dam/illumina-support/help/BaseSpaceHelp_v2/Content/Vault/Informatics/Sequencing_Analysis/BS/swSEQ_mBS_FASTQFiles.htm), these items will be parsed out as well:

* `illumina_filename_sample`
* `illumina_filename_barcode_sequence` OR `illumina_filename_sample_number`
* `illumina_filename_lane`
* `illumina_filename_read`
* `illumina_filename_set_number`

_For example:_

`EA-CFB-2-421_S1_L001_R1_001.fastq.gz` would be parsed into:

* `illumina_filename_sample` = EA-CFB-2-421
* `illumina_filename_sample_number` = S1
* `illumina_filename_lane` = L001
* `illumina_filename_read` =  R1
* `illumina_filename_set_number` 1

`fq profile` creates a .checksum file in every directory containing fastqs that it is run on. The `.checksum` file is used as a cache of file hashes to make retrieval of data easier and help with file tracking.

__FastQC Stats__

{'fraction_non_alphanumeric': 0.08608893956670467, 'fraction_numerical': 0.008836944127708095, 'mean_word_length': 5.198763250883392, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1127755', 'n_tokens_mistral': 2210, 'n_tokens_neox': 2085, 'n_words': 753}