<commit_before># CR-index
Compressed read index - bioinf data structure for fast querying substring in reads.

## Installation

### Dependencies

  * [SDSL](https://github.com/simongog/sdsl-lite)
  * [Boost](http://www.boost.org/) (widely available as system package)
  * [PStreams](http://pstreams.sourceforge.net/) (widely available as system package)

### Installation

`make examples` compiles examples and all dependencies

# TODO

  * test e-coli bacteria
  * implement bloom filter?
    * [https://github.com/mavam/libbf]()
    * [https://code.google.com/p/bloom/]()

# Benchmark

  * 6M reads (generated from yarrowia.fasta), read length 100, 1% of errors
    * 352K critical reads
    * superstring compress ratio 2.59
    * superstring size 231M (1.85M reads were missing)
    * RSS ~ 300MB    
<commit_msg>dependencies added<commit_after># CR-index
Compressed read index - bioinf data structure for fast querying substring in reads.

## Installation

### Dependencies

  * [SGA](https://github.com/jts/sga)
  * [SDSL](https://github.com/simongog/sdsl-lite)
  * [Boost](http://www.boost.org/) (widely available as system package)
  * [PStreams](http://pstreams.sourceforge.net/) (widely available as system package)
  * [libboost](http://www.boost.org/) - "libboost-filesystem" and "libboost-system" libs

### Installation

`make examples` compiles examples and all dependencies

# TODO

  * test e-coli bacteria
  * implement bloom filter?
    * [https://github.com/mavam/libbf]()
    * [https://code.google.com/p/bloom/]()

# Benchmark

  * 6M reads (generated from yarrowia.fasta), read length 100, 1% of errors
    * 352K critical reads
    * superstring compress ratio 2.59
    * superstring size 231M (1.85M reads were missing)
    * RSS ~ 300MB    

{'fraction_non_alphanumeric': 0.13080407701019253, 'fraction_numerical': 0.022650056625141562, 'mean_word_length': 3.6256544502617802, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7670960', 'n_tokens_mistral': 624, 'n_tokens_neox': 568, 'n_words': 155}