<commit_before># 2-bit reader

This is a (work in progress) reader implementation, based on the
specs at UCSC, http://genome.ucsc.edu/FAQ/FAQformat#format7. The
implementation is intended to be more or less thread safe. each
TwoBitSequence has it's own read-only access to the 2-bit data.

## TODO
- fix extra nucleotides at byte boundaries
<commit_msg>notes<commit_after># 2-bit reader

This is a (work in progress) reader implementation, based on the
specs at UCSC, http://genome.ucsc.edu/FAQ/FAQformat#format7. The
implementation is intended to be more or less thread safe. each
TwoBitSequence has it's own read-only access to the 2-bit data.

## TODO
- fix extra nucleotides at byte boundaries
- probably take out buffering and rely on ifstream, see https://stackoverflow.com/questions/12757904/how-to-optimize-reading-and-writing-by-playing-with-buffer-size.

{'fraction_non_alphanumeric': 0.08333333333333333, 'fraction_numerical': 0.016203703703703703, 'mean_word_length': 4.474683544303797, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6375837', 'n_tokens_mistral': 277, 'n_tokens_neox': 253, 'n_words': 104}