<commit_before>=======================================
Blog posts and additional documentation
=======================================

Hashtable and filtering
=======================

The basic inexact-matching approach used by the hashtable code is
described in this blog post:

   http://ivory.idyll.org/blog/jul-10/kmer-filtering

A test data set (soil metagenomics, 88m reads, 10gb) is here:

   http://angus.ged.msu.edu.s3.amazonaws.com/88m-reads.fa.gz


The `filter-exact <http://github.com/ctb/khmer/blob/master/scripts/filter-exact.py>`__ script can be used as a starting point.

Illumina read abundance profiles
================================

khmer can be used to look at systematic variations in k-mer statistics
across Illumina reads; see, for example, this blog post:

   http://ivory.idyll.org/blog/jul-10/illumina-read-phenomenology

The `fasta-to-abundance-hist
<http://github.com/ctb/khmer/blob/master/sandbox/fasta-to-abundance-hist.py>`__
and `abundance-hist-by-position
<http://github.com/ctb/khmer/blob/master/sandbox/abundance-hist-by-position.py>`__
scripts can be used to generate the k-mer abundance profile data, after
loading all the k-mer counts into a .kh file::

   # first, load all the k-mer counts:
   scripts/load-into-counting.py -k 20 -x 1e7 25k.kh data/25k.fq.gz

   # then, build the '.freq' file that contains all of the counts by pos'n
   python sandbox/fasta-to-abundance-hist.py 25k.kh data/25k.fq.gz

   # sum across positions.
   python sandbox/abundance-hist-by-position.py data/25k.fq.gz.freq > out.dist

The hashtable method 'dump_kmers_by_abundance' can be used to dump
high abundance k-mers, but we don't have a script handy to do that yet.

You can assess high/low abundance k-mer distributions with the
`hi-lo-abundance-by-position script <http://github.com/ctb/khmer/blob/master/scripts/hi-lo-abundance-by-position.py>`__::

   scripts/load-into-counting.py -k 20 25k.kh data/25k.fq.gz
   python sandbox/hi-lo-abundance-by-position.py 25k.kh data/25k.fq.gz

This will produce two output files, <filename>.pos.abund=1 and
<filename>.pos.abund=255.<commit_msg>fixed pos'n to position - @mr-c suggestion<commit_after>=======================================
Blog posts and additional documentation
=======================================

Hashtable and filtering
=======================

The basic inexact-matching approach used by the hashtable code is
described in this blog post:

   http://ivory.idyll.org/blog/jul-10/kmer-filtering

A test data set (soil metagenomics, 88m reads, 10gb) is here:

   http://angus.ged.msu.edu.s3.amazonaws.com/88m-reads.fa.gz


The `filter-exact <http://github.com/ctb/khmer/blob/master/scripts/filter-exact.py>`__ script can be used as a starting point.

Illumina read abundance profiles
================================

khmer can be used to look at systematic variations in k-mer statistics
across Illumina reads; see, for example, this blog post:

   http://ivory.idyll.org/blog/jul-10/illumina-read-phenomenology

The `fasta-to-abundance-hist
<http://github.com/ctb/khmer/blob/master/sandbox/fasta-to-abundance-hist.py>`__
and `abundance-hist-by-position
<http://github.com/ctb/khmer/blob/master/sandbox/abundance-hist-by-position.py>`__
scripts can be used to generate the k-mer abundance profile data, after
loading all the k-mer counts into a .kh file::

   # first, load all the k-mer counts:
   scripts/load-into-counting.py -k 20 -x 1e7 25k.kh data/25k.fq.gz

   # then, build the '.freq' file that contains all of the counts by position
   python sandbox/fasta-to-abundance-hist.py 25k.kh data/25k.fq.gz

   # sum across positions.
   python sandbox/abundance-hist-by-position.py data/25k.fq.gz.freq > out.dist

The hashtable method 'dump_kmers_by_abundance' can be used to dump
high abundance k-mers, but we don't have a script handy to do that yet.

You can assess high/low abundance k-mer distributions with the
`hi-lo-abundance-by-position script <http://github.com/ctb/khmer/blob/master/scripts/hi-lo-abundance-by-position.py>`__::

   scripts/load-into-counting.py -k 20 25k.kh data/25k.fq.gz
   python sandbox/hi-lo-abundance-by-position.py 25k.kh data/25k.fq.gz

This will produce two output files, <filename>.pos.abund=1 and
<filename>.pos.abund=255.
{'fraction_non_alphanumeric': 0.1669798027242837, 'fraction_numerical': 0.018318459370596524, 'mean_word_length': 4.174969623329283, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1731434', 'n_tokens_mistral': 1531, 'n_tokens_neox': 1402, 'n_words': 386}