<commit_before>#filterbank

**filterbank** is a tool for generating datasets at multiple resolutions. The resulting files then allow for quick summarisation and viewing.

##Installation

    pip install http://github.com/malariagen/filterbank/raw/master/dist/filterbank-0.1.0.tar.gz

## Usage

    filterbank [-h] [--verbose] input_file output_directory -c yaml_config

A set of files (one data file and one yaml describing the data file for each channel at each resolution level) will be produced in the output directory. The input file is a tab-delimted file with a header row naming the columns.


## Config
The config is a yaml file with several sections:
    block_sizes:  #The resolution levels at which to output files
      start:      #Integer of smallest block size - e.g. 1 will output 1 line for each line in the input file
      end:        #Integer of largest allowed block size
      mult:       #Integer that is the ratio between successive block sizes
    #For example 1,16,4 for these numbers would produce block sizes of 1,4,16

    channels:     #A associative array defining the outputs
      LONG NAME:    #Name of this channel
        ANYTHING:     #Arbitary metadata that is passed through to the output YAML files
        short_name:   #short name used in output file name
        value:        #Python expression that gives the value of the channel, available variables
                      #are the column names
        accumulators: #A list of classes that determine how to reduce a block of values to one.
                      #Currently avaliable: [LastVal, ArithmeticMean, GeometricMean, Median, 
                      #Percentile, Min, Max] Specify the class with a string, if it needs args 
                      #specify the class with an associative array.
                      #E.g. [{'Percentile': {'percent':0.25}}, 'Max', 'Min'] would give you the
                      # 1st quartile, max and min as output.
        encoder:      #Specify the class used to output the channel. Either FixedWidthBase64 (for
                      #DQX webapp) or TabDelimited. Again either as a string or associative array
                      #with args.
 
###Example config:

        block_sizes:
          start: 1
          end: 1048576
          mult: 8

        defaults: &defaults
          accumulators: [Min, GeometricMean, Max]
          encoder:
            FixedLengthB64:
              length: 3
              range: [0,100]

        channels:
        ##### CORE CHANNELS #####
            Number of reads:<commit_msg>Add note<commit_after>#filterbank

**filterbank** is a tool for generating datasets at multiple resolutions. The resulting files then allow for quick summarisation and viewing.

##Installation

    pip install http://github.com/malariagen/filterbank/raw/master/dist/filterbank-0.1.0.tar.gz

## Usage

    filterbank [-h] [--verbose] input_file output_directory -c yaml_config

A set of files (one data file and one yaml describing the data file for each channel at each resolution level) will be produced in the output directory. The input file is a tab-delimted file with a header row naming the columns.


## Config
The config is a yaml file with several sections:
    block_sizes:  #The resolution levels at which to output files
      start:      #Integer of smallest block size - e.g. 1 will output 1 line for each line in the input file
      end:        #Integer of largest allowed block size
      mult:       #Integer that is the ratio between successive block sizes
    #For example 1,16,4 for these numbers would produce block sizes of 1,4,16

    channels:     #A associative array defining the outputs
      LONG NAME:    #Name of this channel
        ANYTHING:     #Arbitary metadata that is passed through to the output YAML files
        short_name:   #short name used in output file name
        value:        #Python expression that gives the value of the channel, available variables
                      #are the column names
        accumulators: #A list of classes that determine how to reduce a block of values to one.
                      #Currently avaliable: [LastVal, ArithmeticMean, GeometricMean, Median, 
                      #Percentile, Min, Max] Specify the class with a string, if it needs args 
                      #specify the class with an associative array.
                      #E.g. [{'Percentile': {'percent':0.25}}, 'Max', 'Min'] would give you the
                      # 1st quartile, max and min as output.
                      #Note that for a block size of 1 only the LastVal accumulator will be used.
        encoder:      #Specify the class used to output the channel. Either FixedWidthBase64 (for
                      #DQX webapp) or TabDelimited. Again either as a string or associative array
                      #with args.
 
###Example config:

        block_sizes:
          start: 1
          end: 1048576
          mult: 8

        defaults: &defaults
          accumulators: [Min, GeometricMean, Max]
          encoder:
            FixedLengthB64:
              length: 3
              range: [0,100]

        channels:
        ##### CORE CHANNELS #####
            Number of reads:
{'fraction_non_alphanumeric': 0.06109510086455332, 'fraction_numerical': 0.013640730067243036, 'mean_word_length': 2.189950980392157, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7671530', 'n_tokens_mistral': 1393, 'n_tokens_neox': 1256, 'n_words': 619}