<commit_before>Filtri
======

Filtri is a tiny DSL for text substitution.

## Installation

    $ gem install filtri

## Usage

Define a set of substitution rules using `rule` and apply to an input string:


    f = filtri do
      rule /fo+/ => "bar"
      rule "baz" => "bug"
    end

    f.apply("foobazbarfoo")
    => "barbugbarbar"

Rules can also be read from a file:

File `foo.rules`:

     # this is a comment
     rule /fo+bar/ => "bazbag"
     rule /(a)+/ => '!!\1!!'
     rule /(b)+/ => '**\1**'

Load with `Filtri::load`:

    Filtri.load("foo.rules").apply("foobarfoobar")
    => "**b**!!a!!z**b**!!a!!g**b**!!a!!z**b**!!a!!g"

## Command line tool

A command line tool is included in this gem, `filtri`

### Usage

    Usage: filtri [options] [input]

    Options:
        -r, --rule STRING                Single rule
        -f, --file RULE_FILE             File containing rules
        -v, --version                    Version information
        -h, --help                       Help


### Examples

An example applying rules from the files `f1.rule` and `f2.rule` to the input files `input1` and `input2`

File `f1.rules`:

    rule /(foo+)/ => 'THIS WAS \1'
    rule /(ba+r)/ => 'THIS WAS \1'

File `f1.rules`:

    rule /(ba+z)/ => 'THIS WAS \1'
    rule /bag/ => 'THIS WAS BAG'

File `input1`:

    foooo
    baaaaaaaaaaaaaaaaz
    bag

File `input2`:

    and baaaaaaaaaaar

Running `filtri` as

    $ filtri -f f1.rules -f f2.rules input1 input2 > result.txt

..gives `result.txt`:

    THIS WAS foooo
    THIS WAS baaaaaaaaaaaaaaaaz
    THIS WAS BAG
    and THIS WAS baaaaaaaaaaar

Single rules can be provided using the option `-r`:

    $ echo "foo" | filtri -r '"foo" => "bar"'
    bar


## Syntax highlighting

Definitions for syntax highlighting of `.rule` files in Sublime Text 2 and TextMate is available (`resource/filtri_rules.tmLanguage`)

## Version information

* 0.1.0 - Added filtri tool and syntax highlighting
* 0.0.1 - Initial version<commit_msg>README update<commit_after>Filtri
======

Filtri is a tiny DSL for text substitution.

## Installation

    $ gem install filtri

## Usage

Define a set of substitution rules using `rule` and apply to an input string:


    f = filtri do
      rule /fo+/ => "bar"
      rule "baz" => "bug"
    end

    f.apply("foobazbarfoo")
    => "barbugbarbar"

Rules can also be read from a file:

File `foo.rules`:

     # this is a comment
     rule /fo+bar/ => "bazbag"
     rule /(a)+/ => '!!\1!!'
     rule /(b)+/ => '**\1**'

Load with `Filtri::load`:

    Filtri.load("foo.rules").apply("foobarfoobar")
    => "**b**!!a!!z**b**!!a!!g**b**!!a!!z**b**!!a!!g"

## Command line tool

A command line tool is included in this gem, `filtri`

### Usage

    Usage: filtri [options] [input]

    Options:
        -r, --rule STRING                Single rule
        -f, --file RULE_FILE             File containing rules
        -v, --version                    Version information
        -h, --help                       Help


### Examples

An example applying rules from the files `f1.rule` and `f2.rule` to the input files `input1` and `input2`

File `f1.rules`:

    rule /(foo+)/ => 'THIS WAS \1'
    rule /(ba+r)/ => 'THIS WAS \1'

File `f1.rules`:

    rule /(ba+z)/ => 'THIS WAS \1'
    rule /bag/ => 'THIS WAS BAG'

File `input1`:

    foooo
    baaaaaaaaaaaaaaaaz
    bag

File `input2`:

    and baaaaaaaaaaar

Running `filtri` as

    $ filtri -f f1.rules -f f2.rules input1 input2 > result.txt

..gives `result.txt`:

    THIS WAS foooo
    THIS WAS baaaaaaaaaaaaaaaaz
    THIS WAS BAG
    and THIS WAS baaaaaaaaaaar

Single rules can be provided using the option `-r`:

    $ echo "foo" | filtri -r '"foo" => "bar"'
    bar


## Syntax highlighting

Definitions for syntax highlighting of `.rule` files in Sublime Text 2 and TextMate is available (`resource/filtri_rules.tmLanguage`)

## Version information

* 0.1.2 - Bugfixes
* 0.1.0 - Added filtri tool and syntax highlighting
* 0.0.1 - Initial version
{'fraction_non_alphanumeric': 0.14110582937202903, 'fraction_numerical': 0.012759569677257944, 'mean_word_length': 2.3738396624472573, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1231722', 'n_tokens_mistral': 1475, 'n_tokens_neox': 1349, 'n_words': 458}