<commit_before>node-logic-filter
=================

[![Build Status](https://travis-ci.org/phoebesimon/node-logic-filter.svg?branch=master)](https://travis-ci.org/phoebesimon/node-logic-filter)

logic-filter performs filtering using arbitrary rules on streams of JSON objects and outputs objects tagged with the label of the rule they matched. If an object matches more than one rule, it will be output multiple times with each of the labels.

#Example
For this example, assume you have a file with JSON objects in `exampleObjects.json` that look like:

```
{
  "a": 1,
  "b": 2
}
{
  "a": 1,
  "b": 3
} 
...
```

``` js
var LogicFilter = require('logic-filter');
var JSONstream = require('JSONstream');
var through = require('through');
var fs = require('fs');

var lf = new LogicFilter();

lf.add('testRule', {
  "and": {
    "a": 1,
    "not": {
      "b": 3
    }
  }
});

fs.createReadStream('./exampleObjects.json').pipe(JSONstream.parse()).pipe(lf).pipe(through(function(obj) {
  console.log(JSON.stringify(obj, null, 4));
}));

```

##Filter Language
Filter rules are JSON objects that tell LogicFilter which objects to allow through. LogicFilter considers a rule to be a match if the value for a key in the rule equals the value for the same key in the object in the stream. There are also a few reserved keywords used for logical expression construction: `and`, `or` and `not`. The following are examples of rules and a simple object they will match:

Rule:
```
{
    "a": 1,
    "b": "two"
}
```

Matches:
```
{
  "a": 1,
  "b": "two",
  "c": "three"
}
```
Note that in the above, "and" is implied when there is no operator

Rule:
This is equivalent to the previous example:
```
{
    "and": {
      "a": 1,
      "b": "two"
    }
}
```

Matches:
```
{
  "a": 1,
  "b": "two",
  "c": "three"
}
```

Rule:
```
{
    'or': {
      "a": 1,
      "b": "two"
    }
}
```
**Note this is logical or, so both may be true

Matches:
```
{
  "a": 1,
  "b": "one"
}
```
<commit_msg>Updated README with all the new rules<commit_after>node-logic-filter
=================

[![Build Status](https://travis-ci.org/phoebesimon/node-logic-filter.svg?branch=master)](https://travis-ci.org/phoebesimon/node-logic-filter)

logic-filter performs filtering using arbitrary rules on streams of JSON objects and outputs objects tagged with the label of the rule they matched. If an object matches more than one rule, it will be output multiple times with each of the labels.

#Example

``` js
var LogicFilter = require('logic-filter');
var through = require('through');

var lf = new LogicFilter();

lf.add('testFilter', {
  "and": {
    "a": 1,
    "not": {
      "b": 3
    }
  }
});

lf.write({"a": 1, "b": 2});
lf.write({"a": 1, "b": 3});
lf.write({"a": 1});
lf.write({"b": 2});
lf.write({});

lf.pipe(through(function(obj) {
  console.log(JSON.stringify(obj, null, 4));
}));

```
Which will output:
```
{
    "a": 1,
    "b": 2,
    "label": "testFilter"
}
{
    "a": 1,
    "label": "testFilter"
}
```


##Filter Language
Filter rules are JSON objects that tell LogicFilter which objects to allow through. LogicFilter considers a rule to be a match if the value for a key in the rule equals the value for the same key in the object in the stream. There are also a few reserved keywords used for logical expression construction: `and`, `or` and `not`, and two other keywords: `value` and `exists`. The following are examples of rules and a simple object they will match:

Rule: No operators
In this case, all the fields must be present in the compared object and equal to the values provided
```
{
    "a": 1,
    "b": "two"
}
```

Matches:
```
{
  "a": 1,
  "b": "two",
  "c": "three"
}
```
Note that in the above, "and" is implied when there is no operator

Rule: And
This is equivalent to the previous example:
```
{
    "and": {
      "a": 1,
      "b": "two"
    }
}
```

Matches:
```
{
  "a": 1,
  "b": "two",
  "c": "three"
}
```

Rule: Or
```
{
    "or": {
      "a": 1,
      "b": "two"
    }
}
```
**Note this is logical or, so both may be true

Matches:
```
{
  "a": 1,
  "b": "one"
}
```

Rule: Not
With `not`, note that `and` is implied when there is more than one key/value pair in the closure. In this case it must be true that !(a == 1 && b == "two").
```
{
  "not": {
    "a": 1,
    "b": "two"
  }
}
```

Matches:
```
{
  "a": 1,
  "b": "one"
}
{
  "a": 2,
  "b": "two"
}
{
  "c": 3
}
```

Rule: Any of several values
Use an array to specify that a field can be equal to any of several values:
```
{
  "a": [1, 2, 3]
}
```

Matches:
```
{
  "a": 1
}
{
  "a": 2
}
{
  "a": 3
}
```

Rule: Equals literal array
Use the `value` keyword to denote that you want to actually compare the value specified:
```
{
  "a": {
    "value": [1, 2, 3]
  }
}
```

Matches:
```
{
  "a": [1, 2, 3]
}
```

Rule: Compare fields in a nested object
In this example, `and` is implied. You can pass arbitrary operators in here as well
```
{
  "a": {
    "foo": "bar",
    "bar": "baz"
  }
}
```

Matches:
```
{
  "a": {
    "foo": "bar",
    "bar": "baz"
  }
}
```

Rule: Object deep equal
Using the `value` keyword will compare the entire value in the filter to the entire value in the JSON object and only pass if they are both deep equal
```
{
  "a": {
    "value": {
      "b": 2,
      "c": 3
    }
  }
}
```

Matches:
```
{
  "a": {
    "b": 2,
    "c": 3
  }
}
```
{'fraction_non_alphanumeric': 0.1795254997197833, 'fraction_numerical': 0.009714178965066317, 'mean_word_length': 3.16006216006216, 'pattern_counts': {'":': 82, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6731811', 'n_tokens_mistral': 2049, 'n_tokens_neox': 1909, 'n_words': 695}