<commit_before>Dissolve
========

Parse and consume binary streams with a neat DSL.

Overview
--------

Dissolve allows you to parse packed binary data into numbers, buffers, strings
and more*! With a simple syntax inspired by [node-binary](https://github.com/substack/node-binary)
and a solid, minimal implementation, you can be up and running in no time.

(* implementing "more" is left as an exercise to the reader)

If you want to produce binary data, might I suggest [concentrate](https://github.com/deoxxa/concentrate)?

Features
--------

* Accurate handling of [u]int{8,16,32} numbers in both signed and unsigned
  variants using fast, built-in [Buffer](http://nodejs.org/docs/latest/api/buffer.html)
  methods
* Fast approximation of [u]int64 numbers in signed and unsigned variants
* Extendable base class for building your own parsers and implementing
  custom types
* Tiny (~250 LoC) implementation, allowing for easy debugging

Installation
------------

Available via [npm](http://npmjs.org/):

> $ npm install dissolve

Or via git:

> $ git clone git://github.com/deoxxa/dissolve.git node_modules/dissolve

Usage
-----

Also see [example.js](https://github.com/deoxxa/dissolve/blob/master/example.js),
[example-complex.js](https://github.com/deoxxa/dissolve/blob/master/example-complex.js)
and [example-loop.js](https://github.com/deoxxa/dissolve/blob/master/example-loop.js).

```javascript
#!/usr/bin/env node

var Dissolve = require("./index");

var parser = Dissolve().loop(function(end) {
  this.uint8("id").tap(function() {
    if (this.vars.id === 0x01) {
      this.uint16be("a").uint16be("b");
    } else if (this.vars.id === 0x02) {
      this.uint32be("x").uint32be("y");
    }
  }).tap(function() {
    this.emit("data", this.vars);
    this.vars = {};
  });
});

parser.on("data", function(e) {
  console.log(e);
});

parser.write(new Buffer([0x01, 0x00, 0x02, 0x00, 0x03])); // {id: 1, a: 2, b: 3}
parser.write(new Buffer([0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x05])); // {id: 2, x: 4, y: 5}
parser.write(new Buffer([0x01]));
parser.write(new Buffer([0x00, 0x02, 0x00]));
parser.write(new Buffer([0x03])); // {id: 1, a: 2, b: 3}
```

Methods
-------

All parser methods are chainable and return the same parser instance they were
called on.

Tap
---

`tap(name, callback)`

This method allows you to "tap" into the parser at an arbitrary point. The
callback will be called bound to the parser instance, so you can use parser
methods on `this`. Any additional parser steps you introduce inside the callback
will be executed before any existing steps that are already scheduled to run
after the `tap` call.
<commit_msg>update docs for 0.10 stream API<commit_after>Dissolve
========

Parse and consume binary streams with a neat DSL.

Overview
--------

Dissolve allows you to parse packed binary data into numbers, buffers, strings
and more*! With a simple syntax inspired by [node-binary](https://github.com/substack/node-binary)
and a solid, minimal implementation, you can be up and running in no time.

(* implementing "more" is left as an exercise to the reader)

If you want to produce binary data, might I suggest [concentrate](https://github.com/deoxxa/concentrate)?

Features
--------

* Accurate handling of [u]int{8,16,32} numbers in both signed and unsigned
  variants using fast, built-in [Buffer](http://nodejs.org/docs/latest/api/buffer.html)
  methods
* Fast approximation of [u]int64 numbers in signed and unsigned variants
* Extendable base class for building your own parsers and implementing
  custom types
* Tiny (~250 LoC) implementation, allowing for easy debugging

Installation
------------

Available via [npm](http://npmjs.org/):

> $ npm install dissolve

Or via git:

> $ git clone git://github.com/deoxxa/dissolve.git node_modules/dissolve

Usage
-----

Also see [example.js](https://github.com/deoxxa/dissolve/blob/master/example.js),
[example-complex.js](https://github.com/deoxxa/dissolve/blob/master/example-complex.js)
and [example-loop.js](https://github.com/deoxxa/dissolve/blob/master/example-loop.js).

```javascript
#!/usr/bin/env node

var Dissolve = require("./index");

var parser = Dissolve().loop(function(end) {
  this.uint8("id").tap(function() {
    if (this.vars.id === 0x01) {
      this.uint16be("a").uint16be("b");
    } else if (this.vars.id === 0x02) {
      this.uint32be("x").uint32be("y");
    }
  }).tap(function() {
    this.push(this.vars);
    this.vars = {};
  });
});

parser.on("readable", function() {
  var e;
  while (e = parser.read()) {
    console.log(e);
  }
});

parser.write(new Buffer([0x01, 0x00, 0x02, 0x00, 0x03])); // {id: 1, a: 2, b: 3}
parser.write(new Buffer([0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x05])); // {id: 2, x: 4, y: 5}
parser.write(new Buffer([0x01]));
parser.write(new Buffer([0x00, 0x02, 0x00]));
parser.write(new Buffer([0x03])); // {id: 1, a: 2, b: 3}
```

Methods
-------

All parser methods are chainable and return the same parser instance they were
called on.

Tap
---

`tap(name, callback)`

This method allows you to "tap" into the parser at an arbitrary point. The
callback will be called bound to the parser instance, so you can use parser
methods on `this`. Any additional parser steps you introduce inside the callback
will be executed before any existing steps that are already scheduled to run
after the `tap` call.

{'fraction_non_alphanumeric': 0.1543186895011169, 'fraction_numerical': 0.034437825763216676, 'mean_word_length': 3.9158279963403477, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '437298', 'n_tokens_mistral': 1917, 'n_tokens_neox': 1744, 'n_words': 567}