<commit_before># pull-stream

Minimal Pipeable Pull-stream

In [classic-streams](1),
streams _push_ data to the next stream in the pipeline.
In [new-streams](https://github.com/joyent/node/blob/v0.10/doc/api/stream.markdown),
data is pulled out of the source stream, into the destination.
In [new-classic-streams](
`pull-stream` is a minimal take on streams,
pull streams work great for "object" streams as well as streams of raw text or binary data.


## Quick Example

Stat some files:

```js
pull(
  pull.values(['file1', 'file2', 'file3']),
  pull.asyncMap(fs.stat),
  pull.collect(function (err, array) {
    console.log(array)
  })
)
```
note that `pull(a, b, c)` is basically the same as `a.pipe(b).pipe(c)`.

The best thing about pull-stream is that it can be completely lazy.
This is perfect for async traversals where you might want to stop early.

## Compatibily with node streams

pull-streams are not _directly_ compatible with node streams,
but pull-streams can be converted into node streams with
[pull-stream-to-stream](https://github.com/dominictarr/pull-stream-to-stream)
and node streams can be converted into pull-stream using [stream-to-pull-stream](https://github.com/dominictarr/stream-to-pull-stream)


### Readable & Reader vs. Readable & Writable

Instead of a readable stream, and a writable stream, there is a `readable` stream,
 (aka "Source") and a `reader` stream (aka "Sink"). Through streams
is a Sink that returns a Source.

See also:
* [Sources](https://github.com/dominictarr/pull-stream/blob/master/docs/sources.md)
* [Throughs](https://github.com/dominictarr/pull-stream/blob/master/docs/throughs.md)
* [Sinks](https://github.com/dominictarr/pull-stream/blob/master/docs/sinks.md)

### Source (aka, Readable)

The readable stream is just a `function read(end, cb)`,
that may be called many times,
and will (asynchronously) `cb(null, data)` once for each call.

To signify an end state, the stream eventually returns `cb(err)` or `cb(true)`.
When indicating a terminal state, `data` *must* be ignored.

The `read` function *must not* be called until the previous call has called back.
Unless, it is a call to abort the stream (`read(truthy, cb)`).

```js
//a stream of 100 random numbers.
var i = 100
var random = function () {
  return function (end, cb) {
    if(end) return cb(end)
    //only read 100 times
    if(i-- < 0) return cb(true)
    cb(null, Math.random())
  }
}

```

### Sink; (aka, Reader, "writable")

A sink is just a `reader` function that calls a Source (read function),
until it decideds to stop, or the readable ends. `cb(err || true)`

All [Throughs](https://github.com/dominictarr/pull-stream/blob/master/docs/throughs.md)
and [Sinks](https://github.com/dominictarr/pull-stream/blob/master/docs/sinks.md)
are reader streams.

```js
//read source and log it.
var logger = function (read) {
  read(null, function next(end, data) {
    if(end === true) return
    if(end) throw end

    console.log(data)
    read(null, next)
  })
}
```

Since these are just functions, you can pass them to each other!

```js
var rand = random()
var log = logger()

log(rand) //"pipe" the streams.

```

but, it's easier to read if you use's pull-stream's `pull` method

```js
var pull = require('pull-stream')

pull(random(), logger())<commit_msg>fix(docs): log(rand) example is working now<commit_after># pull-stream

Minimal Pipeable Pull-stream

In [classic-streams](1),
streams _push_ data to the next stream in the pipeline.
In [new-streams](https://github.com/joyent/node/blob/v0.10/doc/api/stream.markdown),
data is pulled out of the source stream, into the destination.
In [new-classic-streams](
`pull-stream` is a minimal take on streams,
pull streams work great for "object" streams as well as streams of raw text or binary data.


## Quick Example

Stat some files:

```js
pull(
  pull.values(['file1', 'file2', 'file3']),
  pull.asyncMap(fs.stat),
  pull.collect(function (err, array) {
    console.log(array)
  })
)
```
note that `pull(a, b, c)` is basically the same as `a.pipe(b).pipe(c)`.

The best thing about pull-stream is that it can be completely lazy.
This is perfect for async traversals where you might want to stop early.

## Compatibily with node streams

pull-streams are not _directly_ compatible with node streams,
but pull-streams can be converted into node streams with
[pull-stream-to-stream](https://github.com/dominictarr/pull-stream-to-stream)
and node streams can be converted into pull-stream using [stream-to-pull-stream](https://github.com/dominictarr/stream-to-pull-stream)


### Readable & Reader vs. Readable & Writable

Instead of a readable stream, and a writable stream, there is a `readable` stream,
 (aka "Source") and a `reader` stream (aka "Sink"). Through streams
is a Sink that returns a Source.

See also:
* [Sources](https://github.com/dominictarr/pull-stream/blob/master/docs/sources.md)
* [Throughs](https://github.com/dominictarr/pull-stream/blob/master/docs/throughs.md)
* [Sinks](https://github.com/dominictarr/pull-stream/blob/master/docs/sinks.md)

### Source (aka, Readable)

The readable stream is just a `function read(end, cb)`,
that may be called many times,
and will (asynchronously) `cb(null, data)` once for each call.

To signify an end state, the stream eventually returns `cb(err)` or `cb(true)`.
When indicating a terminal state, `data` *must* be ignored.

The `read` function *must not* be called until the previous call has called back.
Unless, it is a call to abort the stream (`read(truthy, cb)`).

```js
//a stream of 100 random numbers.
var i = 100
var random = function () {
  return function (end, cb) {
    if(end) return cb(end)
    //only read 100 times
    if(i-- < 0) return cb(true)
    cb(null, Math.random())
  }
}

```

### Sink; (aka, Reader, "writable")

A sink is just a `reader` function that calls a Source (read function),
until it decideds to stop, or the readable ends. `cb(err || true)`

All [Throughs](https://github.com/dominictarr/pull-stream/blob/master/docs/throughs.md)
and [Sinks](https://github.com/dominictarr/pull-stream/blob/master/docs/sinks.md)
are reader streams.

```js
//read source and log it.
var logger = function () {
  return function (read) {
    read(null, function next(end, data) {
      if(end === true) return
      if(end) throw end

      console.log(data)
      read(null, next)
    })
  }
}
```

Since these are just functions, you can pass them to each other!

```js
var rand = random()
var log = logger()

log(rand) //"pipe" the streams.

```

but, it's easier to read if you use's pull-stream's `pull` method

```js
var pull = require('pull-stream')

pull(random(), logger())
{'fraction_non_alphanumeric': 0.13456623064200873, 'fraction_numerical': 0.005112013231093069, 'mean_word_length': 4.097318007662835, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3151056', 'n_tokens_mistral': 2223, 'n_tokens_neox': 2148, 'n_words': 768}