<commit_before># readdirp [![Build Status](https://secure.travis-ci.org/thlorenz/readdirp.png)](http://travis-ci.org/thlorenz/readdirp)

Recursive version of [fs.readdir](http://nodejs.org/docs/latest/api/fs.html#fs_fs_readdir_path_callback). Exposes a
**stream api**.

```javascript
var readdirp = require('readdirp'); 
  , path = require('path')
  , es = require('event-stream');

// print out all JavaScript files along with their size

readdirp({ root: path.join(__dirname), fileFilter: '*.js' })
  .on('warn', function (err) { console.error('non-fatal error', err); })
  .on('error', function (err) { console.error('fatal error', err); })
  .pipe(es.mapSync(function (entry) { 
    return { path: entry.path, size: entry.stat.size };
  }))
  .pipe(es.stringify())
  .pipe(process.stdout);
```

Meant to be one of the recursive versions of [fs](http://nodejs.org/docs/latest/api/fs.html) functions, e.g., like [mkdirp](https://github.com/substack/node-mkdirp).

**Table of Contents**  *generated with [DocToc](http://doctoc.herokuapp.com/)*

- [Installation](#installation)
- [API](#api)
	- [entry stream](#entry-stream)
	- [options](#options)
	- [entry info](#entry-info)
	- [Filters](#filters)
	- [Callback API](#callback-api)
		- [allProcessed ](#allprocessed)
		- [fileProcessed](#fileprocessed)
- [More Examples](#more-examples)
	- [stream api](#stream-api)
	- [stream api pipe](#stream-api-pipe)
	- [grep](#grep)
	- [using callback api](#using-callback-api)
	- [tests](#tests)


# Installation

    npm install readdirp

# API

***var entryStream = readdirp (options)***

Reads given root recursively and returns a `stream` of [entry info](#entry-info)s.

## entry stream

Behaves as follows:
  
- `emit('data')` passes an [entry info](#entry-info) whenever one is found
- `emit('warn')` passes a non-fatal `Error` that prevents a file/directory from being processed (i.e., if it is
  inaccessible to the user)
- `emit('error')` passes a fatal `Error` which also ends the stream (i.e., when illegal options where passed)
- `emit('end')` called when all entries were found and no more will be emitted (i.e., we are done)
- the stream is `paused` initially in order to allow `pipe` and `on` handlers be connected before data or errors are
  emitted
- the stream is `resumed` automatically during the next event loop 
- to learn more about streams, consult the [stream-handbook](https://github.com/substack/stream-handbook)

## options
    
- **root**: path in which to start reading and recursing into subdirectories

- **fileFilter**: filter to include/exclude files found (see [Filters](#filters) for more)
<commit_msg>adding stream.destroy info to readme<commit_after># readdirp [![Build Status](https://secure.travis-ci.org/thlorenz/readdirp.png)](http://travis-ci.org/thlorenz/readdirp)

Recursive version of [fs.readdir](http://nodejs.org/docs/latest/api/fs.html#fs_fs_readdir_path_callback). Exposes a
**stream api**.

```javascript
var readdirp = require('readdirp'); 
  , path = require('path')
  , es = require('event-stream');

// print out all JavaScript files along with their size

var stream = readdirp({ root: path.join(__dirname), fileFilter: '*.js' });
stream
  .on('warn', function (err) { 
    console.error('non-fatal error', err); 
    // optionally call stream.destroy() here in order to abort and cause 'close' to be emitted
  })
  .on('error', function (err) { console.error('fatal error', err); })
  .pipe(es.mapSync(function (entry) { 
    return { path: entry.path, size: entry.stat.size };
  }))
  .pipe(es.stringify())
  .pipe(process.stdout);
```

Meant to be one of the recursive versions of [fs](http://nodejs.org/docs/latest/api/fs.html) functions, e.g., like [mkdirp](https://github.com/substack/node-mkdirp).

**Table of Contents**  *generated with [DocToc](http://doctoc.herokuapp.com/)*

- [Installation](#installation)
- [API](#api)
	- [entry stream](#entry-stream)
	- [options](#options)
	- [entry info](#entry-info)
	- [Filters](#filters)
	- [Callback API](#callback-api)
		- [allProcessed ](#allprocessed)
		- [fileProcessed](#fileprocessed)
- [More Examples](#more-examples)
	- [stream api](#stream-api)
	- [stream api pipe](#stream-api-pipe)
	- [grep](#grep)
	- [using callback api](#using-callback-api)
	- [tests](#tests)


# Installation

    npm install readdirp

# API

***var entryStream = readdirp (options)***

Reads given root recursively and returns a `stream` of [entry info](#entry-info)s.

## entry stream

Behaves as follows:
  
- `emit('data')` passes an [entry info](#entry-info) whenever one is found
- `emit('warn')` passes a non-fatal `Error` that prevents a file/directory from being processed (i.e., if it is
  inaccessible to the user)
- `emit('error')` passes a fatal `Error` which also ends the stream (i.e., when illegal options where passed)
- `emit('end')` called when all entries were found and no more will be emitted (i.e., we are done)
- `emit('close')` called when the stream is destroyed via `stream.destroy()` (which could be useful if you want to
  manually abort even on a non fatal error) - no more entries, warning or errors are emitted at that point
- the stream is `paused` initially in order to allow `pipe` and `on` handlers be connected before data or errors are
  emitted
- the stream is `resumed` automatically during the next event loop 
- to learn more about streams, consult the [stream-handbook](https://github.com/substack/stream-handbook)

## options
    
- **root**: path in which to start reading and recursing into subdirectories

- **fileFilter**: filter to include/exclude files found (see [Filters](#filters) for more)

{'fraction_non_alphanumeric': 0.15567798116225343, 'fraction_numerical': 0.0, 'mean_word_length': 4.375358166189112, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7498677', 'n_tokens_mistral': 1875, 'n_tokens_neox': 1776, 'n_words': 553}