<commit_before># batch-cluster

**Support external batch-mode tools within Node.js.**

[![npm version](https://badge.fury.io/js/batch-cluster.svg)](https://badge.fury.io/js/batch-cluster)
[![Build status](https://travis-ci.org/mceachen/batch-cluster.js.svg?branch=master)](https://travis-ci.org/mceachen/batch-cluster.js)
[![Build status](https://ci.appveyor.com/api/projects/status/4564x6lvc8s6a55l/branch/master?svg=true)](https://ci.appveyor.com/project/mceachen/batch-cluster-js/branch/master)

Many command line tools, like
[ExifTool](https://sno.phy.queensu.ca/~phil/exiftool/) and
[GraphicsMagick](http://www.graphicsmagick.org/), support running in a "batch
mode" that accept commands provided through stdin and results through stdout.
As these tools can be fairly large, spinning them up can be expensive
(especially on Windows).

This module expedites these commands, or "Tasks," by managing a cluster of
these "batch" processes, feeding tasks to idle processes, retrying tasks when
the tool crashes, and preventing memory leaks by restarting tasks after
performing a given number of tasks or after a given set of time has elapsed.

This package powers
[exiftool-vendored](https://github.com/mceachen/exiftool-vendored.js), whose
source you can examine as an example consumer.

## Installation

Depending on your yarn/npm preference:

```bash
$ yarn add batch-cluster
# or
$ npm install --save batch-cluster
```

## Usage

The child process must use `stdin` and `stdout` for control/response.
BatchCluster will ensure a given process is only given one task at a time.

_If these links are broken, use <https://batch-cluster.js.org/>_

1.  Create a singleton instance of [BatchCluster](/classes/_batchcluster_.batchcluster.html).

    Note the
    [constructor options](/classes/_batchcluster_.batchcluster.html#constructor) takes a union type of

    * [ChildProcessFactory](/interfaces/_batchcluster_.childprocessfactory.html) and
    * [BatchProcessOptions](/interfaces/_batchcluster_.batchprocessoptions.html), both of which have no
      defaults, and
    * [BatchClusterOptions](/classes/_batchcluster_.batchclusteroptions.html), which has defaults that may
      or may not be relevant to your application.

1.  The [default](/modules/_logger_.consolelogger.html) logger writes warning
    and error messages to `console.warn` and `console.error`. You can change
    this to your logger by using
    [setLogger](/modules/_logger_.html#setlogger).

1.  Implement the [Parser](/modules/_task_.html#parser) class to parse results from your child
    process.

1.  Construct a [Task](/classes/_task_.task.html) with the desired command and
    the parser you built in the previous step, and submit it to your
    BatchCluster singleton's
    [enqueueTask](/classes/_batchcluster_.batchcluster.html#enqueuetask)
    method.

See
[src/test.ts](https://github.com/mceachen/batch-cluster.js/blob/master/src/test.ts)
for an example child process. Note that the script is *designed* to be flaky
on order to test BatchCluster's retry and error handling code.

## Versioning

### The `MAJOR` or `API` version is incremented for

* 💔 Non-backwards-compatible API changes

### The `MINOR` or `UPDATE` version is incremented for

* ✨ Backwards-compatible features

### The `PATCH` version is incremented for

* 🐞 Backwards-compatible bug fixes
* 📦 Minor packaging changes

## Changelog

### v1.10.0

* 🐞 Explicitly use `timers.setInterval`. May address [this
  issue](https://stackoverflow.com/questions/48961238/electron-setinterval-implementation-difference-between-chrome-and-node).
  Thanks for the PR, [<NAME>](https://github.com/timfish)!

### v1.9.1

* 📦 Changed `BatchProcess.end()` to use `until()` rather than `Promise.race`,
  and always use `kill(pid, forced)` after waiting the shutdown grace period
  to prevent child process leaks.

### v1.9.0

* ✨ New `Logger.setLogger()` for debug, info, warning, and errors. `debug` and
  `info` defaults to Node's
  [debuglog](https://nodejs.org/api/util.html#util_util_debuglog_section),
  `warn` and `error` default to `console.warn` and `console.error`,
  respectively.<commit_msg>prep v1.11.0<commit_after># batch-cluster

**Support external batch-mode tools within Node.js.**

[![npm version](https://badge.fury.io/js/batch-cluster.svg)](https://badge.fury.io/js/batch-cluster)
[![Build status](https://travis-ci.org/mceachen/batch-cluster.js.svg?branch=master)](https://travis-ci.org/mceachen/batch-cluster.js)
[![Build status](https://ci.appveyor.com/api/projects/status/4564x6lvc8s6a55l/branch/master?svg=true)](https://ci.appveyor.com/project/mceachen/batch-cluster-js/branch/master)

Many command line tools, like
[ExifTool](https://sno.phy.queensu.ca/~phil/exiftool/) and
[GraphicsMagick](http://www.graphicsmagick.org/), support running in a "batch
mode" that accept commands provided through stdin and results through stdout.
As these tools can be fairly large, spinning them up can be expensive
(especially on Windows).

This module expedites these commands, or "Tasks," by managing a cluster of
these "batch" processes, feeding tasks to idle processes, retrying tasks when
the tool crashes, and preventing memory leaks by restarting tasks after
performing a given number of tasks or after a given set of time has elapsed.

This package powers
[exiftool-vendored](https://github.com/mceachen/exiftool-vendored.js), whose
source you can examine as an example consumer.

## Installation

Depending on your yarn/npm preference:

```bash
$ yarn add batch-cluster
# or
$ npm install --save batch-cluster
```

## Usage

The child process must use `stdin` and `stdout` for control/response.
BatchCluster will ensure a given process is only given one task at a time.

_If these links are broken, use <https://batch-cluster.js.org/>_

1.  Create a singleton instance of [BatchCluster](/classes/_batchcluster_.batchcluster.html).

    Note the
    [constructor options](/classes/_batchcluster_.batchcluster.html#constructor) takes a union type of

    * [ChildProcessFactory](/interfaces/_batchcluster_.childprocessfactory.html) and
    * [BatchProcessOptions](/interfaces/_batchcluster_.batchprocessoptions.html), both of which have no
      defaults, and
    * [BatchClusterOptions](/classes/_batchcluster_.batchclusteroptions.html), which has defaults that may
      or may not be relevant to your application.

1.  The [default](/modules/_logger_.consolelogger.html) logger writes warning
    and error messages to `console.warn` and `console.error`. You can change
    this to your logger by using
    [setLogger](/modules/_logger_.html#setlogger).

1.  Implement the [Parser](/modules/_task_.html#parser) class to parse results from your child
    process.

1.  Construct a [Task](/classes/_task_.task.html) with the desired command and
    the parser you built in the previous step, and submit it to your
    BatchCluster singleton's
    [enqueueTask](/classes/_batchcluster_.batchcluster.html#enqueuetask)
    method.

See
[src/test.ts](https://github.com/mceachen/batch-cluster.js/blob/master/src/test.ts)
for an example child process. Note that the script is *designed* to be flaky
on order to test BatchCluster's retry and error handling code.

## Versioning

### The `MAJOR` or `API` version is incremented for

* 💔 Non-backwards-compatible API changes

### The `MINOR` or `UPDATE` version is incremented for

* ✨ Backwards-compatible features

### The `PATCH` version is incremented for

* 🐞 Backwards-compatible bug fixes
* 📦 Minor packaging changes

## Changelog

### v1.11.0

* ✨ Added new `BatchClusterObserver` for error and lifecycle monitoring
* 📦 Added a number of additional logging calls

### v1.10.0

* 🐞 Explicitly use `timers.setInterval`. May address [this
  issue](https://stackoverflow.com/questions/48961238/electron-setinterval-implementation-difference-between-chrome-and-node).
  Thanks for the PR, [<NAME>](https://github.com/timfish)!

### v1.9.1

* 📦 Changed `BatchProcess.end()` to use `until()` rather than `Promise.race`,
  and always use `kill(pid, forced)` after waiting the shutdown grace period
  to prevent child process leaks.

### v1.9.0

* ✨ New `Logger.setLogger()` for debug, info, warning, and errors. `debug` and
  `info` defaults to Node's
  [debuglog](https://nodejs.org/api/util.html#util_util_debuglog_section),
  `warn` and `error` default to `console.warn` and `console.error`,
  respectively.
{'fraction_non_alphanumeric': 0.11477164605137964, 'fraction_numerical': 0.008325404376784015, 'mean_word_length': 4.572564612326044, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 26, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '155950', 'n_tokens_mistral': 2624, 'n_tokens_neox': 2521, 'n_words': 808}