<commit_before>![Node.js CI](https://github.com/Borewit/strtok3/workflows/Node.js%20CI/badge.svg)
[![NPM version](https://badge.fury.io/js/strtok3.svg)](https://npmjs.org/package/strtok3)
[![npm downloads](http://img.shields.io/npm/dm/strtok3.svg)](https://npmcharts.com/compare/strtok3,token-types?start=1200&interval=30)
[![DeepScan grade](https://deepscan.io/api/teams/5165/projects/8526/branches/103329/badge/grade.svg)](https://deepscan.io/dashboard#view=project&tid=5165&pid=8526&bid=103329)
[![Known Vulnerabilities](https://snyk.io/test/github/Borewit/strtok3/badge.svg?targetFile=package.json)](https://snyk.io/test/github/Borewit/strtok3?targetFile=package.json)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/Borewit/strtok3.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Borewit/strtok3/alerts/)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/59dd6795e61949fb97066ca52e6097ef)](https://www.codacy.com/app/Borewit/strtok3?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=Borewit/strtok3&amp;utm_campaign=Badge_Grade)
[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/Borewit/strtok3.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Borewit/strtok3/context:javascript)
# strtok3

A promise based streaming [*tokenizer*](#tokenizer) for [Node.js](http://nodejs.org) and browsers.
This node module is a successor of [strtok2](https://github.com/Borewit/strtok2).

The `strtok3` contains a few methods to turn different input into a [*tokenizer*](#tokenizer). Designed to
*   Support a streaming environment
*   Decoding of binary data, strings and numbers in mind
*   Read [predefined](https://github.com/Borewit/token-types) or custom tokens.
*   Optimized [*tokenizers*](#tokenizer) for reading from [file](#method-strtok3fromfile), [stream](#method-strtok3fromstream) or [buffer](#method-strtok3frombuffer).

It can read from:
*   A file (taking a file path as an input)
*   A Node.js [stream](https://nodejs.org/api/stream.html).
*   A [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array)
*   HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).
*   Chunked [Amazon S3](https://aws.amazon.com/s3) access provided by [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).

## Installation

```sh
npm install strtok3
```

## API

Use one of the methods to instantiate an [*abstract tokenizer*](#tokenizer):
*   [strtok3.fromFile](#method-strtok3fromfile)
*   [strtok3.fromStream](#method-strtok3fromstream)
*   [strtok3.fromBuffer](#method-strtok3fromBuffer)
*   [strtok3.fromUint8Array](#method-strtok3fromUint8Array)

### strtok3 methods

All of the strtok3 methods return a [*tokenizer*](#tokenizer), either directly or via a promise.

#### Method `strtok3.fromFile()`<commit_msg>README: Install `@tokenizer/token` for TypeScript<commit_after>![Node.js CI](https://github.com/Borewit/strtok3/workflows/Node.js%20CI/badge.svg)
[![NPM version](https://badge.fury.io/js/strtok3.svg)](https://npmjs.org/package/strtok3)
[![npm downloads](http://img.shields.io/npm/dm/strtok3.svg)](https://npmcharts.com/compare/strtok3,token-types?start=1200&interval=30)
[![DeepScan grade](https://deepscan.io/api/teams/5165/projects/8526/branches/103329/badge/grade.svg)](https://deepscan.io/dashboard#view=project&tid=5165&pid=8526&bid=103329)
[![Known Vulnerabilities](https://snyk.io/test/github/Borewit/strtok3/badge.svg?targetFile=package.json)](https://snyk.io/test/github/Borewit/strtok3?targetFile=package.json)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/Borewit/strtok3.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Borewit/strtok3/alerts/)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/59dd6795e61949fb97066ca52e6097ef)](https://www.codacy.com/app/Borewit/strtok3?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=Borewit/strtok3&amp;utm_campaign=Badge_Grade)
[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/Borewit/strtok3.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Borewit/strtok3/context:javascript)
# strtok3

A promise based streaming [*tokenizer*](#tokenizer) for [Node.js](http://nodejs.org) and browsers.
This node module is a successor of [strtok2](https://github.com/Borewit/strtok2).

The `strtok3` contains a few methods to turn different input into a [*tokenizer*](#tokenizer). Designed to
*   Support a streaming environment
*   Decoding of binary data, strings and numbers in mind
*   Read [predefined](https://github.com/Borewit/token-types) or custom tokens.
*   Optimized [*tokenizers*](#tokenizer) for reading from [file](#method-strtok3fromfile), [stream](#method-strtok3fromstream) or [buffer](#method-strtok3frombuffer).

It can read from:
*   A file (taking a file path as an input)
*   A Node.js [stream](https://nodejs.org/api/stream.html).
*   A [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array)
*   HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).
*   Chunked [Amazon S3](https://aws.amazon.com/s3) access provided by [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).

## Installation

```sh
npm install strtok3
```

Using TypeScript you should also install [@tokenizer/token](https://github.com/Borewit/tokenizer-token) as a development
dependency:

```shell
npm install --save-dev @tokenizer/token
```

## API

Use one of the methods to instantiate an [*abstract tokenizer*](#tokenizer):
*   [strtok3.fromFile](#method-strtok3fromfile)
*   [strtok3.fromStream](#method-strtok3fromstream)
*   [strtok3.fromBuffer](#method-strtok3fromBuffer)
*   [strtok3.fromUint8Array](#method-strtok3fromUint8Array)

### strtok3 methods

All of the strtok3 methods return a [*tokenizer*](#tokenizer), either directly or via a promise.

#### Method `strtok3.fromFile()`
{'fraction_non_alphanumeric': 0.16760145166611679, 'fraction_numerical': 0.03365225998020455, 'mean_word_length': 6.099531615925058, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 45, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2571844', 'n_tokens_mistral': 2366, 'n_tokens_neox': 2172, 'n_words': 315}