<commit_before>## s3-streamlogger
[![NPM version](https://badge.fury.io/js/s3-streamlogger.svg)](http://badge.fury.io/js/s3-streamlogger)


A Writable Stream object that uploads to s3 objects, periodically rotating to a
new object name.

See also [tails3](http://github.com/coggle/tails3) for a script to tail the log
files produced by s3-streamlogger.

### Installation
```bash
npm install --save s3-streamlogger
```

### Basic Usage
```js
var S3StreamLogger = require('s3-streamlogger').S3StreamLogger;

var s3stream = new S3SteamLogger({
               bucket: "mys3bucket",
        access_key_id: "...",
    secret_access_key: "..."
});

s3stream.write("hello S3");
```

### Use with Winston: Log to S3
```sh
npm install --save winston
npm install --save s3-streamlogger
```

```js
var winston        = require('winston');
var S3StreamLogger = require('s3-streamlogger').S3StreamLogger;

var s3_stream = new S3SteamLogger({
             bucket: "mys3bucket",
      access_key_id: "...",
  secret_access_key: "..."
});

var logger = new (winston.Logger)({
  transports: [
    new (winston.transports.File)({
      stream: s3_stream
    })
  ]
});

logger.info('Hello Winston!');
```

### Add hostname information for tails3
tails3 expects messages to be logged as json (the default for the file
transport), with hostname and (for critical errors), stack properties to each
log object, in addition to the standard timestamp, level and message
properties. You can provide these using the third "metadata" option to
winston's log method:

```js
logger.log(level, message, {hostname: ... , stack: ...});
```

### Handling logging errors
When there is an error writing to s3, the stream emits an 'error' event with
details. You should take care **not** to log these errors back to the same
stream (as that is likely to cause infinite recursion). Instead log them to the
console, to a file, or to SNS using [winston-sns](https://github.com/jesseditson/winston-sns).


### Options

#### bucket *(required)*
Name of the S3 bucket to upload data to. Must exist.
Can also be provided as the environment variable `BUCKET_NAME`.

#### access_key_id *(required)*
AWS access key ID, must have putObject permission on the specified bucket.
Can also be provided as the environment variable `AWS_SECRET_ACCESS_KEY`.

#### secret_access_key *(required)*
AWS secret key for the `access_key_id` specified.
Can also be provided as the environment variable `AWS_SECRET_KEY_ID`.

#### name_format
Format of file names to create, accepts [strftime specifiers](https://github.com/samsonjs/strftime). Defaults to `"%Y-%m-%d-%H-%M-unknown-unknown.log"`. The Date() used to fill the format specifiers is created with the current UTC time, but still *has the current timezone*, so any specifiers that perform timezone conversion will return incorrect dates.

If you use a format of the form `%Y-%m-%d-%H-%M-<stage>-<hostname>.log`, then
you can use [tails3](http://github.com/coggle/tails3) to tail the log files
being generated by `S3StreamLogger`.

#### rotate_every
Files will be rotated every `rotate_every` milliseconds. Defaults to 3600000 (60
minutes).

#### max_file_size
Files will be rotated when they reach `max_file_size` bytes. Defaults to 200 KB.

#### upload_every
Files will be uploaded every `upload_every` milliseconds. Defaults to 20
seconds.

#### buffer_size
Files will be uploaded if the un-uploaded data exceeds `buffer_size` bytes.
Defaults to 10 KB.

#### server_side_encryption
The server side encryption `AES256` algorithm used when storing objects in S3.
Defaults to false.


### License
[ISC](http://opensource.org/licenses/ISC): equivalent to 2-clause BSD.

<commit_msg>fix typo in readme<commit_after>## s3-streamlogger
[![NPM version](https://badge.fury.io/js/s3-streamlogger.svg)](http://badge.fury.io/js/s3-streamlogger)


A Writable Stream object that uploads to s3 objects, periodically rotating to a
new object name.

See also [tails3](http://github.com/coggle/tails3) for a script to tail the log
files produced by s3-streamlogger.

### Installation
```bash
npm install --save s3-streamlogger
```

### Basic Usage
```js
var S3StreamLogger = require('s3-streamlogger').S3StreamLogger;

var s3stream = new S3StreamLogger({
               bucket: "mys3bucket",
        access_key_id: "...",
    secret_access_key: "..."
});

s3stream.write("hello S3");
```

### Use with Winston: Log to S3
```sh
npm install --save winston
npm install --save s3-streamlogger
```

```js
var winston        = require('winston');
var S3StreamLogger = require('s3-streamlogger').S3StreamLogger;

var s3_stream = new S3StreamLogger({
             bucket: "mys3bucket",
      access_key_id: "...",
  secret_access_key: "..."
});

var logger = new (winston.Logger)({
  transports: [
    new (winston.transports.File)({
      stream: s3_stream
    })
  ]
});

logger.info('Hello Winston!');
```

### Add hostname information for tails3
tails3 expects messages to be logged as json (the default for the file
transport), with hostname and (for critical errors), stack properties to each
log object, in addition to the standard timestamp, level and message
properties. You can provide these using the third "metadata" option to
winston's log method:

```js
logger.log(level, message, {hostname: ... , stack: ...});
```

### Handling logging errors
When there is an error writing to s3, the stream emits an 'error' event with
details. You should take care **not** to log these errors back to the same
stream (as that is likely to cause infinite recursion). Instead log them to the
console, to a file, or to SNS using [winston-sns](https://github.com/jesseditson/winston-sns).


### Options

#### bucket *(required)*
Name of the S3 bucket to upload data to. Must exist.
Can also be provided as the environment variable `BUCKET_NAME`.

#### access_key_id *(required)*
AWS access key ID, must have putObject permission on the specified bucket.
Can also be provided as the environment variable `AWS_SECRET_ACCESS_KEY`.

#### secret_access_key *(required)*
AWS secret key for the `access_key_id` specified.
Can also be provided as the environment variable `AWS_SECRET_KEY_ID`.

#### name_format
Format of file names to create, accepts [strftime specifiers](https://github.com/samsonjs/strftime). Defaults to `"%Y-%m-%d-%H-%M-unknown-unknown.log"`. The Date() used to fill the format specifiers is created with the current UTC time, but still *has the current timezone*, so any specifiers that perform timezone conversion will return incorrect dates.

If you use a format of the form `%Y-%m-%d-%H-%M-<stage>-<hostname>.log`, then
you can use [tails3](http://github.com/coggle/tails3) to tail the log files
being generated by `S3StreamLogger`.

#### rotate_every
Files will be rotated every `rotate_every` milliseconds. Defaults to 3600000 (60
minutes).

#### max_file_size
Files will be rotated when they reach `max_file_size` bytes. Defaults to 200 KB.

#### upload_every
Files will be uploaded every `upload_every` milliseconds. Defaults to 20
seconds.

#### buffer_size
Files will be uploaded if the un-uploaded data exceeds `buffer_size` bytes.
Defaults to 10 KB.

#### server_side_encryption
The server side encryption `AES256` algorithm used when storing objects in S3.
Defaults to false.


### License
[ISC](http://opensource.org/licenses/ISC): equivalent to 2-clause BSD.


{'fraction_non_alphanumeric': 0.11747110808973488, 'fraction_numerical': 0.014411964649898029, 'mean_word_length': 4.338171262699564, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '848990', 'n_tokens_mistral': 2324, 'n_tokens_neox': 2180, 'n_words': 843}