<commit_before>Stream to S3
========================

A quick script to **stream large files to Amazon S3 using node.js**

[![Build Status](https://travis-ci.org/nelsonic/stream-to-s3.png?branch=master)](https://travis-ci.org/nelsonic/stream-to-s3)
[![Code Climate](https://codeclimate.com/github/nelsonic/stream-to-s3.png)](https://codeclimate.com/github/nelsonic/stream-to-s3)
[![Coverage Status](https://coveralls.io/repos/nelsonic/stream-to-s3/badge.png?branch=master)](https://coveralls.io/r/nelsonic/stream-to-s3?branch=master)
[![Dependencies](https://david-dm.org/nelsonic/stream-to-s3.png?theme=shields.io)](https://david-dm.org/nelsonic/stream-to-s3)
[![devDependency Status](https://david-dm.org/nelsonic/stream-to-s3/dev-status.svg)](https://david-dm.org/nelsonic/stream-to-s3#info=devDependencies)

[![NPM stats](https://nodei.co/npm/stream-to-s3.png?downloads=true)](https://www.npmjs.org/package/stream-to-s3)

Node.js is the *perfect* tool for streaming large files to Amazon Web Services S3.


![Node.js and Amazon S3](http://i.imgur.com/w1S84gJ.png)

## Use Cases

- Rolling your own cloud backup service
- Uploading files/photos from a mobile app or website
- Drop Box Clone using S3 as backend


## Usage

Install from NPM:

```sh
npm install stream-to-s3 --save
```

```
var S3 = require('stream-to-s3');
var file = '/your-file-name.jpg';      // any file format!
S3.streamFileToS3(file, function(err) { // standard callback function:
  console.log(file,' Was uploaded. Visit:',S.S3FileUrl(file));
});
```

### Require Environment Variables

`stream-to-s3` uses [environment variables](https://github.com/dwyl/**learn-environment-variables)
for Amazon WebServices Secret Keys  
(*to help people avoid hard-coding passwords in your code...*)

> If you're ***new to Environment Variables***
check out our ***complete beginners guide***:
[https://github.com/dwyl/**learn-environment-variables**](https://github.com/dwyl/learn-environment-variables)

you will need to *set* the following environment variables:

```sh
AWSAccessKeyId=ReplaceWithYourActualKey
AWSSecretKey=DownloadThisFromYourAWSConsole
S3BUCKET=YourS3BucketName
AWSREGION=eu-west-1
ACL=public-read
```

> While you are developing your app, we recommend managing your environment
variables using [**env2**](https://github.com/dwyl/env2)  
this will allow you to use a *file* to keep your AWS/S3 keys which
you can easily share with your co-developers and still *exculde* from GitHub
(*by listing it in your `.gitignore` file*)

# tl;dr

**Note**: we have deliberately kept the stream-uploader simple,
if you need to transform the data in the read-stream before
uploading it, fork this repo, add a new test/method and submit a PR.

## The Solution

See: **index.js** for the implementation details.

## Tests! :open_mouth:

**Three steps** to running the test suite:

1. Install (Dev) Dependencies (Mocha etc.) <br/>
```sh
npm install
```

2. **Copy** the **config_example.json** file **to config.json**
and put your *real* Amazon S3 credentials in the config.json file:
```sh
cp config_example.json config.json  &&  atom config.json
```

3. Run ***real***tests using using the following command:
```sh
npm run realtest
```


***Note***: Because I don't want to make my S3 Keys ***Public***
(on GitHub), <br />
I've had to "fake" the `npm test` command for Travis.<br/>
But don't fear, there are tests and they are easy to run. <br/>
(and I don't encourage others to do the **exit 0** "hack" unless <br />
there's some sensitive config you don't want to open-source...)

## Useful Links

Node.js Docs for Readable Streams:
- http://nodejs.org/api/stream.html#stream_class_stream_readable

Best place to learn Node.js Streams:
- SubStack's Streams Intro: [**github.com/substack/stream-handbook**](http://github.com/substack/stream-handbook)
- Max's stream intro: http://maxogden.com/node-streams.html
- http://ejohn.org/blog/node-js-stream-playground
- http://codewinds.com/blog/2013-08-04-nodejs-readable-streams.html


## Background

### "It Cannot Be Done" (Challenge Accepted!)<commit_msg>adds more detailed usage for env2 fixes https://github.com/dwyl/stream-to-s3/issues/5<commit_after>Stream to S3
========================

A quick script to **stream large files to Amazon S3 using node.js**

[![Build Status](https://travis-ci.org/dwyl/stream-to-s3.png?branch=master)](https://travis-ci.org/dwyl/stream-to-s3)
[![Code Climate](https://codeclimate.com/github/dwyl/stream-to-s3.png)](https://codeclimate.com/github/dwyl/stream-to-s3)
[![codecov.io](http://codecov.io/github/dwyl/stream-to-s3/coverage.svg?branch=master)](http://codecov.io/github/dwyl/stream-to-s3?branch=master)
[![Dependencies](https://david-dm.org/dwyl/stream-to-s3.png?theme=shields.io)](https://david-dm.org/dwyl/stream-to-s3)
[![devDependency Status](https://david-dm.org/dwyl/stream-to-s3/dev-status.svg)](https://david-dm.org/dwyl/stream-to-s3#info=devDependencies)

[![NPM stats](https://nodei.co/npm/stream-to-s3.png?downloads=true)](https://www.npmjs.org/package/stream-to-s3)

Node.js is the *perfect* tool for streaming large files to Amazon Web Services S3.


![Node.js and Amazon S3](http://i.imgur.com/w1S84gJ.png)

## Use Cases

- Rolling your own cloud backup service
- Uploading files/photos from a mobile app or website
- Drop Box Clone using S3 as backend


## Usage

Install from NPM:

```sh
npm install stream-to-s3 --save
```

```js
var S3 = require('stream-to-s3');
var file = '/your-file-name.jpg';      // any file format!
S3.streamFileToS3(file, function(err) { // standard callback function:
  console.log(file,' Was uploaded. Visit:',S.S3FileUrl(file));
});
```

### Require Environment Variables

`stream-to-s3` uses [environment variables](https://github.com/dwyl/**learn-environment-variables)
for Amazon WebServices Secret Keys  
(*to help people avoid hard-coding passwords in your code...*)

> If you're ***new to Environment Variables***
check out our ***complete beginners guide***:
[https://github.com/dwyl/**learn-environment-variables**](https://github.com/dwyl/learn-environment-variables)

you will need to *set* the following environment variables:

```sh
AWSAccessKeyId=ReplaceWithYourActualKey
AWSSecretKey=DownloadThisFromYourAWSConsole
S3BUCKET=YourS3BucketName
AWSREGION=eu-west-1
ACL=public-read
```

> While you are developing your app, we recommend managing your environment
variables using [**env2**](https://github.com/dwyl/env2)  
this will allow you to use a *file* to keep your AWS/S3 keys which
you can easily share with your co-developers and still *exculde* from GitHub
(*by listing it in your `.gitignore` file*)

To help you get started we have created a sample `config.env` file
to use it, simply copy it to your working directory:

```sh
cp node_modules/stream-to-s3/config.env_example ./config.env && echo 'config.env' >> .gitignore
```

Then download your S3 keys from your AWS Console and set both keys and S3 bucket in your `config.env` file.

Next load your environment variables using [**env2**](https://github.com/dwyl/env2)

```sh
npm install env2 --save
```

```js
require('env2')('config.env');         // load S3 Keys from config.env
var S3 = require('stream-to-s3');
var file = '/your-file-name.jpg';      // any file format!
S3.streamFileToS3(file, function(err) { // standard callback function:
  console.log(file,' Was uploaded. Visit:',S.S3FileUrl(file));
});
```


# tl;dr

**Note**: we have deliberately kept the stream-uploader simple,
if you need to transform the data in the read-stream before
uploading it, fork this repo, add a new test/method and submit a PR.

## The Solution

See: **index.js** for the implementation details.


## Useful Links

Node.js Docs for Readable Streams:
- http://nodejs.org/api/stream.html#stream_class_stream_readable

Best place to learn Node.js Streams:
- SubStack's Streams Intro: [**github.com/substack/stream-handbook**](http://github.com/substack/stream-handbook)
- Max's stream intro: http://maxogden.com/node-streams.html
- http://ejohn.org/blog/node-js-stream-playground
- http://codewinds.com/blog/2013-08-04-nodejs-readable-streams.html


## Background

### "It Cannot Be Done" (Challenge Accepted!)
{'fraction_non_alphanumeric': 0.15710967110893753, 'fraction_numerical': 0.012715490891307005, 'mean_word_length': 4.660899653979238, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 14, 'https://': 32, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1606627', 'n_tokens_mistral': 2820, 'n_tokens_neox': 2686, 'n_words': 732}