<commit_before># DataFire

Version 2.0 will be an open-source integration framework built on top of [Serverless](/serverless/serverless).
Users will be able to run Dataflows on their own AWS account, or on DataFire.

## Installation
You'll need to install DataFire both globally and as a project dependency.
```bash
npm install -g bobby-brennan/datafire
npm install --save bobby-brennan/datafire
```

## Quick Start
This quick tutorial will fetch issues from a repository on GitHub, and copy them to
a local file.

First, let's add the GitHub integration:
```bash
datafire integrate --name github
```

Now we need to create a Flow. Edit `./copyIssues.js`:
```js
const fs = require('fs');
const datafire = require('datafire');

let github = new datafire.Integration('github');

let flow = module.exports = new datafire.Flow('copyIssues', 'Copies issues from GitHub to a local file');

flow.setDefaults({
  username: 'bobby-brennan',
  repo: 'rss-parser',
});
flow.step('issues',
          github.get('/repos/{owner}/{repo}/issues'),
          {owner: flow.options.username, repo: flow.options.repo})
    .step('write_file',
          (data) => {
            fs.writeFileSync('./issues.json', JSON.stringify(data, null, 2));
          })

```

Now let's run it:
```bash
datafire run -f ./copyIssues.js
```

You should see `issues.json` in your current directory.

## Serverless Execution
To run a flow on a regular schedule, you can use [crontab](https://en.wikipedia.org/wiki/Cron),
but DataFire also offers native support for execution on AWS Lambda,
via the [Serverless](https://github.com/serverless/serverless) framework. You can then
run your flow on a schedule or in response to a webhook.

Just set your handler in `serverless.yml` to `yourFlow.handler`:

```yml
service: copyIssues

provider:
  name: aws
  runtime: nodejs4.3

functions:
  copyIssues:<commit_msg>README work<commit_after># DataFire

Version 2.0 will be an open-source integration framework built on top of [Serverless](/serverless/serverless).
Users will be able to run Dataflows on their own AWS account, or on DataFire.

## Installation
You'll need to install DataFire both globally and as a project dependency.
```bash
npm install -g bobby-brennan/datafire
npm install --save bobby-brennan/datafire
```

## Quick Start
This quick tutorial will fetch issues from a repository on GitHub, and copy them to
a local file.

First, let's add the GitHub integration:
```bash
datafire integrate --name github
```

Now we need to create a Flow. Edit `./copyIssues.js`:
```js
const fs = require('fs');
const datafire = require('datafire');

let github = new datafire.Integration('github');

let flow = module.exports =
      new datafire.Flow('copyIssues', 'Copies issues from GitHub to a local file');

flow.setDefaults({
  username: 'torvalds',
  repo: 'linux',
});
flow.step('issues',
          github.get('/repos/{owner}/{repo}/issues'),
          {owner: flow.options.username, repo: flow.options.repo})
    .step('write_file',
          (data) => {
            fs.writeFileSync('./issues.json', JSON.stringify(data, null, 2));
          })

```

Now let's run it:
```bash
datafire run -f ./copyIssues.js
```

You should see `issues.json` in your current directory.

You can also set options via the command line:
```bash
datafire run -f ./copyIssues.js --options.username="expressjs" --options.repo="express"
```

## Serverless Execution
To run a flow on a regular schedule, you can use [crontab](https://en.wikipedia.org/wiki/Cron),
but DataFire also offers native support for execution on AWS Lambda,
via the [Serverless](https://github.com/serverless/serverless) framework. You can then
run your flow on a schedule or in response to a webhook.

Just set your handler in `serverless.yml` to `yourFlow.handler`:

```yml
service: copyIssues

provider:
  name: aws
  runtime: nodejs4.3

functions:
  copyIssues:
{'fraction_non_alphanumeric': 0.12081401339515714, 'fraction_numerical': 0.002575991756826378, 'mean_word_length': 3.965473145780051, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '844315', 'n_tokens_mistral': 1219, 'n_tokens_neox': 1153, 'n_words': 440}