<commit_before># Flows
Every flow is a series of steps. Each step will generally make one or more calls
to a given API endpoint, and store the resulting data in `flow.data`. However,
you can add steps that do anything.

Flows use a waterfall design pattern - each step has access to all data returned in
previous steps, and can use this to generate the next request.

Below is an example of DataFlow code. The steps are:
* `stories` - Gets a list of story IDs from Hacker News, e.g. `[234, 2352, 1834]`
* `story_details` - Gets details for the fist story
* `write_file` - Writes the details to a local file

```js
const datafire = require('datafire');
const fs = require('fs');
const hacker_news = datafire.Integration.new('hacker_news');

const flow = module.exports =
      new datafire.Flow('copyStory', 'Copies the top HN story to a local file');

flow
  .step('stories', {
    do: hacker_news.getStories(),
    params: {storyType: 'top'},
  })
  .step('story_details', {
    do: hacker_news.getItem(),
    params: data => {
      return {itemID: data.stories[0]}
    }
  })
  .step('write_file', {
    do: data => {
      fs.writeFileSync('./story.json', JSON.stringify(data.story_details, null, 2));
    }
  });
```

# API

## Flow
### `Flow.constructor(name, description)`
Creates a new flow

Example:
```js
let flow = new datafire.Flow('copyStuff', "Copies stuff from here to there");
```

### `Flow.step(name, options)`
Adds a new step to the flow.
* `name` - a unique name for this step.
* `options.do` - either a function or a datafire `Operation`
* `options.params` - an object with the parameters to pass to `operation`, or
a function that returns that object
* `options.finish` - a synchronous function to run after `do` has executed.
Use this to modify or check the response in `data[name]`

#### Setting parameters
The following two steps are equivalent. The first uses an object literal for `params`,
while the second wraps it inside a function.
```js
flow.step('user', {
  do: github.get('/users/{username}'),
  params: {username: 'torvalds'}
})
```

```js
flow.step('user', {
  do: github.get('/users/{username}'),
  params: data => {
    return {username: 'torvalds'}
  }
})
```

#### Chaining
You can also chain calls to `step()`. Each step has access to the responses
from all the previous steps.

```js
flow
  .step('users', {
    do: github.get('/users'),
  })
  .step('repos', {
    do: github.get('/users/{username}/repos'),
    params: data => {
      return {
        username: data.users[0].login,
      }
    }
  });
```

#### `options.do`
`options.do` is generally an Operation returned by an Integration, e.g.
`hacker_news.getItem()`.  However, as you might have noticed in the
quickstart example, you can also pass in your own function:
```js
flow.step('write_file', {
  do: data => {
    fs.writeFileSync('./story.json', JSON.stringify(data.story))
  }
});
```

You can also make your function asynchronous:
```js
flow.step('write_file', {
  do: (data, callback) => {
    fs.writeFile('./story.json', JSON.stringify(data.story), (err) => {
      if (err) return callback(err);
      callback(null, "Success");
    })
  }
});
```

### `Flow.repeatStep(name, operation, request)`
**Unimplemented**

### `Flow.catch(callback)`
Catches all HTTP errors (e.g. 404 or 500), and thrown errors.
Use this to react to errors e.g. by sending an e-mail.

```js
flow
  .step('messages', {
    do: gmail.get('/messages'),
    params: {limit: 10},
  })
  .catch((err, data) => {
    if (err.statusCode === 429) {
      console.log('Gmail rate limit exceeded, ignoring')
      flow.succeed();
    } else {
      throw err;
    }
  })
  .step('add_issue', {
    do: github.post('/issues'),
    params: data => {
      return {
        body {
          title: "There are " + data.messages.length + " new messages",
          body: messages.map(m => m.subject).join('\n'),
          assignee: 'bobby-brennan',
        }
      }
    }
  });
```


### `Flow.fail(message)`
Can be called inside of a step to exit early. No subsequent steps will be called.

### `Flow.succeed(message)`
Can be called inside of a step to exit early. No subsequent steps will be called.

### `Flow.setDefaults(defaults)`
Use `setDefaults` to parameterize your flow with options:

```js
flow.setDefaults({
  username: 'torvalds',
  repo: 'linux',
})
flow.step('issues', {
  do: github.get('/repos/{owner}/{repo}/issues'),
  params: () => {
    return {
      repo: flow.options.repo,
      username: flow.options.username,
    }
  }
})
```

You can then pass options via the command line:
```
datafire run -f ./copyIssues.js --options.username="expressjs" --options.repo="express"
```

Or via an HTTP request (if you're using Serverless):
```
curl http://something.execute-api.us-east-1.amazonaws.com/dev/copyIssues?username="expresjs"&repo="expres"
```
<commit_msg>bump headings<commit_after># Flows
Every flow is a series of steps. Each step will generally make one or more calls
to a given API endpoint, and store the resulting data in `flow.data`. However,
you can add steps that do anything.

Flows use a waterfall design pattern - each step has access to all data returned in
previous steps, and can use this to generate the next request.

Below is an example of DataFlow code. The steps are:
* `stories` - Gets a list of story IDs from Hacker News, e.g. `[234, 2352, 1834]`
* `story_details` - Gets details for the fist story
* `write_file` - Writes the details to a local file

```js
const datafire = require('datafire');
const fs = require('fs');
const hacker_news = datafire.Integration.new('hacker_news');

const flow = module.exports =
      new datafire.Flow('copyStory', 'Copies the top HN story to a local file');

flow
  .step('stories', {
    do: hacker_news.getStories(),
    params: {storyType: 'top'},
  })
  .step('story_details', {
    do: hacker_news.getItem(),
    params: data => {
      return {itemID: data.stories[0]}
    }
  })
  .step('write_file', {
    do: data => {
      fs.writeFileSync('./story.json', JSON.stringify(data.story_details, null, 2));
    }
  });
```

# API
## `Flow.constructor(name, description)`
Creates a new flow

Example:
```js
let flow = new datafire.Flow('copyStuff', "Copies stuff from here to there");
```

## `Flow.step(name, options)`
Adds a new step to the flow.
* `name` - a unique name for this step.
* `options.do` - either a function or a datafire `Operation`
* `options.params` - an object with the parameters to pass to `operation`, or
a function that returns that object
* `options.finish` - a synchronous function to run after `do` has executed.
Use this to modify or check the response in `data[name]`

### Setting parameters
The following two steps are equivalent. The first uses an object literal for `params`,
while the second wraps it inside a function.
```js
flow.step('user', {
  do: github.get('/users/{username}'),
  params: {username: 'torvalds'}
})
```

```js
flow.step('user', {
  do: github.get('/users/{username}'),
  params: data => {
    return {username: 'torvalds'}
  }
})
```

### Chaining
You can also chain calls to `step()`. Each step has access to the responses
from all the previous steps.

```js
flow
  .step('users', {
    do: github.get('/users'),
  })
  .step('repos', {
    do: github.get('/users/{username}/repos'),
    params: data => {
      return {
        username: data.users[0].login,
      }
    }
  });
```

### `options.do`
`options.do` is generally an Operation returned by an Integration, e.g.
`hacker_news.getItem()`.  However, as you might have noticed in the
quickstart example, you can also pass in your own function:
```js
flow.step('write_file', {
  do: data => {
    fs.writeFileSync('./story.json', JSON.stringify(data.story))
  }
});
```

You can also make your function asynchronous:
```js
flow.step('write_file', {
  do: (data, callback) => {
    fs.writeFile('./story.json', JSON.stringify(data.story), (err) => {
      if (err) return callback(err);
      callback(null, "Success");
    })
  }
});
```

## `Flow.repeatStep(name, operation, request)`
**Unimplemented**

## `Flow.catch(callback)`
Catches all HTTP errors (e.g. 404 or 500), and thrown errors.
Use this to react to errors e.g. by sending an e-mail.

```js
flow
  .step('messages', {
    do: gmail.get('/messages'),
    params: {limit: 10},
  })
  .catch((err, data) => {
    if (err.statusCode === 429) {
      console.log('Gmail rate limit exceeded, ignoring')
      flow.succeed();
    } else {
      throw err;
    }
  })
  .step('add_issue', {
    do: github.post('/issues'),
    params: data => {
      return {
        body {
          title: "There are " + data.messages.length + " new messages",
          body: messages.map(m => m.subject).join('\n'),
          assignee: 'bobby-brennan',
        }
      }
    }
  });
```


## `Flow.fail(message)`
Can be called inside of a step to exit early. No subsequent steps will be called.

## `Flow.succeed(message)`
Can be called inside of a step to exit early. No subsequent steps will be called.

## `Flow.setDefaults(defaults)`
Use `setDefaults` to parameterize your flow with options:

```js
flow.setDefaults({
  username: 'torvalds',
  repo: 'linux',
})
flow.step('issues', {
  do: github.get('/repos/{owner}/{repo}/issues'),
  params: () => {
    return {
      repo: flow.options.repo,
      username: flow.options.username,
    }
  }
})
```

You can then pass options via the command line:
```
datafire run -f ./copyIssues.js --options.username="expressjs" --options.repo="express"
```

Or via an HTTP request (if you're using Serverless):
```
curl http://something.execute-api.us-east-1.amazonaws.com/dev/copyIssues?username="expresjs"&repo="expres"
```

{'fraction_non_alphanumeric': 0.15557618900237286, 'fraction_numerical': 0.005364696172495615, 'mean_word_length': 3.396371882086168, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1916246', 'n_tokens_mistral': 3214, 'n_tokens_neox': 3075, 'n_words': 1022}