<commit_before>Express Gateway
=======

This is an API Gateway built using Express and Express middleware.

Configuration
-------------

The configuration file is a JSON document. It should be a JSON object at the
top level, with the following keys:

- `bindPort`: the port to listen on. Defaults to 8080
- `bindHost`: the IP address to listen on. Defaults to "0.0.0.0"
- `tls`: described below
- `privateEndpoints`: the URLs that the gateway will proxy to. Represented as a
  mapping of endpoint name to an object with the following keys:
  - `url`: the URL to forward requests to
- `pipelines`: a list of objects with the following keys:
  - `name`: the name of the pipeline
  - `publicEndpoints`: the list of URL paths for which requests will be
     handled. Represented as a list of objects with the following keys:
      - `path`: the URL path, e.g. "/products" or "/orders". The gateway will
        listen for requests for any URL that starts with the given string.
  - `processors`: the set of actions that should take place when a request is
    received on one of the public endpoints. Represented as a list of objects
    with the following keys (see below for more information):
    - `condition`. This condition must be satisfied to trigger the action.
    - `action`. The name of the action to carry out.
    - `params`. The parameters for the action.

### TLS

The gateway supports TLS, including SNI (domain-specific TLS certificates). To
configure, use the `tls` option. This option should be an object. Each key
should be a wildcard pattern for matching the domain, and the value should be
an object with `key` and `cert` as keys and paths to the files containing the
data in PEM format.

The special key `default` specifies the cert data to be used if none of the
other domain patterns can be matched, or if SNI is not being used by the
client.

For example:

```json
  {
    "*.lunchbadger.io": {
      "key": "example/keys/lunchbadger.io.key.pem",
      "cert": "example/keys/lunchbadger.io.cert.pem"
    },
    "api.lunchbadger.com": {
      "key": "example/keys/lunchbadger.com.key.pem",
      "cert": "example/keys/lunchbadger.com.cert.pem"
    },
    "default": {
      "key": "example/keys/lunchbadger.io.key.pem",
      "cert": "example/keys/lunchbadger.io.cert.pem"
    }
  }
```

### Processor conditions

Each processor in a pipeline can be gated with a condition specification. Each
condition specification is in the format:

```js
[name, ...params]
```

The name specifies a conditional function. This can be one of the following:

  - `always`: Always matches. If the conditional is missing, it will default to
    this.
  - `never`: Never matches.
  - `pathExact`: Matches if the request's path is an exact match for the
    parameter. Example:

        ["pathExact", "/foo/bar"]

  - `pathMatch`. Matches if the request's path matches the given regular
    expression parameter. Example:

        ["pathMatch", "/foo(/bar)?"]

  - `method`. Parameter can be either a string (e.g. 'GET') or an array of such
    strings. Matches if the request's method matches the parameter.
  - `hostMatch`. Parameter should be a regular expression. Matches if the
    `Host` header passed with the request matches the parameter.

In addition, several functions are provided that allow you to create logical
combinations of conditions. The parameters to these functions should be other
conditional statements:

  - `allOf`: Matches only if all of its parameters match.
  - `oneOf`: Matches if at least one of its parameters matches.
  - `not`: Matches only if its parameter does not.

Example:

```json
["allOf",
  ["pathExact", "/foo/bar"],
  ["not"
    ["method", ["POST", "HEAD"]]]]
```

The above will match only if the exact request path is "/foo/bar" and the
request is *not* a POST or HEAD.

### Processors

Several processors are available. Please note that the order of processors
is important.

#### Throttling

Throttles the requests to a specific rate limit. If the rate limit is reached,
the gateway will return an HTTP 429 return code with an error message.

Each request can be attached to one or more _throttle groups_. Each throttle
group can be specified with its own rate limit. When a request is a member of
multiple groups, each rate limit will be evaluated; exceeding any of the limits
will cause the request to be rejected.

To add a request to a throttle group, use the `throttleGroup` action. The
params format is an object with the following keys:

- `key`: the name of the throttle group to add the request to.

To specify a rate limit use the `throttle` action. The params format is an
object. Each key corresponds to the name of the throttle group. Each value is
an object with the keys:

- `rate`: the number of requests to allow per period
- `period`: the period. Could be one of `second`, `minute`, `hour`.

Example:

```json
{
  "condition": ["pathExact", "/foo"],
  "action": "throttleGroup",
  "params": {
    "key": "foo"
  }
},
{
  "condition": ["always"],
  "action": "throttleGroup",
  "params": {
    "key": "all"
  }
},
{
  "condition": ["always"],
  "action": "throttle",
  "params": {
    "all": {
      "rate": 100,
      "period": "minute"
    },
    "foo": {
      "rate": 1000,
      "period": "minute"
    }
  }
}
```
<commit_msg>Fixed typo in throttling example<commit_after>Express Gateway
=======

This is an API Gateway built using Express and Express middleware.

Configuration
-------------

The configuration file is a JSON document. It should be a JSON object at the
top level, with the following keys:

- `bindPort`: the port to listen on. Defaults to 8080
- `bindHost`: the IP address to listen on. Defaults to "0.0.0.0"
- `tls`: described below
- `privateEndpoints`: the URLs that the gateway will proxy to. Represented as a
  mapping of endpoint name to an object with the following keys:
  - `url`: the URL to forward requests to
- `pipelines`: a list of objects with the following keys:
  - `name`: the name of the pipeline
  - `publicEndpoints`: the list of URL paths for which requests will be
     handled. Represented as a list of objects with the following keys:
      - `path`: the URL path, e.g. "/products" or "/orders". The gateway will
        listen for requests for any URL that starts with the given string.
  - `processors`: the set of actions that should take place when a request is
    received on one of the public endpoints. Represented as a list of objects
    with the following keys (see below for more information):
    - `condition`. This condition must be satisfied to trigger the action.
    - `action`. The name of the action to carry out.
    - `params`. The parameters for the action.

### TLS

The gateway supports TLS, including SNI (domain-specific TLS certificates). To
configure, use the `tls` option. This option should be an object. Each key
should be a wildcard pattern for matching the domain, and the value should be
an object with `key` and `cert` as keys and paths to the files containing the
data in PEM format.

The special key `default` specifies the cert data to be used if none of the
other domain patterns can be matched, or if SNI is not being used by the
client.

For example:

```json
  {
    "*.lunchbadger.io": {
      "key": "example/keys/lunchbadger.io.key.pem",
      "cert": "example/keys/lunchbadger.io.cert.pem"
    },
    "api.lunchbadger.com": {
      "key": "example/keys/lunchbadger.com.key.pem",
      "cert": "example/keys/lunchbadger.com.cert.pem"
    },
    "default": {
      "key": "example/keys/lunchbadger.io.key.pem",
      "cert": "example/keys/lunchbadger.io.cert.pem"
    }
  }
```

### Processor conditions

Each processor in a pipeline can be gated with a condition specification. Each
condition specification is in the format:

```js
[name, ...params]
```

The name specifies a conditional function. This can be one of the following:

  - `always`: Always matches. If the conditional is missing, it will default to
    this.
  - `never`: Never matches.
  - `pathExact`: Matches if the request's path is an exact match for the
    parameter. Example:

        ["pathExact", "/foo/bar"]

  - `pathMatch`. Matches if the request's path matches the given regular
    expression parameter. Example:

        ["pathMatch", "/foo(/bar)?"]

  - `method`. Parameter can be either a string (e.g. 'GET') or an array of such
    strings. Matches if the request's method matches the parameter.
  - `hostMatch`. Parameter should be a regular expression. Matches if the
    `Host` header passed with the request matches the parameter.

In addition, several functions are provided that allow you to create logical
combinations of conditions. The parameters to these functions should be other
conditional statements:

  - `allOf`: Matches only if all of its parameters match.
  - `oneOf`: Matches if at least one of its parameters matches.
  - `not`: Matches only if its parameter does not.

Example:

```json
["allOf",
  ["pathExact", "/foo/bar"],
  ["not"
    ["method", ["POST", "HEAD"]]]]
```

The above will match only if the exact request path is "/foo/bar" and the
request is *not* a POST or HEAD.

### Processors

Several processors are available. Please note that the order of processors
is important.

#### Throttling

Throttles the requests to a specific rate limit. If the rate limit is reached,
the gateway will return an HTTP 429 return code with an error message.

Each request can be attached to one or more _throttle groups_. Each throttle
group can be specified with its own rate limit. When a request is a member of
multiple groups, each rate limit will be evaluated; exceeding any of the limits
will cause the request to be rejected.

To add a request to a throttle group, use the `throttleGroup` action. The
params format is an object with the following keys:

- `key`: the name of the throttle group to add the request to.

To specify a rate limit use the `throttle` action. The params format is an
object. Each key corresponds to the name of the throttle group. Each value is
an object with the keys:

- `rate`: the number of requests to allow per period
- `period`: the period. Could be one of `second`, `minute`, `hour`.

Example:

```json
{
  "condition": ["pathExact", "/foo"],
  "action": "throttleGroup",
  "params": {
    "key": "foo"
  }
},
{
  "condition": ["always"],
  "action": "throttleGroup",
  "params": {
    "key": "all"
  }
},
{
  "condition": ["always"],
  "action": "throttle",
  "params": {
    "all": {
      "rate": 1000,
      "period": "minute"
    },
    "foo": {
      "rate": 100,
      "period": "minute"
    }
  }
}
```

{'fraction_non_alphanumeric': 0.10211964201601507, 'fraction_numerical': 0.003391427225624117, 'mean_word_length': 3.3526035260352605, 'pattern_counts': {'":': 52, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1934243', 'n_tokens_mistral': 3166, 'n_tokens_neox': 2998, 'n_words': 1464}