<commit_before># Compoxure Composition Middleware
[![Build Status](https://travis-ci.org/tes/compoxure.svg?branch=master)](https://travis-ci.org/tes/compoxure) [![Dependencies](https://david-dm.org/tes/compoxure.svg)](https://david-dm.org/tes/compoxure) [![Coverage Status](https://img.shields.io/coveralls/tes/compoxure.svg)](https://coveralls.io/r/tes/compoxure?branch=master)

Composition proxy replacement for ESI or SSI uses [htmlparser2](https://github.com/fb55/htmlparser2/) to parse HTML from backend services and compose fragments from microservices into the response.  This is exposed as connect middleware to allow quick creation of a proxy server.

For rationale (e.g. why the heck would anyone build this), please see the rationale section at the bottom.

## Running Example App

```bash
git clone <EMAIL>:tes/compoxure.git
cd compoxure
npm install
node example
```
Visit [http://localhost:5000/](http://localhost:5000/)

## What is it

Compoxure is a composition proxy - you put it in front of a back end service that acts as a template into which content from microservices is composed.  It is designed to be simple, fast and failure tolerant - e.g. you won't need to build complex patterns across and around all of your micro services to deal with failure scenarios, they just serve out HTML and should try to fail fast.

## Examples of usage

 - Branding: Imagine that you need to share common branding (masthead, navigation, footer) across a range of applications.  You can put this in front of the application and it can then include these elements in the page from a common set of microservices responsible for each.

 - Expose an app within a CMS: Instead of building your apps in your CMS (arrgh) instead include them in CMS managed pages with a simple declaration.  The apps won't need to know about the site branding, the CMS takes care of all of that.  Also makes it easy to have those apps expose small amounts of functionality across your site (e.g. small Top 10 lists etc.).

 - Complex application decomposition:  Image you have to deliver a page that has some product data, as well as a range of content that depends on the user.  Compoxure will let you serve these fragments of the page at different TTLs, and if you add middleware that parses your user cookie it can actually pass back user information to these services so they dont have to - making those easier to test and reason about.

## How it works

You have a back end service (e.g. a CMS) that returns HTML containing the declarative markup explained below. Compoxure then responds to requests, calling the backend service that matches the URL, parses the HTML it returns and makes any requests to other micro services whose responses are then inserted into the HTML on the way through.

The various responses (e.g. the backend HTML page, each service fragment response) can all be cached at differing TTLs (or not at all) and using a simple key construction method that covers the scenario that some fragments may differ for not logged in users vs logged in users.

Typically the backend service will be a CMS or a static HTML page containing specific markup, with the thing that is included being the output of more complex applications.

```html
<div cx-url='{{server:local}}/application/widget/{{cookie:userId}}' cx-cache-ttl='10s' cx-cache-key='widget:user:{{cookie:userId}}' cx-timeout='1s' cx-statsd-key="widget_user">
     This content will be replaced on the way through
</div>
```

### Configuration

The full configuration options can be found in /examples/example.json, but it is expected that you will store this configuration within your own application and pass it through into the middleware.

The configuration object looks as follows:

```json
{
    "backend": [{
        "pattern": ".*",
        "target":"http://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "quietFailure":false,
        "leaveContentOnFail":false,
        "dontPassUrl":false,
        "passThrough":false,
        "contentTypes":["html"]
    }],
    "parameters": {
        "urls": [
            {"pattern": "/teaching-resource/.*-(\\d+)", "names": ["resourceId"]}
        ],
        "servers": {
            "local": "http://localhost:5001"
        }
    },
    "cdn": {
        "host": "server.cloudfront.net",
        "protocol": "https",
        "path": "/assets"
    },
    "environment":{
        "name":"development"
    },
    "hogan":{
        "delimiters":"{{ }}"
    },
    "cache": {
        "engine": "redis"
    },
    "followRedirect": false,
    "fragmentPasses": 5
}
```

#### Backend

These properties configure the backend server that the initial request goes to grab the HTML that is then processed.

| Property           | Description |
|--------------------|-------------|
| pattern            | The regexp used to select this backend based on the incoming request to compoxure.  First match wins.|
| fn                 | Name of a selector function of type: function(req, variables) { return true; } to allow more dynamic selection of backend (instead of pattern).  Function must be passed in on the config.functions object, and is referenced by name.|
| target             | The base URL to the backend service that will serve HTML.  URLs are passed directly through, so /blah will be passed through to the backend as http://backend/blah|
| host               | The name to be passed through in the request (given the hostname of compoxure is likely to be different to that of the backend server.  The host is used as the key for all the statds stats.|
| ttl                | The amount of time to cache the backend response (TODO : make it honor cache headers)|
| timeout            | Time to wait for backend to respond - should set low|
| quietFailure       | Used to determine if compoxure will serve some error text in response to a microservice failure or fail silently (e.g. serve nothing into the space).
| dontPassUrl        | Used to decide if the URL in the request is passed through to the backend.  Set to true if the backend should ignore the front URL and just serve the same page for all requests (e.g. a fixed template)|
| contentTypes       | An array of content types which are accepted by this backend. Defaults to `['html']`. See the [accepts](https://www.npmjs.org/package/accepts) documentation regarding how headers are parsed. *Note: The order is important! We recommend that you always put `html` as the first item in the array.* |
| headers            | An array of header names to specify headers forwarded to the backend server. |
| noCache            | Do not cache this backend. |
| passThroughHeaders | Pass this array of headers to the client if the backend sets them. E.g., `"passThroughHeaders": ["x-robot-tag","cache-control"]`|
| addResponseHeaders | Set the response headers in this map always. E.g., `"addResponseHeaders": { "x-robots-tag": "noindex" }`

You can define multiple backends, by adding as many declarations for backends as you like, with differing patterns.  The first match wins e.g.

```json
 "backend": [{
        "fn": "selectResource",
        "target":"http://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "replaceOuter":false,
        "quietFailure":true<commit_msg>Resolve #60<commit_after># Compoxure Composition Middleware
[![Build Status](https://travis-ci.org/tes/compoxure.svg?branch=master)](https://travis-ci.org/tes/compoxure) [![Dependencies](https://david-dm.org/tes/compoxure.svg)](https://david-dm.org/tes/compoxure) [![Coverage Status](https://img.shields.io/coveralls/tes/compoxure.svg)](https://coveralls.io/r/tes/compoxure?branch=master)

Composition proxy replacement for ESI or SSI uses [htmlparser2](https://github.com/fb55/htmlparser2/) to parse HTML from backend services and compose fragments from microservices into the response.  This is exposed as connect middleware to allow quick creation of a proxy server.

For rationale (e.g. why the heck would anyone build this), please see the rationale section at the bottom.

## Running Example App

```bash
git clone <EMAIL>:tes/compoxure.git
cd compoxure
npm install
node example
```
Visit [http://localhost:5000/](http://localhost:5000/)

## What is it

Compoxure is a composition proxy - you put it in front of a back end service that acts as a template into which content from microservices is composed.  It is designed to be simple, fast and failure tolerant - e.g. you won't need to build complex patterns across and around all of your micro services to deal with failure scenarios, they just serve out HTML and should try to fail fast.

## Examples of usage

 - Branding: Imagine that you need to share common branding (masthead, navigation, footer) across a range of applications.  You can put this in front of the application and it can then include these elements in the page from a common set of microservices responsible for each.

 - Expose an app within a CMS: Instead of building your apps in your CMS (arrgh) instead include them in CMS managed pages with a simple declaration.  The apps won't need to know about the site branding, the CMS takes care of all of that.  Also makes it easy to have those apps expose small amounts of functionality across your site (e.g. small Top 10 lists etc.).

 - Complex application decomposition:  Image you have to deliver a page that has some product data, as well as a range of content that depends on the user.  Compoxure will let you serve these fragments of the page at different TTLs, and if you add middleware that parses your user cookie it can actually pass back user information to these services so they dont have to - making those easier to test and reason about.

## How it works

You have a back end service (e.g. a CMS) that returns HTML containing the declarative markup explained below. Compoxure then responds to requests, calling the backend service that matches the URL, parses the HTML it returns and makes any requests to other micro services whose responses are then inserted into the HTML on the way through.

The various responses (e.g. the backend HTML page, each service fragment response) can all be cached at differing TTLs (or not at all) and using a simple key construction method that covers the scenario that some fragments may differ for not logged in users vs logged in users.

Typically the backend service will be a CMS or a static HTML page containing specific markup, with the thing that is included being the output of more complex applications.

```html
<div cx-url='{{server:local}}/application/widget/{{cookie:userId}}' cx-cache-ttl='10s' cx-cache-key='widget:user:{{cookie:userId}}' cx-timeout='1s' cx-statsd-key="widget_user">
     This content will be replaced on the way through
</div>
```

### Configuration

The full configuration options can be found in /examples/example.json, but it is expected that you will store this configuration within your own application and pass it through into the middleware.

The configuration object looks as follows:

```json
{
    "backend": [{
        "pattern": ".*",
        "target":"http://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "quietFailure":false,
        "leaveContentOnFail":false,
        "dontPassUrl":false,
        "passThrough":false,
        "cacheKey":"{{url:path}}",
        "contentTypes":["html"]
    }],
    "parameters": {
        "urls": [
            {"pattern": "/teaching-resource/.*-(\\d+)", "names": ["resourceId"]}
        ],
        "servers": {
            "local": "http://localhost:5001"
        }
    },
    "cdn": {
        "host": "server.cloudfront.net",
        "protocol": "https",
        "path": "/assets"
    },
    "environment":{
        "name":"development"
    },
    "hogan":{
        "delimiters":"{{ }}"
    },
    "cache": {
        "engine": "redis"
    },
    "followRedirect": false,
    "fragmentPasses": 5
}
```

#### Backend

These properties configure the backend server that the initial request goes to grab the HTML that is then processed.

| Property           | Description |
|--------------------|-------------|
| pattern            | The regexp used to select this backend based on the incoming request to compoxure.  First match wins.|
| fn                 | Name of a selector function of type: function(req, variables) { return true; } to allow more dynamic selection of backend (instead of pattern).  Function must be passed in on the config.functions object, and is referenced by name.|
| target             | The base URL to the backend service that will serve HTML.  URLs are passed directly through, so /blah will be passed through to the backend as http://backend/blah|
| host               | The name to be passed through in the request (given the hostname of compoxure is likely to be different to that of the backend server.  The host is used as the key for all the statds stats.|
| ttl                | The amount of time to cache the backend response (TODO : make it honor cache headers)|
| timeout            | Time to wait for backend to respond - should set low|
| quietFailure       | Used to determine if compoxure will serve some error text in response to a microservice failure or fail silently (e.g. serve nothing into the space).
| dontPassUrl        | Used to decide if the URL in the request is passed through to the backend.  Set to true if the backend should ignore the front URL and just serve the same page for all requests (e.g. a fixed template)|
| contentTypes       | An array of content types which are accepted by this backend. Defaults to `['html']`. See the [accepts](https://www.npmjs.org/package/accepts) documentation regarding how headers are parsed. *Note: The order is important! We recommend that you always put `html` as the first item in the array.* |
| headers            | An array of header names to specify headers forwarded to the backend server. |
| cacheKey            | Cache key to use for requests to this backend |
| noCache            | Do not cache this backend. |
| passThroughHeaders | Pass this array of headers to the client if the backend sets them. E.g., `"passThroughHeaders": ["x-robot-tag","cache-control"]`|
| addResponseHeaders | Set the response headers in this map always. E.g., `"addResponseHeaders": { "x-robots-tag": "noindex" }`

You can define multiple backends, by adding as many declarations for backends as you like, with differing patterns.  The first match wins e.g.

```json
 "backend": [{
        "fn": "selectResource",
        "target":"http://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "cacheKey":"{{params:resourceId}}",
        "replaceOuter":false,
        "quietFailure":true
{'fraction_non_alphanumeric': 0.09503767564998981, 'fraction_numerical': 0.0036657389179281788, 'mean_word_length': 3.3663307646710137, 'pattern_counts': {'":': 78, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 16, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2289068', 'n_tokens_mistral': 3913, 'n_tokens_neox': 3756, 'n_words': 1864}