<commit_before>---
title: DataSources
layout: default
category: Kettle
---
# DataSources

A DataSource is an Infusion component which meets a simple contract for read/write access to indexed data.
DataSource is a simple semantic, broadly the same as that encoded in
[CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete), although the current DataSource semantic does
not provide explicitly for deletion.

The concrete DataSources in Kettle provide support for HTTP endpoints (with a particular variety specialised for
accessing CouchDB databases with CRUDlike semantics) as well as the filesystem, with an emphasis on JSON payloads.

The DataSource API is drawn from the following two methods – a read-only DataSource will just implement `get`, and a
writeable DataSource will implement both `get` and `set`:

```javascript
    /* @param directModel {Object} A JSON structure holding the "coordinates" of the state to be read -
     * this model is morally equivalent to (the substitutable parts of) a file path or URL
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just
     * this request. These will be specially interpreted by the particular concrete grade of DataSource
     * – there are no options valid across all implementations of this grade.
     * @return {Promise} A promise representing successful or unsuccessful resolution of the read state
     */
    dataSource.get(directModel, options);
    /* @param directModel {Object} As for get
     * @param model {Object} The state to be written to the coordinates
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just
     * this request. These will be specially interpreted by the
     * particular concrete grade of DataSource – there are no options valid across all implementations
     * of this grade. For example, a URL DataSource will accept an option `writeMethod` which will
     * allow the user to determine which HTTP method (PUT or POST) will be used to implement the write
     * operation.
     * @return {Promise} A promise representing resolution of the written state,
     * which may also optionally resolve to any returned payload from the write process
     */
    dataSource.set(directModel, model, options);
```

## Simple example of using an HTTP dataSource

In this example we define and instantiate a simple HTTP-backed dataSource accepting one argument to configure a URL
segment:

```javascript
var fluid = require("infusion"),
    kettle = require("../../kettle.js"),
    examples = fluid.registerNamespace("examples");


fluid.defaults("examples.httpDataSource", {
    gradeNames: "kettle.dataSource.URL",
    url: "http://jsonplaceholder.typicode.com/posts/%postId",
    termMap: {
        postId: "%directPostId"
    }
});

var myDataSource = examples.httpDataSource();
var promise = myDataSource.get({directPostId: 42});

promise.then(function (response) {
    console.log("Got dataSource response of ", response);
}, function (error) {
    console.error("Got dataSource error response of ", error);
});
```

You can run this snippet from our code samples by running `node simpleDataSource.js` from
[examples/simpleDataSource](../examples/simpleDataSource) in our samples area.
This contacts the useful JSON placeholder API service at
[`jsonplaceholder.typicode.com`](http://jsonplaceholder.typicode.com/)
to retrieve a small JSON document holding some placeholder text. If you get a 404 or an error, please contact us and
we'll update this sample to contact a new service.

An interesting element in this snippet is the `termMap` configured as options of our dataSource. This sets up an
indirection between the `directModel` supplied as the argument to the `dataSource.get` call, and the URL issued in the
HTTP request. The keys in the `termMap` are interpolation variables in the URL, which in the URL are prefixed by `%`.
The values in the `termMap` represent either

* Plain values to be interpolated as strings directly into the URL, or
* If the first character of the value in the `termMap` is %, the remainder of the string represents a path which will
  be dereferenced from the `directModel` argument to the current `set` or `get` request.

In addition, if the term value has the prefix `noencode:`, it will be interpolated without any
[URI encoding](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent).

We document these configuration options in the next section:

## Configuration options accepted by `kettle.dataSource.URL`

<table>
    <thead>
        <tr>
            <th colspan="3">Supported configurable options for a <code>kettle.dataSource.URL</code></th>
        </tr>
        <tr>
            <th>Option Path</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>writable</code></td>
            <td><code>Boolean</code> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a <code>set</code> method will be fabricated for this
                dataSource – otherwise, it will implement only a <code>get</code> method.</td>
        </tr>
        <tr>
            <td><code>writeMethod</code></td>
            <td><code>String</code> (default: <code>PUT</code>)</td>
            <td>The HTTP method to be used when the <code>set</code> method is operated on this writable DataSource
                (with <code>writable: true</code>). This defaults to <code>PUT</code> but
                <code>POST</code> is another option. Note that this option can also be supplied within the
                <code>options</code> argument to the <code>set</code> method itself.</td>
        </tr>
        <tr>
            <td><code>url</code></td>
            <td><code>String</code></td>
            <td>A URL template, with interpolable elements expressed by terms beginning with the <code>%</code>
                character, for the URL which will be operated by the <code>get</code> and <code>set</code> methods of
                this dataSource.</td>
        </tr>
        <tr>
            <td><code>termMap</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>String</code>)</td>
            <td>A map, of which the keys are some of the interpolation terms held in the <code>url</code> string,
                and the values will be used to perform the interpolation. If a value begins with <code>%</code>,
                the remainder of the string represents a
                <a href="http://docs.fluidproject.org/infusion/development/FrameworkConcepts.html#el-paths">path</a>
                into the <code>directModel</code> argument accepted by the <code>get</code> and <code>set</code>
                methods of the DataSource. By default any such values looked up will be
                <a href="https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent">
                URI Encoded</a> before being interpolated into the URL – unless their value in the termMap is
                prefixed by the string <code>noencode:</code>.</td>
        </tr>
        <tr>
            <td><code>notFoundIsEmpty</code></td>
            <td><code>Boolean</code></a> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a fetch of a nonexistent resource (that is, a
            nonexistent file, or an HTTP resource giving a 404) will result in a <code>resolve</code> with an empty
            payload rather than a <code>reject</code> response.</td>
        </tr>
        <tr>
            <td><code>censorRequestOptionsLog</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>Boolean</code>) (default: <code>{auth: true}</code>)</td>
            <td>A map of members of the <a href="https://nodejs.org/api/http.html#http_http_request_options_callback">
                request options</a> which should be censored from appearing in logs. Any name which maps to <code>true</code>
                will not appear either in the logging output derived from the request options parsed from the url
                or the url itself.
            </td>
        </tr><commit_msg>KETTLE-73: Doc fixes after review<commit_after>---
title: DataSources
layout: default
category: Kettle
---
# DataSources

A DataSource is an Infusion component which meets a simple contract for read/write access to indexed data.
DataSource is a simple semantic, broadly the same as that encoded in
[CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete), although the current DataSource semantic does
not provide explicitly for deletion.

The concrete DataSources in Kettle provide support for HTTP endpoints (with a particular variety specialised for
accessing CouchDB databases with CRUDlike semantics) as well as the filesystem, with an emphasis on JSON payloads.

The DataSource API is drawn from the following two methods – a read-only DataSource will just implement `get`, and a
writeable DataSource will implement both `get` and `set`:

```javascript
    /* @param directModel {Object} A JSON structure holding the "coordinates" of the state to be read -
     * this model is morally equivalent to (the substitutable parts of) a file path or URL
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just
     * this request. These will be specially interpreted by the particular concrete grade of DataSource
     * – there are no options valid across all implementations of this grade.
     * @return {Promise} A promise representing successful or unsuccessful resolution of the read state
     */
    dataSource.get(directModel, options);
    /* @param directModel {Object} As for get
     * @param model {Object} The state to be written to the coordinates
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just
     * this request. These will be specially interpreted by the
     * particular concrete grade of DataSource – there are no options valid across all implementations
     * of this grade. For example, a URL DataSource will accept an option `writeMethod` which will
     * allow the user to determine which HTTP method (PUT or POST) will be used to implement the write
     * operation.
     * @return {Promise} A promise representing resolution of the written state,
     * which may also optionally resolve to any returned payload from the write process
     */
    dataSource.set(directModel, model, options);
```

## Simple example of using an HTTP dataSource

In this example we define and instantiate a simple HTTP-backed dataSource accepting one argument to configure a URL
segment:

```javascript
var fluid = require("infusion"),
    kettle = require("../../kettle.js"),
    examples = fluid.registerNamespace("examples");


fluid.defaults("examples.httpDataSource", {
    gradeNames: "kettle.dataSource.URL",
    url: "http://jsonplaceholder.typicode.com/posts/%postId",
    termMap: {
        postId: "%directPostId"
    }
});

var myDataSource = examples.httpDataSource();
var promise = myDataSource.get({directPostId: 42});

promise.then(function (response) {
    console.log("Got dataSource response of ", response);
}, function (error) {
    console.error("Got dataSource error response of ", error);
});
```

You can run this snippet from our code samples by running `node simpleDataSource.js` from
[examples/simpleDataSource](../examples/simpleDataSource) in our samples area.
This contacts the useful JSON placeholder API service at
[`jsonplaceholder.typicode.com`](http://jsonplaceholder.typicode.com/)
to retrieve a small JSON document holding some placeholder text. If you get a 404 or an error, please contact us and
we'll update this sample to contact a new service.

An interesting element in this snippet is the `termMap` configured as options of our dataSource. This sets up an
indirection between the `directModel` supplied as the argument to the `dataSource.get` call, and the URL issued in the
HTTP request. The keys in the `termMap` are interpolation variables in the URL, which in the URL are prefixed by `%`.
The values in the `termMap` represent either

* Plain values to be interpolated as strings directly into the URL, or
* If the first character of the value in the `termMap` is %, the remainder of the string represents a path which will
  be dereferenced from the `directModel` argument to the current `set` or `get` request.

In addition, if the term value has the prefix `noencode:`, it will be interpolated without any
[URI encoding](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent).

We document these configuration options in the next section:

## Configuration options accepted by `kettle.dataSource.URL`

<table>
    <thead>
        <tr>
            <th colspan="3">Supported configurable options for a <code>kettle.dataSource.URL</code></th>
        </tr>
        <tr>
            <th>Option Path</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>writable</code></td>
            <td><code>Boolean</code> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a <code>set</code> method will be fabricated for this
                dataSource – otherwise, it will implement only a <code>get</code> method.</td>
        </tr>
        <tr>
            <td><code>writeMethod</code></td>
            <td><code>String</code> (default: <code>PUT</code>)</td>
            <td>The HTTP method to be used when the <code>set</code> method is operated on this writable DataSource
                (with <code>writable: true</code>). This defaults to <code>PUT</code> but
                <code>POST</code> is another option. Note that this option can also be supplied within the
                <code>options</code> argument to the <code>set</code> method itself.</td>
        </tr>
        <tr>
            <td><code>url</code></td>
            <td><code>String</code></td>
            <td>A URL template, with interpolable elements expressed by terms beginning with the <code>%</code>
                character, for the URL which will be operated by the <code>get</code> and <code>set</code> methods of
                this dataSource.</td>
        </tr>
        <tr>
            <td><code>termMap</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>String</code>)</td>
            <td>A map, of which the keys are some of the interpolation terms held in the <code>url</code> string,
                and the values will be used to perform the interpolation. If a value begins with <code>%</code>,
                the remainder of the string represents a
                <a href="http://docs.fluidproject.org/infusion/development/FrameworkConcepts.html#el-paths">path</a>
                into the <code>directModel</code> argument accepted by the <code>get</code> and <code>set</code>
                methods of the DataSource. By default any such values looked up will be
                <a href="https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent">
                URI Encoded</a> before being interpolated into the URL – unless their value in the termMap is
                prefixed by the string <code>noencode:</code>.</td>
        </tr>
        <tr>
            <td><code>notFoundIsEmpty</code></td>
            <td><code>Boolean</code></a> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a fetch of a nonexistent resource (that is, a
            nonexistent file, or an HTTP resource giving a 404) will result in a <code>resolve</code> with an empty
            payload rather than a <code>reject</code> response.</td>
        </tr>
        <tr>
            <td><code>censorRequestOptionsLog</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>Boolean</code>) (default:
                <code>{auth: true, "headers.Authorization": true}</code>)
            </td>
            <td>A map of paths into the <a href="https://nodejs.org/api/http.html#http_http_request_options_callback">
                request options</a> which should be censored from appearing in logs. Any path which maps to <code>true</code>
                will not appear either in the logging output derived from the request options parsed from the url
                or the url itself.
            </td>
        </tr>
{'fraction_non_alphanumeric': 0.09839501736734939, 'fraction_numerical': 0.001197748233321356, 'mean_word_length': 3.084882583170254, 'pattern_counts': {'":': 1, '<': 317, '<?xml version=': 0, '>': 317, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6050532', 'n_tokens_mistral': 4456, 'n_tokens_neox': 4277, 'n_words': 1824}