<commit_before># kartotherian-core

The core components of the Kartotherian maps tile service

## Sources
Sources is a way to set up data processing pipelines for Kartotherian and Tilerator.
Sources could be either stored in a set of standalone files, or be embedded in the
main configuration file, or a mix of both. The sources value in the config file
could be a string (file), an object defining the source, or an array of strings and objects.

`uri` is the only mandatory field, and it specifies how [tilelive.js](https://github.com/mapbox/tilelive.js)
will locate and initialize the new source. The protocol determines which tile provider will be used.

Since sometimes not everything can be added as query parameters to the Uri, there is a set of additional keys to help.
Values can either be hardcoded as strings/numbers/booleans, or can be calculated on the fly.

A simple source configuration to set up a tile storage as files in the ./vectors dir:
```
filestore:
    uri: file://./vectors
```
The path can also be set via a parameter:
```
filestore:
    uri: file://
    pathname: ./vectors
```
The value does not have to be given in the source, but instead could be dynamically generated.
For example, the `var` generator pulls the value from the variable store.
The variables are defined in a separate file(s), similar to sources.
```
filestore:
    uri: file://
    pathname: {var: tilepath}  # Uses a variable named tilepath
```

More parameters can be set using `params` - a set of additional values to be set in URI:
```
oz:
  # "overzoom:" is a tile source that will attempt to get a tile from another source,
  # and if tile is missing, attempt to get a portion of the lower-zoom tile.
  uri: overzoom://
  # Specify the tile source - this adds a properly escaped query parameter
  #   ?source=sourceref:///?ref=gen
  param:
    source: {ref: gen}
```

## Value substitutions

In general, these value substitutions are available:
* `{var:varname}` - the value becomes the value of the variable `varname` from the variables file / variables conf section of the main config file. This might be useful if you want to make all the settings public except for the passwords that are stored in a secure location.
* `{ref:sourceId}` - the value becomes a reference to another source. Some sources function as filters/converters, pulling data internally from other sources and converting the result on the fly. For example, the [overzoom](https://github.com/kartotherian/kartotherian-overzoom) source pulls data from another source, and if it's not available, tries to find a lower-zoom tile above the given one, and extract a portion of it. Internally, it uses a forwarding sourceref: source.
* `{npm: ['npm-module-name', 'subdir', 'subdir', 'filename']}`
Some files may be located inside the NPM modules added to the Kartotherian project, i.e. [osm-bright-source](https://github.com/kartotherian/osm-bright.tm2source). To reference a file inside npm, set npm's value to an array, with the first value being the name of the npm module (resolves to the root of the npm module), and all subsequent strings being subdirs and lastly - the name of the file. Subdirs may be ommited:
```
# resolves to a rooted path /.../node_modules/osm-bright-source/data.xml
npm: ["osm-bright-source", "data.xml"]
```

## XML-based sources
The `xml` parameter is used to load and alter XML for some sources like
[tilelive-bridge](https://github.com/mapbox/tilelive-bridge) (SQL→VectorTile or TIFF→RasterTile) and
[tilelive-vector](https://github.com/mapbox/tilelive-vector) (Style VectorTile → PNG).
The `xml` field must evaluate to the xml file path.

```
gen:                # The name of the source (could be referenced later)
  uri: bridge://    # Required - the URI used to construct the source
  xml:              # Init source with this xml instead of the URI's other parameters
    # Set xml to the location of the 'data.xml', which is located inside the osm-bright-source npm
    npm: ["osm-bright-source", "data.xml"]
  xmlSetDataSource: # Before loading, update the datasource section of the standard mapnik config file
    if:             # Only update datasources that match all these values (logical AND)
      dbname: gis   # Instead of 'gis', you can use {npm:...}, {ref:..}, and {var:...}
      host: ''
      type: postgis
    set:            # Replace these keys with the new values
      host: localhost
      user: {var: osmdb-user}  # Instead of hardcoding, use the value from the variables file or conf section
      password: {var: <PASSWORD>}
```

* `xmlSetAttrs` - for xml, overrides the attributes of the root element with the new ones. For example, you may change the font directory of the `<Map>` element:
```
s2:
  uri: vector://
  xml:
    npm: ["osm-bright-style", "project.xml"]    # stylesheet xml is in npm
  xmlSetAttrs:
    font-directory: {npm: ["osm-bright-fonts", "fonts/"]}
```
* `xmlSetParams` - for xml, overrides the top level `<Parameters>` values with the new ones. For example, the `vector` source requires xml stylesheet to point to the proper source of PBFs:
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    npm: ["osm-bright-style", "project.xml"]    # stylesheet xml is in npm
  xmlSetParams:
    source: {ref: gen}                          # set source parameter to the 'gen' source
```
* `xmlLayers` - keep all non-layer data, but only keep those layers that are listed in this value (whitelist):
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    npm: ["osm-bright-style", "project.xml"]    # stylesheet xml is in npm
  xmlLayers: ['landuse', 'road']                # Only include these layers when rendering
```
* `xmlExceptLayers` - same as `xmlLayers`, but instead of whitelisting, blacklist (allow all except these):
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    npm: ["osm-bright-style", "project.xml"]    # stylesheet xml is in npm
  xmlExceptLayers: ['water']                    # Exclude water layer when rendering
```
* `xmlSetDataSource` - change all layer's datasources' parameters if they match conditions:
`if` is a set of parameter values that all must match,
`xmlLayers` and `xmlExcludeLayers` just like above set which layers to address,
and `set` specifies the new parameter values to be set.

Instead of an object, `xmlSetDataSource` can be set to an array of objects to provide
multple change sets.

## Kartotherian-specific parameters:
* `public` (boolean) - should this be source be accessible via `/<sourceId>/z/x/y.format` requests
* `minzoom` (int) - minimum allowable zoom for the public request (public requests only)
* `maxzoom` (int) - maximum allowable zoom for the public request (public requests only)
* `defaultHeaders` (object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)
* `headers` (object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)
* `formats` (array of strings) - one string or a list of string values specifying allowed formats, e.g. `['png','jpeg']`
* `scales` (array of numbers) - one number or a list of number values specifying allowed scalings, e.g. `[1.3, 1.5, 2, 2.6, 3]`
* `pbfsource` (string) - expose a raw vector (pbf) tile source as if it is coming from the current source. All requests to the pbf tiles from the current source will be redirected to the given source.<commit_msg>Added loader info to README<commit_after># kartotherian-core

The core components of the Kartotherian maps tile service

## Sources
Sources is a way to set up data processing pipelines for Kartotherian and Tilerator.
Sources could be either stored in a set of standalone files, or be embedded in the
main configuration file, or a mix of both. The sources value in the config file
could be a string (file), an object defining the source, or an array of strings and objects.

`uri` is the only mandatory field, and it specifies how [tilelive.js](https://github.com/mapbox/tilelive.js)
will locate and initialize the new source. The protocol determines which tile provider will be used.

Since sometimes not everything can be added as query parameters to the Uri, there is a set of additional keys to help.
Values can either be hardcoded as strings/numbers/booleans, or can be calculated on the fly.

A simple source configuration to set up a tile storage as files in the ./vectors dir:
```
filestore:
    uri: file://./vectors
```
The path can also be set via a parameter:
```
filestore:
    uri: file://
    pathname: ./vectors
```
The value does not have to be given in the source, but instead could be dynamically generated.
For example, the `var` generator pulls the value from the variable store.
The variables are defined in a separate file(s), similar to sources.
```
filestore:
    uri: file://
    pathname: {var: tilepath}  # Uses a variable named tilepath
```

More parameters can be set using `params` - a set of additional values to be set in URI:
```
oz:
  # "overzoom:" is a tile source that will attempt to get a tile from another source,
  # and if tile is missing, attempt to get a portion of the lower-zoom tile.
  uri: overzoom://
  # Specify the tile source - this adds a properly escaped query parameter
  #   ?source=sourceref:///?ref=gen
  param:
    source: {ref: gen}
```

## Value substitutions

In general, these value substitutions are available:
* `{var:varname}` - the value becomes the value of the variable `varname` from the variables file / variables conf section of the main config file. This might be useful if you want to make all the settings public except for the passwords that are stored in a secure location.
* `{ref:sourceId}` - the value becomes a reference to another source. Some sources function as filters/converters, pulling data internally from other sources and converting the result on the fly. For example, the [overzoom](https://github.com/kartotherian/kartotherian-overzoom) source pulls data from another source, and if it's not available, tries to find a lower-zoom tile above the given one, and extract a portion of it. Internally, it uses a forwarding sourceref: source.
* `{loader: npm-module-name}` or `{loader: ['npm-module-name', 'arg1', 'arg2', ...]}` - if npm module supports loading customization, it should be loaded via the loader. Loader is only available inside the source's `xml` key.
* `{npm: ['npm-module-name', 'subdir', 'subdir', 'filename']}` - some files may be located inside the NPM modules added to the Kartotherian project, i.e. [osm-bright-source](https://github.com/kartotherian/osm-bright.tm2source). To reference a file inside npm, set npm's value to an array, with the first value being the name of the npm module (resolves to the root of the npm module), and all subsequent strings being subdirs and lastly - the name of the file. Subdirs may be omitted. `npm: ["osm-bright-source", "data.xml"]` would resolve to a rooted path `/.../node_modules/osm-bright-source/data.xml`

## XML-based sources
The `xml` parameter is used to load and alter XML for some sources like
[tilelive-bridge](https://github.com/mapbox/tilelive-bridge) (SQL→VectorTile or TIFF→RasterTile) and
[tilelive-vector](https://github.com/mapbox/tilelive-vector) (Style VectorTile → PNG).
The `xml` field must evaluate to the xml file path.

```
gen:                # The name of the source (could be referenced later)
  uri: bridge://    # Required - the URI used to construct the source
  xml:              # Init source with this xml instead of the URI's other parameters
    # Set xml to the location of the 'data.xml', which is located inside the osm-bright-source npm
    npm: ["osm-bright-source", "data.xml"]
  xmlSetDataSource: # Before loading, update the datasource section of the standard mapnik config file
    if:             # Only update datasources that match all these values (logical AND)
      dbname: gis   # Instead of 'gis', you can use {npm:...}, {ref:..}, and {var:...}
      host: ''
      type: postgis
    set:            # Replace these keys with the new values
      host: localhost
      user: {var: osmdb-user}  # Instead of hardcoding, use the value from the variables file or conf section
      password: {var: osmdb-pswd}
```

* `xmlSetAttrs` - for xml, overrides the attributes of the root element with the new ones. For example, you may change the font directory of the `<Map>` element:
```
s2:
  uri: vector://
  xml:
    loader: osm-bright-style    # stylesheet xml is in npm
  xmlSetAttrs:
    # Note that this is not needed for osm-bright-style because that module does this internally
    font-directory: {npm: ["osm-bright-fonts", "fonts/"]}
```
* `xmlSetParams` - for xml, overrides the top level `<Parameters>` values with the new ones. For example, the `vector` source requires xml stylesheet to point to the proper source of PBFs:
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    loader: osm-bright-style    # stylesheet xml is in npm
  xmlSetParams:
    source: {ref: gen}                          # set source parameter to the 'gen' source
```
* `xmlLayers` - keep all non-layer data, but only keep those layers that are listed in this value (whitelist):
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    loader: osm-bright-style    # stylesheet xml is in npm
  xmlLayers: ['landuse', 'road']                # Only include these layers when rendering
```
* `xmlExceptLayers` - same as `xmlLayers`, but instead of whitelisting, blacklist (allow all except these):
```
s2:
  public: true
  uri: vector://
  formats: [png,json,headers,svg,jpeg]
  xml:
    loader: osm-bright-style    # stylesheet xml is in npm
  xmlExceptLayers: ['water']                    # Exclude water layer when rendering
```
* `xmlSetDataSource` - change all layer's datasources' parameters if they match conditions:
`if` is a set of parameter values that all must match,
`xmlLayers` and `xmlExcludeLayers` just like above set which layers to address,
and `set` specifies the new parameter values to be set.

Instead of an object, `xmlSetDataSource` can be set to an array of objects to provide
multple change sets.

## Kartotherian-specific parameters:
* `public` (boolean) - should this be source be accessible via `/<sourceId>/z/x/y.format` requests
* `minzoom` (int) - minimum allowable zoom for the public request (public requests only)
* `maxzoom` (int) - maximum allowable zoom for the public request (public requests only)
* `defaultHeaders` (object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)
* `headers` (object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)
* `formats` (array of strings) - one string or a list of string values specifying allowed formats, e.g. `['png','jpeg']`
* `scales` (array of numbers) - one number or a list of number values specifying allowed scalings, e.g. `[1.3, 1.5, 2, 2.6, 3]`
* `pbfsource` (string) - expose a raw vector (pbf) tile source as if it is coming from the current source. All requests to the pbf tiles from the current source will be redirected to the given source.
{'fraction_non_alphanumeric': 0.09341660161332292, 'fraction_numerical': 0.0018214936247723133, 'mean_word_length': 3.730153846153846, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 75}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2287286', 'n_tokens_mistral': 4493, 'n_tokens_neox': 4256, 'n_words': 2109}