<commit_before>[[configuration]]
== Configuration

{eh} behavior can be customized through the properties below, typically by setting them on the target job's Hadoop `Configuration`. However some of them can be specified through other means depending on the library used (see the relevant section).

****
{eh} uses the same conventions and reasonable defaults as {es} so you can try it out without bothering with the configuration. Most of the time, these defaults are just fine for running a production cluster; if you are fine-tunning your cluster or wondering about the effect of certain configuration options, please _do ask_ for more information.
****

NOTE: All configuration properties start with the `es` prefix. The namespace `es.internal` is reserved by the library for its internal use and should _not_ be used by the user at any point.

[float]
=== Required settings

`es.resource`::
{es} resource location, where data is read _and_ written to. Requires the format `<index>/<type>` (relative to the {es} host/port (see <<cfg-network,below>>))).

[source,ini]
----
es.resource = twitter/tweet   # index 'twitter', type 'tweet'
----

`es.resource.read` (defaults to `es.resource`)::
{es} resource used for reading (but not writing) data. Useful when reading and writing data to different {es} indices within the _same_ job. Typically set automatically (except for the {mr} module which requires manual configuration).

`es.resource.write`(defaults to `es.resource`)::
{es} resource used for writing (but not reading) data. Used typically for __dynamic resource__ writes or when writing and reading data to different {es} indices within the _same_ job. Typically set automatically (except for the {mr} module which requires manual configuration).

Note that https://www.elastic.co/guide/en/elasticsearch/guide/current/multi-index-multi-type.html[multiple] indices and/or types are allowed *only* for reading. Use `_all/types` to search `types` in all indices or `index/` to search
all types within `index`.
Do note that reading multiple indices/types typically works only when they have the same structure and only with some libraries. Integrations that require a strongly typed mapping (such as a table like Hive or SparkSQL) are likely to fail.

[[cfg-multi-writes]]
[float]
===== Dynamic/multi resource writes

For writing, {eh} allows the target resource to be resolved at runtime by using patterns (by using the `{<field-name>}` format), resolved at runtime based on the data being streamed to {es}. That is, one can save documents to a certain `index` or `type` based on one or multiple fields resolved from the document about to be saved.

For example, assuming the following document set (described here in JSON for readability - feel free to translate this into the actual Java objects):

[source,json]
----
{
    "media_type":"game",
    "title":"Final Fantasy VI",
    "year":"1994"
},
{
    "media_type":"book",
    "title":"<NAME>",
    "year":"2010"
},<commit_msg>[DOCS] Clarifying docs around multi-resource reads and writes<commit_after>[[configuration]]
== Configuration

{eh} behavior can be customized through the properties below, typically by setting them on the target job's Hadoop `Configuration`. However some of them can be specified through other means depending on the library used (see the relevant section).

****
{eh} uses the same conventions and reasonable defaults as {es} so you can try it out without bothering with the configuration. Most of the time, these defaults are just fine for running a production cluster; if you are fine-tunning your cluster or wondering about the effect of certain configuration options, please _do ask_ for more information.
****

NOTE: All configuration properties start with the `es` prefix. The namespace `es.internal` is reserved by the library for its internal use and should _not_ be used by the user at any point.

[float]
=== Required settings

`es.resource`::
{es} resource location, where data is read _and_ written to. Requires the format `<index>/<type>` (relative to the {es} host/port (see <<cfg-network,below>>))).

[source,ini]
----
es.resource = twitter/tweet   # index 'twitter', type 'tweet'
----

`es.resource.read` (defaults to `es.resource`)::
{es} resource used for reading (but not writing) data. Useful when reading and writing data to different {es} indices within the _same_ job. Typically set automatically (except for the {mr} module which requires manual configuration).

`es.resource.write`(defaults to `es.resource`)::
{es} resource used for writing (but not reading) data. Used typically for __dynamic resource__ writes or when writing and reading data to different {es} indices within the _same_ job. Typically set automatically (except for the {mr} module which requires manual configuration).

Note that specifying https://www.elastic.co/guide/en/elasticsearch/guide/current/multi-index-multi-type.html[multiple]
indices and/or types in the above resource settings are allowed *only* for reading. Specifying multiple indices for
writes is only supported through the use of dynamic resource (described <<cfg-multi-writes, below>>). Use `_all/types`
to search `types` in all indices or `index/` to search all types within `index`. Do note that reading multiple
indices/types typically works only when they have the same structure and only with some libraries. Integrations that
require a strongly typed mapping (such as a table like Hive or SparkSQL) are likely to fail.

[[cfg-multi-writes]]
[float]
===== Dynamic/multi resource writes

For writing, {eh} allows the target resource to be resolved at runtime by using patterns (by using the `{<field-name>}` format), resolved at runtime based on the data being streamed to {es}. That is, one can save documents to a certain `index` or `type` based on one or multiple fields resolved from the document about to be saved.

For example, assuming the following document set (described here in JSON for readability - feel free to translate this into the actual Java objects):

[source,json]
----
{
    "media_type":"game",
    "title":"Final Fantasy VI",
    "year":"1994"
},
{
    "media_type":"book",
    "title":"<NAME>",
    "year":"2010"
},
{'fraction_non_alphanumeric': 0.09115843820587287, 'fraction_numerical': 0.0025814778960955146, 'mean_word_length': 4.7133640552995395, 'pattern_counts': {'":': 12, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5006516', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1550, 'n_words': 836}