<commit_before>==========================
Elasticsearch InOut Plugin
==========================

This Elasticsearch plugin provides the ability to export data by query
on server side, by outputting the data directly on the according node.
The export can happen on all indexes, on a specific index or on a specific
document type.

The data will get exported as one json object per line::

    {"_id":"id1","_source":{"type":"myObject","value":"value1"},"_version":1}
    {"_id":"id2","_source":{"type":"myObject","value":"value2"},"_version":2}


Usage
=====

Examples
--------

Below are some examples demonstrating what can be done with the elasticsearch
inout plugin. The example commands require installation on a UNIX system.
The plugin may also works with different commands on other operating
systems supporting elasticsearch, but is not tested yet.

Export data to files in the node's file system. The filenames will be expanded
by index and shard names (p.e. /tmp/dump-myIndex-0)::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/dump-${index}-${shard}"
    }
    '

Do GZIP compression on file exports::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/dump-${index}-${shard}.gz",
        "compression": "gzip"
    }
    '

Pipe the export data through a single argumentless command on the corresponding
node, like `cat`. This command actually returns the export data in the JSON
result's stdout field::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_cmd": "cat"
    }
    '

Pipe the export data through argumented commands (p.e. a shell script, or
provide your own sophisticated script on the node). This command will
result in transforming the data to lower case and write the file to the
node's file system::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_cmd": ["/bin/sh", "-c", "tr [A-Z] [a-z] > /tmp/outputcommand.txt"]
    }
    '

Limit the exported data with a query. The same query syntax as for search can
be used::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/query-${index}-${shard}",
        "query": {
            "match": {
                "someField": "someValue"
            }
        }
    }
    '


Elements of the request body
----------------------------

``fields``
~~~~~~~~~~

A list of fields to export. Each field must be defined in the mapping.

    "fields": ["name", "address"]

The mapping of fields to export has to be defined with ``"store": true``.
Unlike the ``_search`` endpoint, the field ``_version`` will also be
exported if defined as field (without the additional parameter
``"version": true``).

- Required

``output_cmd``
~~~~~~~~~~~~~~

    "output_cmd": "cat"

    "output_cmd": ["/location/yourcommand", "argument1", "argument2"]

The command to execute. Might be defined as string or as array. The
content to export will get piped to Stdin of the command to execute.
Some variable substitution is possible (see Variable Substitution)

- Required (if ``output_file`` has been omitted)<commit_msg>More explanation<commit_after>==========================
Elasticsearch InOut Plugin
==========================

This Elasticsearch plugin provides the ability to export data by query
on server side, by outputting the data directly on the according node.
The export can happen on all indexes, on a specific index or on a specific
document type.

The data will get exported as one json object per line::

    {"_id":"id1","_source":{"type":"myObject","value":"value1"},"_version":1}
    {"_id":"id2","_source":{"type":"myObject","value":"value2"},"_version":2}


Usage
=====

Examples
--------

Below are some examples demonstrating what can be done with the elasticsearch
inout plugin. The example commands require installation on a UNIX system.
The plugin may also works with different commands on other operating
systems supporting elasticsearch, but is not tested yet.

Export data to files in the node's file system. The filenames will be expanded
by index and shard names (p.e. /tmp/dump-myIndex-0)::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/dump-${index}-${shard}"
    }
    '

Do GZIP compression on file exports::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/dump-${index}-${shard}.gz",
        "compression": "gzip"
    }
    '

Pipe the export data through a single argumentless command on the corresponding
node, like `cat`. This command actually returns the export data in the JSON
result's stdout field::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_cmd": "cat"
    }
    '

Pipe the export data through argumented commands (p.e. a shell script, or
provide your own sophisticated script on the node). This command will
result in transforming the data to lower case and write the file to the
node's file system::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_cmd": ["/bin/sh", "-c", "tr [A-Z] [a-z] > /tmp/outputcommand.txt"]
    }
    '

Limit the exported data with a query. The same query syntax as for search can
be used::

    curl -X POST 'http://localhost:9200/_export' -d '{
        "fields": ["_id", "_source", "_version"],
        "output_file": "/tmp/query-${index}-${shard}",
        "query": {
            "match": {
                "someField": "someValue"
            }
        }
    }
    '


Elements of the request body
----------------------------

``fields``
~~~~~~~~~~

A list of fields to export. Describes which data is exported for every
object. A field name can be any property that is defined in the index/type
mapping with ``"store": "yes"`` or one of the following special fields
(prefixed with _)::

    * _id: Delivers the ID of the object
    * _index: Delivers the index of the object
    * _source: Delivers the stored JSON values of the object
    * _type: Delivers the document type of the object
    * _version: Delivers the current version of the object

Example assuming that the properties ``name`` and ``address`` are defined
in the index/type mapping with the property ``"store": "yes"``::

    "fields": ["_id", "name", "address"]

The ``fields`` element is required in the POST data of the request.

``output_cmd``
~~~~~~~~~~~~~~

    "output_cmd": "cat"

    "output_cmd": ["/location/yourcommand", "argument1", "argument2"]

The command to execute. Might be defined as string or as array. The
content to export will get piped to Stdin of the command to execute.
Some variable substitution is possible (see Variable Substitution)

- Required (if ``output_file`` has been omitted)
{'fraction_non_alphanumeric': 0.15276804484933426, 'fraction_numerical': 0.008128941836019622, 'mean_word_length': 3.3565323565323566, 'pattern_counts': {'":': 58, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2514011', 'n_tokens_mistral': 2077, 'n_tokens_neox': 1971, 'n_words': 830}