<commit_before>[[docs-reindex]]
== Reindex API

IMPORTANT: Reindex requires <<mapping-source-field,`_source`>> to be enabled for
all documents in the source index.

IMPORTANT: Reindex does not attempt to set up the destination index.  It does
not copy the settings of the source index.  You should set up the destination
index prior to running a `_reindex` action, including setting up mappings, shard
counts, replicas, etc.

The most basic form of `_reindex` just copies documents from one index to another.
This will copy documents from the `twitter` index into the `new_twitter` index:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:big_twitter]

That will return something like this:

[source,js]
--------------------------------------------------
{
  "took" : 147,
  "timed_out": false,
  "created": 120,
  "updated": 0,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "total": 120,
  "failures" : [ ]
}
--------------------------------------------------
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]

Just like <<docs-update-by-query,`_update_by_query`>>, `_reindex` gets a
snapshot of the source index but its target must be a **different** index so
version conflicts are unlikely. The `dest` element can be configured like the
index API to control optimistic concurrency control. Just leaving out
`version_type` (as above) or setting it to `internal` will cause Elasticsearch
to blindly dump documents into the target, overwriting any that happen to have
the same type and id:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "internal"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

Setting `version_type` to `external` will cause Elasticsearch to preserve the
`version` from the source, create any documents that are missing, and update
any documents that have an older version in the destination index than they do
in the source index:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "external"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

Settings `op_type` to `create` will cause `_reindex` to only create missing
documents in the target index. All existing documents will cause a version
conflict:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

By default version conflicts abort the `_reindex` process but you can just
count them by settings `"conflicts": "proceed"` in the request body:

[source,js]
--------------------------------------------------
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

You can limit the documents by adding a type to the `source` or by adding a
query. This will only copy tweets made by `kimchy` into `new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter",
    "type": "_doc",
    "query": {
      "term": {
        "user": "kimchy"
      }
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

`index` and `type` in `source` can both be lists, allowing you to copy from
lots of sources in one request. This will copy documents from the `_doc` and
`post` types in the `twitter` and `blog` index. The copied documents would include the
`post` type in the `twitter` index and the `_doc` type in the `blog` index. For more
specific parameters, you can use `query`.

The Reindex API makes no effort to handle ID collisions. For such issues, the target index
will remain valid, but it's not easy to predict which document will survive because
the iteration order isn't well defined.

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": ["twitter", "blog"],
    "type": ["_doc", "post"]
  },
  "dest": {
    "index": "all_together"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[s/^/PUT twitter\nPUT blog\n/]

It's also possible to limit the number of processed documents by setting
`size`. This will only copy a single document from `twitter` to
`new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "size": 1,
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

If you want a particular set of documents from the `twitter` index you'll
need to use `sort`. Sorting makes the scroll less efficient but in some contexts
it's worth it. If possible, prefer a more selective query to `size` and `sort`.
This will copy 10000 documents from `twitter` into `new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "size": 10000,<commit_msg>Reindex: Clean up docs around multi-index<commit_after>[[docs-reindex]]
== Reindex API

IMPORTANT: Reindex requires <<mapping-source-field,`_source`>> to be enabled for
all documents in the source index.

IMPORTANT: Reindex does not attempt to set up the destination index.  It does
not copy the settings of the source index.  You should set up the destination
index prior to running a `_reindex` action, including setting up mappings, shard
counts, replicas, etc.

The most basic form of `_reindex` just copies documents from one index to another.
This will copy documents from the `twitter` index into the `new_twitter` index:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:big_twitter]

That will return something like this:

[source,js]
--------------------------------------------------
{
  "took" : 147,
  "timed_out": false,
  "created": 120,
  "updated": 0,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "total": 120,
  "failures" : [ ]
}
--------------------------------------------------
// TESTRESPONSE[s/"took" : 147/"took" : "$body.took"/]

Just like <<docs-update-by-query,`_update_by_query`>>, `_reindex` gets a
snapshot of the source index but its target must be a **different** index so
version conflicts are unlikely. The `dest` element can be configured like the
index API to control optimistic concurrency control. Just leaving out
`version_type` (as above) or setting it to `internal` will cause Elasticsearch
to blindly dump documents into the target, overwriting any that happen to have
the same type and id:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "internal"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

Setting `version_type` to `external` will cause Elasticsearch to preserve the
`version` from the source, create any documents that are missing, and update
any documents that have an older version in the destination index than they do
in the source index:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "external"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

Settings `op_type` to `create` will cause `_reindex` to only create missing
documents in the target index. All existing documents will cause a version
conflict:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

By default version conflicts abort the `_reindex` process but you can just
count them by settings `"conflicts": "proceed"` in the request body:

[source,js]
--------------------------------------------------
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

You can limit the documents by adding a type to the `source` or by adding a
query. This will only copy tweets made by `kimchy` into `new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": "twitter",
    "type": "_doc",
    "query": {
      "term": {
        "user": "kimchy"
      }
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

`index` and `type` in `source` can both be lists, allowing you to copy from
lots of sources in one request. This will copy documents from the `_doc` and
`post` types in the `twitter` and `blog` indices.

[source,js]
--------------------------------------------------
POST _reindex
{
  "source": {
    "index": ["twitter", "blog"],
    "type": ["_doc", "post"]
  },
  "dest": {
    "index": "all_together",
    "type": "_doc"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]
// TEST[s/^/PUT blog\/post\/post1?refresh\n{"test": "foo"}\n/]

NOTE: The Reindex API makes no effort to handle ID collisions so the last
document written will "win" but the order isn't usually predictable so it is
not a good idea to rely on this behavior. Instead, make sure that IDs are unique
using a script.

It's also possible to limit the number of processed documents by setting
`size`. This will only copy a single document from `twitter` to
`new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "size": 1,
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

If you want a particular set of documents from the `twitter` index you'll
need to use `sort`. Sorting makes the scroll less efficient but in some contexts
it's worth it. If possible, prefer a more selective query to `size` and `sort`.
This will copy 10000 documents from `twitter` into `new_twitter`:

[source,js]
--------------------------------------------------
POST _reindex
{
  "size": 10000,
{'fraction_non_alphanumeric': 0.27151167838846924, 'fraction_numerical': 0.005991143526960146, 'mean_word_length': 1.9255778511557022, 'pattern_counts': {'":': 120, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '111869', 'n_tokens_mistral': 3332, 'n_tokens_neox': 3217, 'n_words': 1189}