<commit_before>[role="xpack"]
[[set-up-a-data-stream]]
== Set up a data stream

To set up a data stream, follow these steps:

* <<create-index-lifecycle-policy>>
* <<create-component-templates>>
* <<create-index-template>>
* <<create-data-stream>>
* <<secure-data-stream>>

You can also <<convert-index-alias-to-data-stream,convert an index alias to
a data stream>>.

IMPORTANT: If you use {fleet} or {agent}, skip this tutorial. {fleet} and
{agent} set up data streams for you. See {fleet}'s
{fleet-guide}/data-streams.html[data streams] documentation.

[discrete]
[[create-index-lifecycle-policy]]
=== Step 1. Create an index lifecycle policy

While optional, we recommend using {ilm-init} to automate the management of your
data stream's backing indices. {ilm-init} requires an index lifecycle policy.

To create an index lifecycle policy in {kib}, open the main menu and go to
*Stack Management > Index Lifecycle Policies*. Click *Create policy*.

You can also use the <<ilm-put-lifecycle,create lifecycle policy API>>.

[source,console]
----
PUT _ilm/policy/my-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my-snapshot-repo"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my-snapshot-repo"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----

[discrete]
[[create-component-templates]]
=== Step 2. Create component templates

A data stream requires a matching index template. In most cases, you compose
this index template using one or more component templates. You typically use
separate component templates for mappings and index settings. This lets you
reuse the component templates in multiple index templates.

When creating your component templates, include:

* A <<date,`date`>> or <<date_nanos,`date_nanos`>> mapping for the `@timestamp`
field. If you don't specify a mapping, {es} maps `@timestamp` as a `date` field
with default options.

* Your lifecycle policy in the `index.lifecycle.name` index setting.

To create a component template in {kib}, open the main menu and go to *Stack
Management > Index Management*. In the *Index Templates* view, click *Create a
component template*.

You can also use the <<indices-component-template,create component template
API>>.

[source,console]
----
# Creates a component template for mappings
PUT _component_template/my-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "wildcard"<commit_msg>[DOCS] Add ECS and runtime fields tip to data stream tutorial (#71183)<commit_after>[role="xpack"]
[[set-up-a-data-stream]]
== Set up a data stream

To set up a data stream, follow these steps:

* <<create-index-lifecycle-policy>>
* <<create-component-templates>>
* <<create-index-template>>
* <<create-data-stream>>
* <<secure-data-stream>>

You can also <<convert-index-alias-to-data-stream,convert an index alias to
a data stream>>.

IMPORTANT: If you use {fleet} or {agent}, skip this tutorial. {fleet} and
{agent} set up data streams for you. See {fleet}'s
{fleet-guide}/data-streams.html[data streams] documentation.

[discrete]
[[create-index-lifecycle-policy]]
=== Step 1. Create an index lifecycle policy

While optional, we recommend using {ilm-init} to automate the management of your
data stream's backing indices. {ilm-init} requires an index lifecycle policy.

To create an index lifecycle policy in {kib}, open the main menu and go to
*Stack Management > Index Lifecycle Policies*. Click *Create policy*.

You can also use the <<ilm-put-lifecycle,create lifecycle policy API>>.

[source,console]
----
PUT _ilm/policy/my-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my-snapshot-repo"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my-snapshot-repo"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----

[discrete]
[[create-component-templates]]
=== Step 2. Create component templates

A data stream requires a matching index template. In most cases, you compose
this index template using one or more component templates. You typically use
separate component templates for mappings and index settings. This lets you
reuse the component templates in multiple index templates.

When creating your component templates, include:

* A <<date,`date`>> or <<date_nanos,`date_nanos`>> mapping for the `@timestamp`
field. If you don't specify a mapping, {es} maps `@timestamp` as a `date` field
with default options.

* Your lifecycle policy in the `index.lifecycle.name` index setting.

[TIP]
====
Use the {ecs-ref}[Elastic Common Schema (ECS)] when mapping your fields. ECS
fields integrate with several {stack} features by default.

If you're unsure how to map your fields, use <<runtime-search-request,runtime
fields>> to extract fields from <<mapping-unstructured-content,unstructured
content>> at search time. For example, you can index a log message to a
`wildcard` field and later extract IP addresses and other data from this field
during a search.
====

To create a component template in {kib}, open the main menu and go to *Stack
Management > Index Management*. In the *Index Templates* view, click *Create a
component template*.

You can also use the <<indices-component-template,create component template
API>>.

[source,console]
----
# Creates a component template for mappings
PUT _component_template/my-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "wildcard"
{'fraction_non_alphanumeric': 0.12706713780918727, 'fraction_numerical': 0.005512367491166078, 'mean_word_length': 2.7618288144603933, 'pattern_counts': {'":': 72, '<': 47, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7246712', 'n_tokens_mistral': 2141, 'n_tokens_neox': 2090, 'n_words': 690}