diff --git a/docs/en/ingest-management/data-streams.asciidoc b/docs/en/ingest-management/data-streams.asciidoc index a588786290..ef51546ac6 100644 --- a/docs/en/ingest-management/data-streams.asciidoc +++ b/docs/en/ingest-management/data-streams.asciidoc @@ -97,146 +97,17 @@ These templates are loaded when the integration is installed, and are used to co [[data-streams-ilm]] == Configure an index lifecycle management (ILM) policy -Use the {ref}/index-lifecycle-management.html[index lifecycle +Use the {ref}/getting-started-index-lifecycle-management.html[index lifecycle management] (ILM) feature in {es} to manage your {agent} data stream indices as they age. For example, create a new index after a certain period of time, or delete stale indices to enforce data retention standards. -Installed integrations may have one or many associated data streams--each with an associated ILM policy. -By default, these data streams use an ILM policy that matches their data type. -For example, the data stream `metrics-system.logs-*`, -uses the metrics ILM policy as defined in the `metrics-system.logs` index template. +{agent} uses ILM policies built-in to {es} to manage backing indices for its data streams. +See the {ref}/example-using-index-lifecycle-policy.html[Customize built-in ILM policies] tutorial +to learn how to customize these policies based on your performance, resilience, and retention requirements. -[discrete] -[[data-streams-ilm-tutorial]] -== Tutorial: Customize data retention for integrations - -This tutorial explains how to apply a custom ILM policy to an integration's data stream. - -**Scenario:** You have {agent}s collecting system metrics with the System integration in two environments-- -one with the namespace `development`, and one with `production`. - -**Goal:** Customize the ILM policy for the `system.network` data stream with the `production` namespace. -Specifically, apply the built-in `90-days-default` ILM policy so that data is deleted after 90 days. - -[discrete] -[[data-streams-ilm-one]] -=== Step 1: View data streams - -The **Data Streams** view in {kib} shows you the data streams, -index templates, and ILM policies associated with a given integration. - -. Navigate to **Stack Management** > **Index Management** > **Data Streams**. -. Search for `system` to see all data streams associated with the System integration. -. Select the `metrics-system.network-{namespace}` data stream to view its associated index template and ILM policy. -As you can see, the data stream follows the <> and starts with its type, `metrics-`. -+ -[role="screenshot"] -image::images/data-stream-info.png[Data streams info] - -[discrete] -[[data-streams-ilm-two]] -=== Step 2: Create a component template - -For your changes to continue to be applied in future versions, -you must put all custom index settings into a component template. -The component template must start with `.`, follow the data stream naming scheme, -and end with `@custom`: - -[source,text] ----- -.--@custom ----- - -For example, to create custom index settings for the `system.network` data stream with a namespace of `production`, -the component template name would be: - -[source,text] ----- -.metrics-system.network-production@custom ----- - -. Navigate to **Stack Management** > **Index Management** > **Component Templates** -. Click **Create component template**. -. Use the template above to set the name--in this case, `.metrics-system.network-production@custom`. Click **Next**. -. Under **Index settings**, set the ILM policy name under the `lifecycle.name` key: -+ -[source,json] ----- -{ - "lifecycle": { - "name": "90-days-default" - } -} ----- -. Continue to **Review** and ensure your request looks similar to the image below. -If it does, click **Create component template**. -+ -[role="screenshot"] -image::images/create-component-template.png[Create component template] - -[discrete] -[[data-streams-ilm-three]] -=== Step 3: Clone and modify the existing index template - -Now that you've created a component template, -you need to create an index template to apply the changes to the correct data stream. -The easiest way to do this is to duplicate and modify the integration's existing index template. - -WARNING: When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading. - -. Navigate to **Stack Management** > **Index Management** > **Index Templates**. -. Find the index template you want to clone. The index template will have the `` and `` in its name, -but not the ``. In this case, it's `metrics-system.network`. -. Select **Actions** > **Clone** -. Set the name of the new index template to `metrics-system.network-production`. -. Change the index pattern to include a namespace-- in this case, `metrics-system.network-production*`. -This ensures the previously created component template is only applied to the `production` namespace. -. Set the priority to `250`. This ensures that the new index template takes precedence over other index templates that match the index pattern. -. Under **Component templates**, search for and add the component template created in the previous step. -To ensure your namespace-specific settings are applied over other custom settings, -the new template should be added below the existing `@custom` template. -. Create the index template. - -[role="screenshot"] -image::images/create-index-template.png[Create index template] - -[discrete] -[[data-streams-ilm-four]] -=== Step 4: Roll over the data stream (optional) - -To confirm that the data stream is now using the new index template and ILM policy, -you can either repeat <>, or navigate to **Dev Tools ** and run the following: - -[source,bash] ----- -GET /_data_stream/metrics-system.network-production <1> ----- -<1> The name of the data stream we've been hacking on - -The result should include the following: - -[source,json] ----- -{ - "data_streams" : [ - { - ... - "template" : "metrics-system.network-production", <1> - "ilm_policy" : "90-days-default", <2> - ... - } - ] -} ----- -<1> The name of the custom index template created in step three -<2> The name of the ILM policy applied to the new component template in step two - -New ILM policies only take effect when new indices are created, -so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50GB), -or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]: - -[source,bash] ----- -POST /metrics-system.network-production/_rollover/ ----- +To instead create a new ILM policy, in {kib}, +go to **Stack Management** > **Index Lifecycle Policies**. Click **Create policy**. +Define data tiers for your data, and any associated actions, +like a rollover, freeze, or shrink. +See {ref}/set-up-lifecycle-policy.html[configure a lifecycle policy] for more information. diff --git a/docs/en/ingest-management/images/create-component-template.png b/docs/en/ingest-management/images/create-component-template.png deleted file mode 100644 index cf186bb93d..0000000000 Binary files a/docs/en/ingest-management/images/create-component-template.png and /dev/null differ diff --git a/docs/en/ingest-management/images/create-index-template.png b/docs/en/ingest-management/images/create-index-template.png deleted file mode 100644 index d526039e6b..0000000000 Binary files a/docs/en/ingest-management/images/create-index-template.png and /dev/null differ diff --git a/docs/en/ingest-management/images/data-stream-info.png b/docs/en/ingest-management/images/data-stream-info.png deleted file mode 100644 index 4fe3884bca..0000000000 Binary files a/docs/en/ingest-management/images/data-stream-info.png and /dev/null differ