Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] ILM documentation #2231

Merged
merged 5 commits into from
Jun 12, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuring.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ include::./copied-from-beats/shared-ssl-config.asciidoc[]

include::./template-config.asciidoc[]

include::./ilm-setup.asciidoc[]
include::./ilm.asciidoc[]

include::./copied-from-beats/loggingconfig.asciidoc[]

Expand Down
7 changes: 7 additions & 0 deletions docs/copied-from-beats/command-reference.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -636,8 +636,15 @@ enabled modules in the +{beatname_lc}.yml+ file. If you used the
directory, also specify the `--modules` flag.
endif::[]

ifndef::apm-server[]
*`--template`*::
Sets up the index template only.
endif::apm-server[]

ifdef::apm-server[]
*` --index-management`*::
Sets up <<ilm,index lifecycle management>>.
endif::[]

{global-flags}

Expand Down
11 changes: 11 additions & 0 deletions docs/guide/apm-release-notes.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ For a full list of changes, see the
{apm-server-ref-v}/release-notes.html[APM Server Release Notes] or the
{kibana-ref}/release-notes.html[Kibana Release Notes].

* <<release-highlights-7.2.0>>
* <<release-highlights-7.1.0>>
* <<release-highlights-7.0.0>>
* <<release-highlights-6.8.0>>
Expand All @@ -16,6 +17,16 @@ For a full list of changes, see the
* <<release-notes-6.4.1>>
* <<release-notes-6.4.0>>

[[release-highlights-7.2.0]]
=== APM version 7.2.0

[float]
==== New features

*APM Server*

* Support for {apm-server-ref-v}/ilm.html[index lifecycle management (ILM)]

[[release-highlights-7.1.0]]
=== APM version 7.1.0

Expand Down
10 changes: 3 additions & 7 deletions docs/ilm-setup.asciidoc
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
[role="xpack"]
[[manual-ilm-setup]]
== Manual index lifecycle management
=== Manual index lifecycle management

APM Server can take advantage of the index lifecycle management (ILM) capabilities of Elasticsearch.
ILM enables you to automate how you want to manage your indices over time.
You can base actions on factors such as shard size and performance requirements.

The guide below will help you set up a custom ILM policy for span indices.
You can repeat the actions for any other indices.
The guide below will help you set up a custom index lifecycle management policy for span indices.
You can repeat these actions for any other indices (transactions, errors, etc.).

NOTE: If you're migrating from an existing setup,
any indices present before ILM was configured will need to be managed manually.
Expand Down
131 changes: 131 additions & 0 deletions docs/ilm.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
[[ilm]]
[role="xpack"]
== Index lifecycle management (ILM)

Use the {ref}/getting-started-index-lifecycle-management.html[index lifecycle management (ILM)]
feature in {es} to manage your APM Server indices as they age.
ILM enables you to automate how you want to manage your indices over time,
by automating rollovers to a new index when the existing index reaches a specified size or age.

You can view and edit the index lifecycle policies in the *Index lifecycle policies* UI in {kib}.
For more information about working with the UI,
see {kibana-ref}/index-lifecycle-policies.html[Index lifecycle policies].

APM Server currently offers two ways to get started with ILM:

1. <<ilm-default,**ILM default policy**>> - Get up and running with a default index lifecycle management policy as quickly as possible.
2. <<manual-ilm-setup,**Manual ILM**>> - Customize and manage your own ILM policies.

NOTE: If you're migrating from an existing setup,
any indices present before ILM was configured will need to be managed manually.

[[ilm-default]]
=== ILM default policy

Index lifecycle management will manage an index based on its defined policy.
The default ILM configuration applies hot and warm phase policies.
Cold and delete phases are not defined.
Because errors and spans lose information value faster than metrics and transactions do,
there are two different policies defined:
one for `errors` and `spans`, and one for `metrics` and `transactions`.

[options="header"]
|=======================================================================
|Event type |Hot |Warm
|Errors & Spans
|Rollover: `max_size: 50gb`, `max_age: 1 day`,
`priority: 100`
|`min_age: 7 days`,
readonly,
`priority: 50`
|Transactions & Metrics
|Rollover: `max_size: 50gb`, `max_age: 7 days`,
`priority: 100`
|`min_age: 31 days`,
readonly,
`priority: 50`
|=======================================================================

IMPORTANT: Changes to the default index lifecycle policy do not take effect until the current index has rolled over.
If you'd like to manage a custom policy, see <<manual-ilm-setup,Manual ILM>>.

[float]
[[ilm-setup]]
==== ILM default policy setup

To set up index lifecycle management, set `ilm.enabled` to `true` in apm-server.yml.
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
ILM can only be enabled for `output.elasticsearch`.
When enabled, configurations defined for `output.elasticsearch.index` and `output.elasticsearch.indices` will be ignored.

It is recommended to set up index lifecycle management (ILM) before starting APM Server.
This excludes setup from the ingestion process, which allows you to ensure ILM is set up correctly before using APM.

Run the <<setup-command>> with the ` --index-management` flag to set up the default ILM policy:

[source,js]
-----------------------
./apm-server setup --index-management
-----------------------
// CONSOLE

You can confirm the policy was created with the GET lifecycle policy API.
Here's what the transaction response looks like:

[source,js]
-----------------------
GET _ilm/policy
{
"apm-7.2.0-transaction": {
"version": 1,
"modified_date": "2019-05-28T15:55:26.791Z",
"policy": {
"phases": {
"warm": {
"min_age": "31d",<1>
"actions": {
"readonly": {},
"set_priority": {
"priority": 50
}
}
},
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50gb",<2>
"max_age": "7d"<3>
},
"set_priority": {
"priority": 100<4>
}
}
}
}
}
}
}
-----------------------
// CONSOLE
<1> Move to warm phase after _31 days_
<2> Rollover after _50gb_
<3> Rollover after _7 days_
<4> Priority for recovering your indices after a node restart. Higher priorities are recovered first.

Your indices are now configured to use index lifecycle management. Go ahead and <<setting-up-and-running,run APM Server>>.

NOTE: If you switch between ILM enabled/disabled multiple times,
you should set `setup.template.overwrite` to `true` to ensure a complete setup.

[float]
==== ILM default policy upgrades

If you decide to customize the default ILM policy, any customizations will be overwritten when you upgrade APM Server.
Default policies are also subject to change in future releases

// Policies are versioned so they can change.
// Indices are versioned so they can change.
// An upgrade creates new templates, policies, and indices.
// If you customize anything, it will revert back to the default during an upgrade

include::./ilm-setup.asciidoc[]