Add ILM support to APM Server

[simitt]

This PR adds ILM support to the APM Server, see #1290 (comment).

It introduces an apm-server.ilm.enabled setting, that is set to false by default, so that no breaking change is introduced.

The command apm-server setup --template gets deprecated with this PR, in favor of apm-server setup --index-management.

When apm-server setup, or apm-server setup --index-management is run, one apm template gets created defining all of the fields, dynamic mappings, etc. Additionally one minimal template per event type is created, that only holds information about index lifecycle management. When ILM is disabled, no ilm policy and write alias is set, when ILM is enabled, the default ILM policy and write alias is linked in the specific templates, and the policies and write aliases are created. The templates per event only match against the specific event index.

Note that the order of setting up policies, templates and write aliases is important. The templates need to be created before write aliases are created, as this creates an index, and templates will not be applied after index creation.

The current implementation does not support any configuration options for ILM (policies, pattern, etc). As soon as ILM is enabled, configured output.elasticsearch.index and output.elasticsearch.indices settings, as well as setup.template.name, setup.template.pattern settings are ignored.

The general logic for the setup commands are inherited from libbeat. setup.template.enabled and apm-server.ilm.enabled configs are respected when running the setup command. This is important, as otherwise a distinction whether or not ILM should be set up could not be made.

Since this PR depends on unmerged changes in libbeat, it first pulls in the current libbeat master and then adds the changes from the open libbeat PR.

TODO:

• adapt changelog
• pull in libbeat master and remove direct changes of vendor folder.
• discuss and potentially revisit ILM policies
• check with @nachogiljaldo for whitelisting apm-server.ilm.enabled settings

[graphaelli]

I'm still working through the code but some immediate feedback on general functionality:

• apm-server setup --ilm-policy requires apm-server.ilm.enabled=true be set to get ilm templates which I wouldn't expect since --ilm-policy implies that.
• apm-server setup --ilm-policy installs index templates, not sure if that's expected to only create the policies.
• the ilm index templates created seem to bump the mapping.total_fields.limit to 10k (from 2k), though this doesn't seem to be applied to the resulting index on master - still unclear what's happening, but if we could avoid this altogether, we should. Same issue but apparent no-impact for query.default_field. More concretely, after starting apm-server -e -E apm-server.ilm.enabled=true:

note that order: 1, mapping.total_fields.limit and query.default_field are set.

GET _template/apm-8.0.0-span-ilm

{
  "apm-8.0.0-span-ilm" : {
    "order" : 1,
    "index_patterns" : [
      "apm-8.0.0-span-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "apm-8.0.0-span",
          "rollover_alias" : "apm-8.0.0-span"
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "30",
        "query" : {
          "default_field" : [
            "fields.*"
          ]
        }
      }
    },
...

default template is set as expected

GET _template/apm-8.0.0

{
  "apm-8.0.0" : {
    "order" : 0,
    "index_patterns" : [
      "apm*"
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "30",
        "number_of_shards" : "1",
        "query" : {
          "default_field" : [
            "message",
...

the actual template used is fine

GET apm-8.0.0-span/_settings

{
  "apm-8.0.0-span-000001" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "<apm-8.0.0-span-000001>",
        "query" : {
          "default_field" : [
            "message",
...

Creating another index does what I expected/feared:

PUT apm-8.0.0-span-foo
GET apm-8.0.0-span-foo/_settings
{
  "apm-8.0.0-span-foo" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "apm-8.0.0-span",
          "rollover_alias" : "apm-8.0.0-span"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "apm-8.0.0-span-foo",
        "query" : {
          "default_field" : [
            "fields.*"
          ]
        },
...

What am I missing here, is there some ordering issue?

[simitt]

Closing for now, as some more changes will come through libbeat changes. Reopen once the issues regarding setup and export cmds are addressed in its core.

[simitt]

jenkins, test this please

[simitt]

@elastic/apm-server is adding the ILM related behaviour and is ready for review now.

[graphaelli]

This is amazing work 🎉 - all of my previous concerns have been addressed and nothing new has come up, I think it's ready to 🚢

[simitt]

Thanks @graphaelli! I had one more change - removed the force_merge from the policy, after some more digging into it, as this could be a heavy operation better suited to be scheduled by the users.