[Fleet] Revert custom ILM policy documentation

[hop-dev]

As part of elastic/observability-docs#986 we documented how customers could apply a custom ILM policy to integration data streams by duplicating the data streams' index template and applying a higher priority. The documentation is here.

We have realised that the documentation is based on the false assumption that mappings are applied through @mapping component templates, when in fact they are set on the index template itself. This means that if a user followed the guide, then the new index template they create would not receive mapping updates when integrations are updated.

In the short term I think we need to revert this documentation and remove the UI link from 7.16 while we look at our options (Issue for adding the link #108554)

The APM team also used this document as a base for elastic/apm-server#6553 so this will need amending as well.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

+1 on this decision. It also seems the behavior differs from stream-to-stream. Some streams do have a @mappings template that only contains { "dynamic" : false } while others do not have one at all. Curious if there's a good reason for this @jen-huang @ruflin. I think the best behavior would be to always put everything in the @mappings component templates so that every stream is configured the same way.

[nchaulet]

@hop-dev I am wondering if you tested to use the *@custom component template to specify the ILM policy this way you would not have to duplicate the index template

Edit: I just saw the whole discussion here elastic/observability-docs#986

[jen-huang]

Some streams do have a @mappings template that only contains { "dynamic" : false } while others do not have one at all. Curious if there's a good reason for this @jen-huang @ruflin.

@mappings was added with the intention to support package authors defining some some custom mapping settings on their data streams:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

Line 225 in 8c8c62e

if (registryElasticsearch && registryElasticsearch['index_template.mappings']) {

APM is an example of a package that has various values for dynamic in their data streams: https://github.com/elastic/package-storage/tree/production/packages/apm/0.4.0/data_stream

This means that if a user followed the guide, then the new index template they create would not receive mapping updates when integrations are updated.

I think the best behavior would be to always put everything in the @mappings component templates so that every stream is configured the same way.

Yeah, when @custom was originally added, we weren't thinking that the user could have to copy the index template for each namespace (the solution we ended up going with for the docs). I agree that all the base mappings and other settings should be stored away in a component template so that they can be reused in custom index templates. We should revisit what is stored on each index template and move as much as possible to component templates:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

Line 388 in 8c8c62e

const template = getTemplate({