Cannot use copy_settings=true for shrink action

[zsandrus]

For usage questions and help

Please create a topic at https://discuss.elastic.co/c/elasticsearch

Perhaps a topic there already has an answer for you!

To submit a bug or report an issue

When using a shrink action, there is no option to pass a copy_settings=true parameter to elasticsearch. This prevents metricbeat indexes from being shrunk using curator since the index.mapping.total_fields.limit setting can not be copied over. The new index fails to start due to the number of fields being higher than the default limit of 1000.

Expected Behavior

The shrink API copy_settings parameter should be an option to set.

Actual Behavior

The new index fails to start due to the number of fields being higher than the default limit of 1000.

Steps to Reproduce the Problem

Create a metricbeat index with >2 shards.
Use curator to shrink index.

Specifications

• Version: Elasticsearch 6.5.0, Curator 5.6.0
• Platform: Official Docker containers

Proposed Change

The copy_settings parameter will default to true starting in ES 7, but still needs to be set on ES 6.4+. Add a new option to set copy_settings to true and match default behavior and deprecation of copy_settings parameter of elasticsearch: elastic/elasticsearch#28347

[madisonb]

Is there a solution for folks using an ES version less than 6.4? Right now running on a cluster version 6.3 the copy_settings parameter is inconsistent and the curator.Shrink() operation sometimes needs multiple tries on a single index in order to get copied and shrank correctly.

In some cases:

• The index is created but fails to allocate (due to the field count being higher than 1000) - this causes the cluster to go red
• The operation fails during the DELETE phase, being called with an invalid parameter - this causes duplicate data in your cluster until you manually delete the original index
• Other situations where the aliases is not copied over and the operation fails

If you clean up the cluster and get it back to green, and try the Shrink() enough times, the operation will succeed.

[kmickeletto]

Can anyone shed any light on how to use the copy_settings: True setting? I am running elasticsearch-curator-5.7.6-1.x86_64. Where should it be placed? Documentation on this option seems limited.

actions:
1:
action: shrink
description: >-
Shrink selected indices on the node with the most available space.
options:
timeout_override: 21600
ignore_empty_list: True
disable_action: False
shrink_node: DETERMINISTIC
node_filters:
permit_masters: False
exclude_nodes: [node1,node2]
number_of_shards: 1
number_of_replicas: 1
shrink_prefix:
shrink_suffix: -shrunk
delete_after: False
post_allocation:
allocation_type: include
key: node_tag
value: warm
wait_for_active_shards: 1
extra_settings:
settings:
index.mapping.total_fields.limit: 5000
index.codec: best_compression
refresh_interval: 600s
wait_for_completion: True
wait_for_rebalance: True
wait_interval: 30
max_wait: -1
filters:
- filtertype: pattern
kind: prefix
value: index-name-
- filtertype: pattern
kind: suffix
value: -shrunk
exclude: True
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 1

[untergeek]

The setting has never been added, so, you can't use it yet.