[metricbeat][libbeat] template setup fails with number_of_shards=4

[Feder1co5oave]

• Version: 6.5.0
• Operating System: Ubuntu 18.04
• Steps to Reproduce: edit metricbeat.yml:

setup.template:
  enabled: true
  name: metricbeat
  pattern: metrics-*
  settings:
    index.number_of_shards: 4

The following mapping is produced by metricbeat: (metricbeat export template)

{
  "index_patterns": [
    "metrics-*"
  ],
  "mappings": {
    "doc": {
      "_meta": {
        "version": "6.5.0"
      },
...
"settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "number_of_routing_shards": 30,
      "number_of_shards": 4,
      "refresh_interval": "5s"
    }
  }
}

When the above template is uploaded to elasticsearch, like this

metricbeat export template | curl -XPUT "http://localhost:9200/_template/fail?pretty" -H "Content-Type: application/json" -d @-

the following is returned

{
  "error" : {
    "root_cause" : [
      {
        "type" : "remote_transport_exception",
        "reason" : "[elk3][10.135.125.207:9300][indices:admin/template/put]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "the number of source shards [4] must be a must be a factor of [30]"
  },
  "status" : 400
}

When running metricbeat setup --template -e, the error is lost in the huge output.

This bug was introduced in #5570. Choosing a hard-coded default value of 30 just because it fits the largest number of possible number_of_shards values sounds like a lazy solution to me. It doesn't fit 4.
The offending line is this:

beats/libbeat/template/template.go

Lines 207 to 211 in df74c26

	// number_of_routing shards is only supported for ES version >= 6.1
	version61, _ := common.NewVersion("6.1.0")
	if !t.esVersion.LessThan(version61) {
	indexSettings.Put("number_of_routing_shards", defaultNumberOfRoutingShards)
	}

Proposed fix: why not set number_of_routing_shards to, say, six times the number_of_shards when it is set in the configuration file, or 30 otherwise?

Addendum:
I can't find anything about this number_of_routing_shards in the most recent documentation w.r.t. metricbeat configuration: https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-template.html
even though it seemed clear from the linked PR that the option should have been documented.

Editing the metricbeat configuration like this would have fixed the issue.

setup.template:
  enabled: true
  name: metricbeat-fail
  pattern: "metrics-*"
  settings:
    index.number_of_shards: 4
    index.number_of_routing_shards: 24

[ruflin]

@Feder1co5oave Thanks for bringing this up. I think the first work around we should do is document this better. As you pointed out it is pretty simple to fix it in the config itself. The intention of hardcoding a value was to give the user by default a potential option to split the shards. Instead of guessing a number_of_routing shards when someone modifies the number_of_shards I would rather suggest that we error out and require the user to change it.

[Feder1co5oave]

@ruflin thanks for getting back to me. I agree, had the option been documented, I wouldn't have incurred in this issue. Also, I believe not many people are familiar with the number_of_routing_shards parameter in elasticsearch, so they don't think about it when changing number_of_shards in metricbeat.yml. I strongly suggest pointing out that if the latter is changed, the former should be adjusted accordingly to a multiple of it.

[maziyarpanahi]

Version: 7.0.1
Same error without any explanation.

[TomaszKlosinski]

The same issue in filebeat 7.5.2.

[kaiyan-sheng]

Seems like documenting this behavior is needed. @elastic/beats WDYT?

[urso]

Added docs and bug label. Elasticsearch 7.0 did improve shard splitting support, making the setting optional. We should remove number_of_routing_shards from template generation in Beats 7.x.

For reference:

• Elasticsearch 6.5 shard splitting docs: https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-split-index.html
• Elasticsearch 7.0 shard splitting docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/indices-split-index.html

Some users still might want to set the setting, but we should not include it by default in 7.x anymore. From docs side we should include a small section about shard splitting and link to the Elasticsearch docs.

[urso]

The setting is going to be removed from the default template via #16038. @dedemorton I don't think we need to adapt the 7.x docs, as the setting is a workaround for Elasticsearch 6.x.

[vipinvkmenon]

same issue apm server 7.3.2

[urso]

The fix will be available with the 7.7.0 release. All Beats and apm-server do not set number_of_routing_shards anymore by default if Elasticsearch 7.x is used.