Unknown setting [discovery.seed_hosts] elasticsearch 6.x

[anebi]

Chart version: 7.11

Kubernetes version: 1.13.5

Kubernetes provider: Rancher 2.2.3 / Hetzner Cloud

Describe the bug: I try to install Elasticsearch 6.6.0 and 6.8.0 versions using official helm chart on Rancher 2.2.3. When i run the cluster install and deployment, the elastic instances fail to start returning following exception:

Steps to reproduce:

1. Install elasticsearch 6.6.0 or 6.8.0
2. Monitoring elasticsearch instances nodes, you will notice that cluster fails to start

Expected behavior: Elastic to start successfully

Provide logs and/or server output (if relevant):

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [discovery.seed_hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [discovery.seed_hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Any additional context:

[oleh-hudyma]

The same for 6.8.1. Any updates on it?

[oleh-hudyma]

@anebi, there is workaround. Simply add this to helm command:
--set imageTag=6.8.1,esMajorVersion=6
Complete command I used:
helm upgrade --install elasticsearch elastic/elasticsearch --version 7.2.0 --set imageTag=6.8.1,esMajorVersion=6 --namespace="infrastructure" --wait --debug --atomic

Just simplify it as you need :)

The root cause of this bug is that there is hardcoded ES major version:
https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
esMajorVersion: 7
This value is used to decide which env variable should be added:
https://github.com/elastic/helm-charts/blob/master/elasticsearch/templates/statefulset.yaml

          - name: cluster.initial_master_nodes
            value: "{{ template "endpoints" .Values }}"
          {{- else }}
          - name: discovery.zen.minimum_master_nodes
            value: "{{ .Values.minimumMasterNodes }}"
          {{- end }}
          {{- end }}
          {{- if lt (int .Values.esMajorVersion) 7 }}
          - name: discovery.zen.ping.unicast.hosts
            value: "{{ template "masterService" . }}-headless"
          {{- else }}

That means, even when you build with ES version 6.x.x. major version will be 7.
It causes to adding ES 7 properties to StatefulSet env variables.
Those ES 7 properties later cause ES 6 to fail during startup.

This is definitely bug.
There is 2 options:

1. Move this property to _helpers.tpl and calculate major version basing on Chart version. Also this requires packaging chart with version 6.8.1
2. Move this property to _helpers.tpl and calculate major version basing on imageTag value.

The second one is preferable.

[oleh-hudyma]

@Crazybus, do you any need help with this?

[anebi]

Thank you very much. All clear. I can do my setup now :)

[jordansissel]

@OlegHudyma I do agree it results in a confusing user experience, though I don't exactly know how to improve it.

We need a way to tell Helm what version of Elasticsearch we are using, but docker tags are probably not a completely reliable way to describe this (you could use a custom docker image that has a tag like 'latest').

The chart uses the esMajorVersion value to determine this, and it defaults to 7 (just as the imageTag defaults to 7.2.0 right now). If you change the major version of Elasticsearch (say, 6.8.0), then you would also need to change the esMajorVersion value.

calculate major version basing on Chart version

The chart version is versioned with the stack (7.2.0, etc) but has relaxed version support (ie; it is expected that you can use the 7.2.0 chart to deploy 6.8.x elasticsearch). Likewise, the 7.2.0 chart can probably also successfully deploy Elasticsearch master (8.0.0-SNAPSHOT, for example).

calculate major version basing on imageTag value

This feels the most intuitive, but as you can specify your own image and imageTag, and that imageTag may have a value that does not refer at all to Elasticsearch major version, I don't know if this is the right solution.

[jordansissel]

calculate major version basing on imageTag value

Maybe for the common case, we could have a default esMajorVersion computed if, and only if, the image value is the default (image: "docker.elastic.co/elasticsearch/elasticsearch"). If it's the default image location, then we can probably rely on the first number of the imageTag representing the elasticsearch major version.

@Crazybus thoughts?

[oleh-hudyma]

@jordansissel, as you said, it would be good to use imageTag for calculating esMajorVersion.
And if the imageTag is latest, we just set esMajorVersion value to 7.
Imagine we have 3 charts versions: 6, 7, 8

• 6 is comptible only with ES 6. If imageTag is latest, then esMajorVersion will be 6.
• 7 is comptible both with ES 6 and ES 7. If imageTag is latest, then esMajorVersion will be 7, if 6.x.x then value will be 6.
• 8(future releases) is compatible with ES 6,7,8. If tag lates used, esMajorVersion will be 8.

[Crazybus]

In the readme we have section on compatability which specifies the currently tested versions and links to the automated test examples which includes a test for 6.x. It may not be 100% intuitive but I certainly wouldn't call it a bug or hardcoded since this is fully configurable and documented.

We need a way to tell Helm what version of Elasticsearch we are using, but docker tags are probably not a completely reliable way to describe this (you could use a custom docker image that has a tag like 'latest').

Yup, this is exactly why we created a variable for it instead of relying on the docker tag to be matching for custom images.

Maybe for the common case, we could have a default esMajorVersion computed if, and only if, the image value is the default (image: "docker.elastic.co/elasticsearch/elasticsearch"). If it's the default image location, then we can probably rely on the first number of the imageTag representing the elasticsearch major version.
@Crazybus thoughts?

I really really like this idea. The more I think about it, I think it would be ok to try always try and detect the version from the imageTag as long as the chart still respects esMajorVersion if it is set. Since if this detection does fail the worst thing that will happen is that the cluster doesn't form properly (which is what happens now for someone who tries to deploy version 6). If a user deploys a 7.x cluster with an image tag starting with 6. it will fail to start up, they will read the docs and find how to set esMajorVersion. So I think it makes sense to cater to the most common scenario which is a 6.x user deploying an imageTag starting with 6.. The logic would look like something like:

• Major version is set to esMajorVersion (to keep backwards compatibility)
• If not set then Major version is set to imageTag | split(.)[0] if it is a proper integer
• If an integer can't be parsed from the imageTag then it falls back to the current majorVersion which would be 7 right now.

Likewise, the 7.2.0 chart can probably also successfully deploy Elasticsearch master (8.0.0-SNAPSHOT, for example).

As long as the service discovery doesn't change in a future release (the only version specific logic that we have) then all future releases will continue to work. The deprecated options for 8.x were changed in #100 so it does already work with 8.x . f we had docker images for all Elasticsearch versions then it would also technically work for all those previous versions too.

[Crazybus]

I just opened #206 to implement the auto detection. I ended up going back on my idea of potentially doing this for all images since if someone were to be using a custom image with version 1.0.0 it would be a breaking change.