Be able to configure index settings / ILM with CRD

[mtparet]

Proposal

Be able to configure ILM (and generally index settings) through a CRD.

Why is it so important?

This makes possible to natively industrialize (reducing a lot the cost of creating new clusters) creating ElasticSearch cluster ready for typical use cases with the right configuration.

Also it enables gitops for what is possible to do with these new CRDs.

[andreykaipov]

I actually spoke with @sebgl during Kubecon about the possibility of this feature. It's crazy you had the same thought around the same time! Our use case would definitely fall under the ability to do GitOps, e.g. maintaining a set of ILM CRDs that our pipelines can deploy to K8s, resolved by this operator.

[sebgl]

A few thoughts:

• We should investigate whether this should be a stack (Elasticsearch) feature instead of an ECK feature.
• In general we want to avoid having multiple ways of doing things: ILM is a good example of a feature already available through the Elasticsearch API that we may not want to reimplement.
• ECK would probably reconcile/apply the ILM settings forever. If the user then patches the settings through Elasticsearch API but ECK reverts the user changes to set its own version of ILM policies, things can become confusing.
• The need for configuration-as-code/GitOps probably extends to a larger scope and include eg. the automatic creation of indices with a predefined mapping.

[pebrc]

Related #2155

[easyCZ]

What's the state of this feature? Are you accepting PRs for it?

Improbable has a use-case to use the operator to create and bootstrap a cluster, including users, roles, ILMs, aliases, index-tempaltes and index-patterns for Kibana. I'd be happy to contribute this.

[agup006]

Hey folks, we are still taking a look into this

[weeco]

Adding this feature in the upstream Elasticsearch has been rejected. How can we proceed now?

[mtparet]

They did it https://github.com/Carrefour-Group/elastic-phenix-operator

[knechtionscoding]

@agup006 is this a feature that is on the roadmap? Being able to define the configuration of ES, Kibana, etc using built in k8s resources would be a super useful feature. Right now we have to utilize cronjobs or custom code to generate and ensure the continued existence.

[mtparet]

Elastic made an official blog post to announce elastic stack as code using terraform https://www.elastic.co/blog/streamline-configuration-processes-with-official-elastic-stack-terraform-provider.

This is exactly the same things that using k8s CRDs but with Terraform. This should help the implementation as the problematic (elasticsearch side) is the same and seems already resolved.

[xco-sk]

@mtparet we had the same problem with missing CRDs for Elasticsearch and Kibana resources that made the ECK usage a bit unpleasant because we need to define dozens of indices/templates/ILMs/SLMs as well as plenty of Kibana resources. So I made a k8s operator that manages these resources (https://github.com/xco-sk/eck-custom-resources). It's far from perfect but if you give it a try, I'll be super glad for any feedback or functionality requests.

[ghost]

@mtparet unfortunately Terraform solution requires direct connectivity between Terraform and ElasticSearch cluster which is not possible in some configurations. So ability to use a CRD still would be really nice.

[thbkrkr]

unfortunately Terraform solution requires direct connectivity between Terraform and Elasticsearch cluster which is not possible in some configurations.

Could you say more about that? What if you were running Terraform in a pod in the same k8s cluster where Elasticsearch is deployed?

[ghost]

@thbkrkr

1. Unfortunately we are running TF pods in a separate kube cluster and even in a separate project (GCP), TF runners have no connectivity to any of our databases including ElasticSearch. Everything is being configured either via GCP APIs or by creating kube resources. And I think it is always better to have less direct connections between configuration tools and managed services/databases/etc.
2. Another issue with the ES provider approach - we don't know when our new ECK cluster is actually ready - we don't have an easy way to implement a wait condition between an initial ECK CRDs creation and applying ILM/SLM via the ES provider. And initial ECK setup can take signifcant time - in some cases it enouth for TF to fail by a timeout, so we need to rerun TF multiple times to finally apply it.
3. Our kube clusters are also being created via Terraform. So I am not sure how can we have a TF runner in kube cluster which itself should be created by Terraform. By adding dedicated TF runners to our kube clusters after creating them just to manage ES? Doens't look very practical and again only introduces more complexity and more effort.

So for now we just ended up with deploying a set of kube cron jobs to the same kube cluster when we run ECK. Those cron jobs periodically execute a script which attempts to connect to ECK clusters and create ILM/SLM in them. So essentially a poorman's operator approach. This approach took just few days to implement, but reconfiguring our infra to grant TF direct connectivity to ES can take much much more time and effort. Instead we most likely will end up with something like #2157 (comment)

And in my opinion it doesn't look very logical to configure half of an ES cluster by creating CRDs (for the operator itself, ES and Kibana) and another half using a completely different approach - via the Elasticsearch provider with direct connectivity to that ES cluster (ILM, SLM). It would be much more logical to just create a set of CRDs.

[Kushmaro]

Closing this one, as this feature is now available through "StackConfigPolicy" CRD in the ECK operator.

[an-tex]

bummer this has become an enterprise feature

[zfrhv]

make sure your CRD isn't missing:
#6320