[BUG] Unable to create index patterns in a Kubernetes environment when OpenSearch restarts

[adiforluls]

Describe the bug

I'm running a single node OpenSearch cluster with master node in a Kubernetes environment and an OpenSearch Dashboards instance in the same environment to visualise the data. OpenSearch in this environment is configured to use ephemeral storage.

Now whenever OpenSearch pod restarts, I expect the existing data stored in OpenSearch to be gone. But I'm also seeing a weird problem where I'm not able to create index patterns in OpenSearch Dashboards after OpenSearch is back up after restart. When this happens, I always see a .kibana index in yellow state with replica set as 1.

The problem goes away when I restart OpenSearch Dashboards pod, and I'm able to create index patterns via the Dashboards UI, and I also noticed that the index in this case was .kibana_1 in green state with replica set as 0.

I think this is an unintended behaviour that needs investigation, I shouldn't have to restart my Dashboards pod to fix this problem.

This problem is similar to #1270 but I see that was closed without proper investigation and analysis. I shouldn't have to restart Dashboards instances to fix my problem in the first place.

To Reproduce
Steps to reproduce the behavior:

1. Install OpenSearch in a kubernetes cluster, with one master node that uses ephemeral storage such that all data stored in OpenSearch is wiped out. Also install OpenSearch Dashboards to visualise data.
2. Restart OpenSearch pod.
3. Now try to create index patterns in the Dashboards once OpenSearch is back up. Also check the existing indices to see a .kibana index in yellow state.

Expected behavior

OpenSearch Version
v2.3.0
Dashboards Version
v2.3.0
Plugins

Please list all plugins currently enabled.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Browser and version [e.g. 22]

Additional context

Add any other context about the problem here.

[Flyingliuhub]

@adiforluls Thank you for reaching us for this. According to your describe detail behavior, I'm thinking that expected behavior, let me explain a little here.

1. Restart OpenSearch Pod, ephemeral storage should be wipe out, except for container restart, if restart a container in Opensearch pod(multiple containers in the same pod) , the contents will survive - I saw that you mentioned that only one master node in the pod, so it is not the case. Restart OpenSearch Pod, wipe everything out, includes .kibana alias and .kibana_{num} indexes, .kibana_{num} indexes are used to store Kibana configuration information, such as dashboard configurations and index patterns. By default, Dashboards reads data from .kibana_{num} indexes based on the alias .kibana.
2. Before you restart OpenSearch Pod, OpenSearch Dashboards' .kibana alias and .kibana_{num} indexes should all healthy. after you restarted OpenSearch Pod, .kibana alias gone as ephemeral storage content wipe out, so you saw that .kibana index exist and status yellow, .kibana index should not exist, .kibana_{num} indexes are the correct indexes for storing the configuration. Because the .kibana alias was wiped out, but OpenSearch Dashboards still running, so writing everything to .kibana indexes which should not exist.

If you plan to restart OpenSearch Pod often, I would like to suggest that use the persistent volume for your OpenSearch Pod, sample persistence , it will avoid this situation which wipe out (.kibana alias and .kibana_{num} indexes).

[adiforluls]

Hi @Flyingliuhub thanks for your explanation. I now get the gist of the problem, since OpenSearch Dashboards is still running it assumes .kibana alias to exist but since it was wiped out we see the odd behaviour. I'm wondering if there's a way to recover from this bad state automatically without having to restart OpenSearch Dashboards, like OpenSearch Dashboards checking of .kibana alias exists before writing data and create it if it's missing?

[adiforluls]

@Flyingliuhub Hi, do you have any thoughts on the comment above?