[BUG] Index pattern missing after recreating several times

[psciacca2inno]

I need to use an Index pattern and I created an Index pattern cwl-*

Then I leave the opensearch Dashboard.

When I go back again to Stack Management my index pattern is missing,

[kavilla]

Hello @psciacca2inno ,

Thanks for opening this! Could you please provide more details? How are you running the application? OpenSearch and OpenSearch Dashboards version? Which plugins do you have installed?

Thank you!

[psciacca2inno]

Hi, Suddenly the missing pattern issue appeared again. I'm using the dashboard of opensearch directly in AWS. Opensearch version is:1.0 Opensearch Dashboard version is: v 1.0.0-SNAPSHOT I worked with these logs several days this week. But now I have again problems: [image: image.png] Thanks in advance.

…

On Wed, Feb 23, 2022 at 12:58 AM Kawika Avilla ***@***.***> wrote: Hello @psciacca2inno <https://github.com/psciacca2inno> , Thanks for opening this! Could you please provide more details? How are you running the application? OpenSearch and OpenSearch Dashboards version? Which plugins do you have installed? Thank you! — Reply to this email directly, view it on GitHub <#1270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWWPZM24UC7LLH6HZMB62QLU4RLORANCNFSM5O6ZSZQA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

-- Si usted ha recibido este e-mail por error, comuníquelo inmediatamente por esta vía y elimínelo de su sistema; cualquier revelación, copia o distribución de su contenido está estrictamente prohibido. El mensaje expresa la opinión de su autor pero no necesariamente expresa la opinión oficial de 2Innovate IT. *The information included on this message is intended only for the personal and confidential use of the designated recipient(s) named above.* *If you have received it by mistake please let us know by e-mail immediately and destroy or delete it from your files or system; any disclose, copy or distribution of its contents is prohibited.* *This message reflects the opinion of its author but not necessarily express an official statement of 2Innovate IT.*

[kavilla]

Hello @psciacca2inno,

Within AWS, OpenSearch Dashboards runs with a number of OpenSearch Project plugins, including the Security plugin. My initial reaction is that perhaps the index pattern is being created for one tenant when you are in another tenant.

To explain, when creating an index pattern the data is saved within OpenSearch Dashboards' system index, for reasons it is .kibana. Within the security plugin, .kibana is the index that the global tenant uses. However, the plugin will create another system index .kibana_xyz for another tenant and will save data to this index when using that specific tenant. This can cause some confusion when saving data like index patterns because if you are on .kibana and create an index pattern it will not exist in .kibana_xyz, and vice versa. [DOCS]

Another thing to check is if the user you have logged into has the correct roles-based permissions. If they do not, it will filter out system indices. [DOCS]

If you knew all this and checked all of this, my apologies for assuming. If checked and verified all the system indices and your index-pattern does not exist still, please close this issue and contact your cloud support provider. They should to provide more specific insight on your clusters.

If you checked and find the index-patterns you were looking I'd suggest we route this to doc repo to provide a deeper explanation on multi-tenancy within OpenSearch Dashboards security plugin.

Thank you!

[boktorbb]

Will be closing out this issue due to no activity for over a month. You can reopen the issue if you have anything more questions or traction.

[begna112]

I have also been experiencing this issue for over a year now. The issue almost always presents when a new update becomes available for Elastic/OpenSearch and then only applying the update will sometimes resolve it.

vivaldi_IwYDxLyEF8.mp4

[DBendit]

I'm seeing the exact same behavior as @begna112 now. Seeing as there's no pending update in my domain, I don't have a clear way to resolve this issue.

Edit: I found a workaround! By adding a newline to my access policy, I was able to trigger a blue/green deploy. When the system cut over to the new instance, everything was working again.

[leela782645]

I have seen a similar behaviour. I found out that the problem is temporary resolvable by restarting one (of three) Opensearch Dashboards instances. In the logs there are appearing some migration tasks on startup for "my-new-tenant". After that the index pattern is visible.

Jul 27 09:54:46 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:46Z","tags":["info","savedobjects-service"],"pid":2927,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
Jul 27 09:54:46 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:46Z","tags":["info","savedobjects-service"],"pid":2927,"message":"Starting saved objects migrations"}
Jul 27 09:54:46 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:46Z","tags":["info","plugins-system"],"pid":2927,"message":"Starting [44] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,reportsDashboards,securityDashboards,indexManagementDashboards,anomalyDetectionDashboards,queryWorkbenchDashboards,embeddable,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]"}
Jul 27 09:54:46 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:46Z","tags":["listening","info"],"pid":2927,"message":"Server running at http://10.10.10.34:5601"}
Jul 27 09:54:46 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:46Z","tags":["info","http","server","OpenSearchDashboards"],"pid":2927,"message":"http server running at http://10.10.10.34:5601"}
Jul 27 09:54:47 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:47Z","tags":["info","plugins","securityDashboards"],"pid":2927,"message":"Detected mapping change in \"dynamic\""}
Jul 27 09:54:47 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:47Z","tags":["info","plugins","securityDashboards"],"pid":2927,"message":"Creating index .kibana_129215323_mynewtenant_2."}
Jul 27 09:54:48 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:48Z","tags":["info","plugins","securityDashboards"],"pid":2927,"message":"Migrating .kibana_129215323_mynewtenant_1 saved objects to .kibana_129215323_mynewtenant_2"}
Jul 27 09:54:48 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:48Z","tags":["info","plugins","securityDashboards"],"pid":2927,"message":"Pointing alias .kibana_129215323_mynewtenant to .kibana_129215323_mynewtenant_2."}
Jul 27 09:54:48 master1 opensearch-dashboards[2927]: {"type":"log","@timestamp":"2022-07-27T07:54:48Z","tags":["info","plugins","securityDashboards"],"pid":2927,"message":"Finished in 1925ms."}

How can I solve this without a restart?

[joshuarrrr]

While this is tagged for docs, it seems like there still may be some unintended behavior here.

[bandinib-amzn]

Hi,

Adding to @kavilla's above comment, we don't delete the index pattern behind the scene. Other reason I can think of is Cluster can be configured in such way that it deletes indices after several days. Another reason can be .kibana index is corrupted. In case if you see problem again, do you mid posting the output of GET .kibana/_settings?pretty? It will help to investigate further.

[DBendit]

When I ran into this, I deleted the .kibana index, wondering if it was corrupted. The new index that it created didn't alleviate the issue.

[nmaludy]

Same problem here. I logged into Kibana and my index pattern and dashboards are all missing.

Makes me worried that if i make changes / add things in Kibana that i will one day lose them again?

[ashwin-pc]

@DBendit @nmaludy can you run GET .kibana/_settings?pretty in the dev tools and paste the response here as @bandinib-amzn suggested? Also do you have the security plugin installed and/or have user and global tenants setup?

[DBendit]

{
  ".kibana" : {
    "settings" : {
      "index" : {
        "creation_date" : "1670777582648",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "oBYwGj53RxCbYPO0sGib8A",
        "version" : {
          "created" : "135248027"
        },
        "provided_name" : ".kibana"
      }
    }
  }
}

[DBendit]

Also, turns out that whitespace changes to the access policy no longer trigger deploys, so that workaround is no longer valid.

[DBendit]

Per AWS Support, the issue is that the Kibana alias is missing for the domain.

[sharvil-parekh]

Is there any solution yet to this issue? I am still facing it with opensearch v2.0.1

[Julien-athstat]

I also deleted all .kibana* indices as I reached the shard limits on my dev environment. Felt ridiculous to have 300x 100KB indices that limits my 10GB indices of valued data, only to discover that these indices contains all the Saved Objects of the tenant!
Now I am also encountering the issue that I can't create anything on the Global tenant. I also can create a new index pattern, but when back at the overview, it doesn't appear. I also tried to import saved objects, and I believe that is where the main issue is, there is an error showing up: Unable find saved objects [Error]
This seems to have been removed, as not even the Advanced Settings file is appearing.

Important Note: the issue is only true for the Global tenant, and not affecting the Private tenant or any of the tenants I had created before I wiped out their content.

[AMoo-Miki]

The only time I have seen this exact symptom of index pattern vanishing after being created is when there is _template that covers *. Can you post the output of GET _cat/templates please?

This could be related to opensearch-project/security-dashboards-plugin#1222

[Julien-athstat]

@AMoo-Miki doesn't seems to be the case on my hand:
logstash [logstash-*] 0 60001
tenant_template [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*] 0

[AMoo-Miki]

If you ever delete .kibana-* indices, make sure .kibana* aliases are deleted and restart OpenSearch Dashboards; the indices and aliases get recreated.

[AMoo-Miki]

@Julien-athstat, based on your previous comment, i think the issue you are facing is because .kibana-* indices are missing. Removing the alias(es) and restarting Dashboards should recreate them.

Also, I agree with you that it feels useless that any of the older .kibana-* indices are left around, hogging resources (and limits). Dashboards looks at those existing ones during startup to figure out if it needs to update; I will have to dig into the code to see if the entire sequence is required to exist or if it is sufficient to keep only the latest one (or two).

[abbyhu2000]

Close this issue, and @AMoo-Miki will create a new issue on researching on the reason we are keeping all historical data of Kibana.

[AMoo-Miki]

@Julien-athstat I have created #3236 to research to retaining issue.

@psciacca2inno, we haven't heard from you in a long time; please re-open if you need more help.

[DBendit]

This issue is still occurring - we just lost access to our OpenSearch instance again.

[joshuarrrr]

@DBendit Can you provide more specifics, along with any of the mitigations listed above you tried?

[DBendit]

It's the issue as described initially: going to our Kibana URL, then trying to access "discover", I'm prompted to create a new index pattern. Creating a new index pattern seems to succeed, but then it is unlisted and the behavior trying to use "discover" is unchanged.

I've got a ticket open to AWS support right now, so I haven't done any attempt at mitigation to provide them with a pristine case to look at.

[DBendit]

GET .kibana/_settings?pretty

{
  ".kibana" : {
    "settings" : {
      "index" : {
        "creation_date" : "1674147182642",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "k6sHUCVqREi3SLsP9csBJQ",
        "version" : {
          "created" : "135248027"
        },
        "provided_name" : ".kibana"
      }
    }
  }
}

GET _cat/templates

[200, no data in response]

[UXabre]

I'm also observing this exact behaviour with no clear cause. All the logs seem to indicate all is going well, even more, refreshing the link after the index pattern has been created leads to the same page, meaning it seems it is being created in the background.
The only way to make this work is if I configure it under a global tenant. Private tenant or a specific tenant don't work.
Is there anyone where this function does work?

[blaargh]

yup same issue here.

[blaargh]

I have found the issue for me. I was using the master account in "private tenant" mode. I don't know why, but it just wouldn't save the index pattern there. When I switched to "global tenant" (press your account icon on the top right -> switch tenants), it started working.

[begna112]

I have found the issue for me. I was using the master account in "private tenant" mode. I don't know why, but it just wouldn't save the index pattern there. When I switched to "global tenant" (press your account icon on the top right -> switch tenants), it started working.

What version of OpenSearch are you using @blaargh ? I'm on 1.3 and do not have that option.

[UXabre]

I've just upgraded to 2.5.0 and unfortunately I still can't make this multitenancy work. I noticed in the communication that it tries to fetch the index patterns as a saved object (with a filter for index patterns). I'm guessing my dashboard role is configured incorrectly (but at this exact moment in time it's unclear which configuration is incorrect exactly)

This my current config:

dashboard_role:
            reserved: true
            cluster_permissions:
              - 'cluster_monitor'
              - 'cluster:monitor/nodes/info'
              - 'cluster:monitor/state'
            tenant_permissions:
              - tenant_patterns:
                - "global_tenant"
                allowed_actions:
                - "kibana_all_read"
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices:admin/get'
                  - 'indices:admin/create'
                  - 'indices:admin/refresh'
                  - 'indices:admin/refresh[s]'
                  - 'indices:admin/aliases'
                  - 'indices:admin/aliases/get'
                  - 'indices:data/read/search'
                  - 'indices:data/write/bulk'
                  - 'indices:admin/template/put'

[blaargh]

@begna112 I am actually using ElasticSearch 7.9 as backend. I was logged in with the master user.

[begna112]

AWS Support finally helped us identify this issue for us. We had deployed a lambda to complete periodic index deletion following the guidance from the recommended "Curator".

The default Curator has the following:

    # Filters by age, anything with a time stamp older than 30 days in the index name.
    index_list.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=30)

This coincidentally filters out the kibana index which does not have a timestring.

However, we had customized our Curator to the following:

    #index_list.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=30)
    index_list.filter_by_age(source='creation_date', direction='older', unit='months', unit_count=1)

The fix was to add another filter to only include our indexes, and exclude the kibana index from deletion:

    index_list.filter_by_regex(kind='prefix', value='cwl-')

I imagine a similar filter could be created to specifically exclude the kibana index as well.

Hopefully this helps someone else that's having this issue.

[AMoo-Miki]

GET .kibana/_settings?pretty

This tells me that .kibana is an index which it should not be. It appears that .kibana_<number> indices were deleted and instead .kibana index was created manually. This is almost guaranteed to not have the proper mapping. I would recommend you get rid of .kibana and make sure you have (a) no indices and (b) no aliases named .kibana, or .kibana_* and then (3) restart Dashboards.

[AMoo-Miki]

I would strongly encourage the rest of you to use the forum to get quicker help from the community.