Improve usability of remote cross cluster search in Kibana

[maihde]

Describe the feature:

Currently remote clusters can be included through index patterns, but this causes various usability issues, especially when multiple remote clusters are configured. The biggest issue is that Index Patterns become tightly coupled to remote cluster connections, so you need multiple patterns, such as:

• Local Only: some-local-pattern
• All remotes: *:index-pattern
• Some remotes: remoteA:index-pattern, remoteB:index-pattern
• Some remotes: remoteB:index-pattern, remoteC:index-pattern

Given this complexity, users often resort to building their visualizations/dashboards against *:index-pattern. While this works, it makes queries unnecessarily slow in certain circumstances are prohibits the users from

It would be nice if there was the ability to make an optional feature during Index Pattern creation. These Index Patterns could be called "Dynamic Cross Cluster Index Pattern" and a defined by a simple pattern (i.e. no leading cluster name). Then individual local and remotes can be turned on/off from the query bar. The default of what remotes are enabled can be saved with the dashboard/visualization similar to how filters cascade.

Describe a specific use case for the feature:

Users want to build dashboards/visualizations that can selectively be used for only local data, a subset of clusters (both local/remote), or all clusters. This allows users to get better performance for situations where they know data only resides in a certain cluster. It also provides users a fallback for situations where a remote cluster is not performing well or having other connectivity issues (right now, often the query just has a timeout in those situations).

[kmartastic]

@stacey-gammon do you know how to route or evaluate this request?

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

[mbarretta]

This is still a large pain point and blocker for many of our users. Reopening.

[mattkime]

@mbarretta I think there needs to be more of a user story to this to move it forward. Its unclear to me how the proposed UI would work better than using the existing filter functionality.

[mattkime]

@sixstringcode Curious if you have any ideas

[maihde]

@mattkime as the original author of this request, I'd be happy to help discuss our needs with you @mbarretta can share my contact information with you.

[maihde]

@mattkime we did some investigations, thanks for the concept of trying filters on _index. This is certainly a good quick solution, but it leaves a few things to be desired:

1. The query is still sent to remote clusters. This can impact performance in a variety of ways in both normal and abnormal situations. It's important to us to be able to disable queries hitting the remote cluster entirely.

2. It's not easily discoverable by users when creating the filter, because the autocomplete doesn't fill in. It's somewhat painful to use Discover or visualizations to learn what remote cluster are available and can be filtered.

We aren't tied to any particular UI design, so if you have alternate ideas that satisfy the same goals that would work for us.

[mbarretta]

my super refined take at a remote-cluster aware index selector in Discover

Or like this:

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mbarretta]

EAH talk brought this top of mind! There's room to improve the UX of CCS in kibana

[teresaalvarezsoler]

@mbarretta quick thought, can’t the user achieve the same by having a field in their dataview populating the cluster name for each index and then using the filters in Dashboard and Discover?

[mbarretta]

@teresaalvarezsoler

@mbarretta quick thought, can’t the user achieve the same by having a field in their dataview populating the cluster name for each index and then using the filters in Dashboard and Discover?

I think @maihde addressed the filter approach in an earlier comment

[mattkime]

@teresaalvarezsoler @mbarretta I'd like to see this resolved -

The query is still sent to remote clusters. This can impact performance in a variety of ways in both normal and abnormal situations. It's important to us to be able to disable queries hitting the remote cluster entirely.

What can we do to get this prioritized on the ES side of things?

[teresaalvarezsoler]

@mattkime we are working on a proposal (together with ES) and we will send it soon.

[kertal]

@teresaalvarezsoler I guess this PR elastic/elasticsearch#97865 is part of this initiative. But to support this in Kibana, we would need UI to allow to fine tune which clusters would be excluded in Discover, right

[teresaalvarezsoler]

hey @kertal that's correct, @andreadelrio and @MichaelMarcialis are working on the designs.

[ninoslavmiskovic]

@teresaalvarezsoler Sound good that we are working on some designs. Could you loop me in on it thanks 🙏

[teresaalvarezsoler]

These are the last designs proposed for this issue (the CCS icon will appear only if there is a CCS data view in dashboards, discover, lens, solutions, etc.). Please let us know any comments

[BenShoeSpectric]

The main comment from our user base is that the pip on the new button may not be as visible to the users without significant training. Our suggestion would be that the whole button change colors based on the status of the remote clusters instead of just a pip.