[Metrics UI] Investigate metrics explorer group by field selection logic

[jasonrhodes]

In an SDH ticket, a user was surprised to learn that the fields they were usually grouping by were no longer available in the metrics explorer group by dropdown as of 7.5 -- because we now base which fields are available in that dropdown on the currently selected metrics. Related ticket: #43322

Overall I think this is the right approach, but I'd like us to make sure this is working the way we expect. A couple questions:

• Is there a way for us to show all fields in group by if we haven't narrowed down any from the currently selected metrics, to be more backwards-compatible, at least until 8.0?
• At what level do we expect the narrowing to happen for the group by selection? For examplee, if I choose aerospike.namespace.client.delete.error as the metric to look at, the groupby shows:

which looks like we're narrowing on `aerospike.namespace` at that point, rather than all `aerospike` values. I wasn't able to easily trace the logic on that decision in the code, so I wanted to see what our intention was there?

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[simianhacker]

One issue with using the first part of the prefix, system in system.cpu.user.pct, is that you will end up with a lot of fields that are unrelated to the specific metric you're trying to explore. In the system example you would end up with things like system.network.interface or system.filesystem.mount_point. The biggest issue we face right now is that there isn't a programmatic way to say "Field X belongs to module X and these are ALL the related string fields".

We are resigned to try and follow a combination of convention and a know list of associated fields with popular modules. In the past, we tried to sample 5 documents using an exist filter for the metric field; this approach fails for modules which dynamic field names like Prometheus or Kubernetes.

[antcodd]

I think the logic of only showing ECS fields and fields that match a prefix of the selected metric is misguided. It is common to use fields from a different prefix in a document to correlate fields similar to ECS but not using ECS fields.

For example in the Kafka metricbeat I can't use Metrics Explorer to graph kafka.consumergroup.consumer_lag graph per kafka.partition.id.

At minimum it should be possible to type a full field name and have it actually work, especially given issues like #38073 (not sure if that also applies to the Metrics list).

In the past, we tried to sample 5 documents using an exist filter for the metric field

Why not use the same logic as the existing field dropdowns used elsewhere in the UI such as in Controls (though see #38073), or the index pattern? The inconsistency in behaviour across Kibana in behaviour of tools is getting quite extreme.

On another note the Inventory tab is extremely limited and domain specific as to what it can graph by, even within ECS.

[simianhacker]

The "correct" fix for this is to stop shipping fields in the Metricbeat mappings for modules that are not installed. There is effort underway to fix this with the integrations project.

As for the Metrics UI, we are back to the drawing board. The fix deployed in 7.5 was rolled back with #58553 due to performance issues.

[jasonrhodes]

So just to clarify, with #63398, we no longer filter any fields on group by or the metrics selection, is that right @simianhacker ?

If those queries become less expensive again in the future, I'd like to explore re-introducing the feature as a "recommended fields" feature rather than an absolute field filter. I'll create a ticket for that and close this once we confirm the above is true. Thanks!