Removing system index access deprecation logs from Upgrade Assistant in 7.16

[cjcenizal]

The problem

The 7.16 Upgrade Assistant is the official tool for users to prepare their deployments for upgrade to 8.0. It currently has a "deprecation logging checkpoint" feature, which tracks the timestamp of the latest checkpoint and only shows the user new logs that are triggered after that timestamp.

Here's what it looks like when the user sets the checkpoint and no new logs have shown up. If the user waits long enough and this state doesn't change, the user can consider all of their applications to be compatible with 8.0.

If new logs arrive, this state changes:

At this point, the user will investigate the logs, read the messages, trace them back to the offending applications, fix and redeploy those apps, and then set a new checkpoint. One way to investigate the logs is through the Logs UI:

The highlighted log is emitted when a system index is accessed. This is problematic because there's no way for a user to fix this. It will continue to be emitted whenever a system index is accessed, either by one of their applications or by Kibana itself. In this case, it happens to be Kibana but a user-owned app could be doing this too.

Not only is this confusing for users, it also detracts from the usefulness of this feature in Upgrade Assistant and deprecation logs themselves. Once a user has figured out there's nothing to do with this, they need to work to actively ignore them and focus on the ones that are meaningful. The yellow state in the second screenshot above could be a false positive -- you just have to look at the logs and check.

Solutions

'x-elastic-product-origin': 'kibana' header

This header was introduced to allow Elasticsearch to identify requests that originated from Kibana. More on this topic in elastic/kibana#81536. I need confirmation from engineers, but I think this can be used to suppress logs triggered by requests from Kibana. It won't suppress logs triggered by requests from a user-owned application.

Filter out these logs in Kibana

Kibana retrieves these logs from the logs index. When it receives a response to its query, it can filter out specific log messages based on the the string "this request accesses system indices" or similar. We need to assess the scope of this work to understand what it would entail. Note that these logs will still exist in the logs index or logs directory, so users who access these logs outside of Upgrade Assistant will still see them.

Remove these logs from Elasticsearch in 7.16

We can remove these logs from Elasticsearch in 7.16. They won't appear in Upgrade Assistant, the logs index, or the logs directory. Users will never see them as they go through the upgrade process.

CC @bytebilly @williamrandolph @rjernst @gwbrown @colings86

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[cjcenizal]

CC @jakelandis

[bytebilly]

Filter out these logs in Kibana

Given that the UA links to Logs and Discovery apps, is there any way to filter out logs without using the "standard" filters that would be visible to the user?

[cjcenizal]

After chatting with @colings86, we agreed to:

• Keep the existing deprecation log regarding system indices, because we expect very few users will encounter this log and we want to strongly discourage users from accessing system indices directly
• Improve the log message with a suggestion to use UIs or APIs instead
• Improve the log message with a documentation link that will provide more context

Regarding the log message in the screenshot, this is currently triggered by Kibana itself, and we will address that by applying the x-elastic-product-origin: kibana header (tracked by elastic/kibana#116327).