Get Aliases API fails when a system data stream exists and security is enabled

[gwbrown]

Elasticsearch version (bin/elasticsearch --version): 7.13.0, probably all later versions

Plugins installed: []

JVM version (java -version): built-in

OS version (uname -a if on a Unix-like system): MacOS

Description of the problem including expected versus actual behavior:
When a system data stream exists and Security is enabled, the Get Alias API will always fail.

Steps to reproduce:

1. Start an Elasticsearch cluster with security enabled. Perform all following steps as superuser.
2. Create a system data stream:

curl -X POST -u elastic-admin:elastic-password "localhost:9200/.fleet-actions-results/_doc" -H 'Content-Type: application/json' -H 'X-elastic-product-origin: fleet' -d'{"@timestamp": 0}' | jq

3. Attempt to use the Get Aliases API, with or without any index name, e.g. GET _alias or GET _alias/auditbeat-7.13.0.
4. Observe the error response.

Note that the Get Alias API behaves as expected when security is not enabled.

Provide logs (if relevant):
The error response:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Data stream(s) [.fleet-actions-results] use and access is reserved for system operations"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Data stream(s) [.fleet-actions-results] use and access is reserved for system operations"
  },
  "status": 400
}

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[gwbrown]

This is the cause of elastic/kibana#100019.

[williamrandolph]

It looks to me as if the problem is in TransportGetAliasesAction#masterOperation:

        try (ThreadContext.StoredContext ignore = threadPool.getThreadContext().newStoredContext(false)) {
            concreteIndices = indexNameExpressionResolver.concreteIndexNames(state, request);
        }

We intend for the invocation of newStoredContext(false) to remove the header that denies access to system indices. But it doesn't look like newStoredContext(false) actually does this; it just ensures that the stored context is restored when we exit the try-with-resources block.

The method ThreadContext#stashContext() does remove the existing headers for the code within the try-with-resources block. I have done some initial testing with this change, but I need to discuss it with other devs before committing to it.

[ghost]

Hi @ruflin,

As per the ticket #100019 , We have validated this ticket on 7.13.0 BC7 build and Please find the below observations:

Build Details:

Version:7.13.0 BC7 On-Prem
Commit:6ce6847436ff9bef0ad91268b6585e0f9339c9fd
Build:40864

Observations:

• When we install Fleet Server Agent it is installed with Default Fleet Server Policy, and we are able to install beats if we make no change to Fleet Server Agent.
  

• However when we attempted to Assign Fleet server agent to Default policy having System, Fleet Server and endpoint integration , we are further unable to install beats on the Kibana environment and getting following errors:

SCREENSHOT

Agent Logs after changing the policy
logs.zip

Thanks!!

[EricDavisX]

@deepikakeshav-qasource can you confirm, you have re-tested this in BC8 or newer in the same scenario and that it works? Thank you!

[ghost]

Hi @eric,

We have validated this ticket on 7.13.0 BC9 build and observed that issue Fixed. Now able to install the beats When an agent is installed and policy is changed from Fleet server agent to Default policy having System, Endpoint and Fleet Server integration

Build Details:

Version:7.13.0 BC9 On-Prem
Commit:9863e88bd63ad546b9d36e6b0c0c55cb65dd9081
Build:40865

Screenshot:

Thanks!!