Response from wildcard resolution vs specific index access

[elasticmachine]

Original comment by @jaymode:

The current authorization behavior (5.1.1 RC) is a bit different when a user requests values for indices that do not exist or they are not authorized to see. For example, consider the following table:

Authorized	Specific Index (exists)	Specific Index (nonexisting)	Wildcard (exists)	Wildcard (nonexisting)
Yes	200	404	200	200
Yes (no security)	200	404	200	200
No (no indices permissions)	403	403	403	403
No (indices permissions)	403	403	200	200

The action used here was a search request and nothing with indices options was modified. The first row is from a user with superuser privileges. The third row user only has cluster privileges and the fourth row user cluster monitor indices privileges to a non-existing index.

Another example is the cat indices API, which uses a mix of cluster and index actions:

Authorized	Specific Index (exists)	Specific Index (nonexisting)	Wildcard (exists)	Wildcard (nonexisting)
Yes	200	404	200	200
Yes (no security)	200	404	200	200
No (no indices permissions)	403	403	403	403
No (indices permissions)	403	404	200	200

The one difference is when a specific index is requested but doesn't exist. I think we get the 404 because of the indices options that the API uses (maybe @javanna can confirm).

We should make the third and fourth row behavior consistent.

The other item I'd like to consider is when a user is not authorized to an index then we would show a 404 regardless of if the index exists or not.

[elasticmachine]

Original comment by @javanna:

As for search, we would rather like to have the third row exactly the same as the fourth one. The reason why it is not the case is that the third user has cluster level permissions, no indices level permission, while the fourth one has some kind of indices level permission. In the latter case, the request has lenient indices options and we let it through to core knowing that it will no yield any result, as it resolves to no indices. In the former case we don't let the request through as the user can't do any indices action at all. This can potentially be worked around I think.

As for cat apis, that is trickier. Cat requests are composed of multiple internal requests, one of which is usually a cluster state request. Cluster state is mapped as a cluster monitor action, although it holds indices that are subject to indices resolution. Shield doesn't check its indices though, it doesn't do any resolving either. Elasticsearch does the resolving though, and may throw unexpected 404. The cat indices api calls cluster state by hardcoding the indices option to strict, hence the 404 when an index doesn't exist. The solution would be to special case or map differently the cluster state request and let shield do the indices resolution rather than core. But that's weird as this is not an indices action.

[javanna]

heads up: we should have a look at how much of this is still valid. Behaviour of indices resolution in security has changed quite a bit since d27c4be . I believe that went out with 6.0. We now honour allow_no_indices, so where we previously would throw 404 errors in security, we can now return the same status code that vanilla elasticsearch returns.

[jasontedor]

Before we can make breaking changes like this to the API we need a form of API versioning so that we can gracefully transition clients through a breaking change like this. For this we have #11184 and I consider some functionality like this a prerequisite.