Resolve/cluster allows querying for cluster info only (no index expression required) #119898

quux00 · 2025-01-09T20:08:58Z

Resolve/cluster allows querying for cluster-info-only (no index expression required)

This enhancement provides users with the ability to query the _resolve/cluster API endpoint without specifying
an index expression to match against. This allows users to quickly test what remote clusters are configured on
a cluster and whether they are available for querying.

This new endpoint works with clusters from older versions by querying with the index expression dummy* on those older clusters.

The new endpoint takes no index expression:

GET _resolve/cluster

and returns the same information as before except for the "matching_indices" field. Example response:

{
  "remote1": {
    "connected": false,
    "skip_unavailable": true
  },
  "remote2": {
    "connected": true,
    "skip_unavailable": false,
    "version": {
      "number": "8.17.0",
      "build_flavor": "default",
      "minimum_wire_compatibility_version": "7.17.0",
      "minimum_index_compatibility_version": "7.0.0"
    }
  }
}

elasticsearchmachine · 2025-01-09T20:09:22Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

elasticsearchmachine · 2025-01-09T20:09:23Z

Hi @quux00, I've created a changelog YAML for you.

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestResolveClusterAction.java

...rc/main/java/org/elasticsearch/action/admin/indices/resolve/ResolveClusterActionRequest.java

server/src/main/java/org/elasticsearch/transport/RemoteClusterService.java

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

…ssion required) This enhancement provides users with the ability to query the _resolve/cluster API endpoint without specifying an index expression to match against. This allows users to quickly test what remote clusters are configured on a cluster and whether they are available for querying. This new endpoint works with clusters from older versions. The new endpoint takes no index expression: ``` GET _resolve/cluster ``` and returns the same information as before except for the "matching_indices" field. Example response: ``` { "remote1": { "connected": false, "skip_unavailable": true }, "remote2": { "connected": true, "skip_unavailable": false, "version": { "number": "8.17.0", "build_flavor": "default", "minimum_wire_compatibility_version": "7.17.0", "minimum_index_compatibility_version": "7.0.0" } } } ```

…ng older clusters; updated end user docs

smalyshev · 2025-01-16T00:35:52Z

...rc/main/java/org/elasticsearch/action/admin/indices/resolve/ResolveClusterActionRequest.java

+    // Whether this request is being processed on the primary ("local") cluster being queried or on a remote.
+    // This is needed when clusterInfoOnly=true since we need to know whether to list out all possible remotes
+    // on a node. (We don't want cross-cluster chaining on remotes that might be configured with their own remotes.)
+    private final boolean isQueryingCluster;


I'd name it as something like queryRemotes, since the ultimate goal of this variable is not to identify the querying cluster but to define whether or not remotes should be queried.

No, that's not correct. This variable is intended to specify where the request is being processed on the querying cluster (which should forward it to configured remotes) or remotes (where it should not forward to configured remotes, since we don't support cross-cluster chaining).

smalyshev · 2025-01-16T00:38:00Z

...rc/main/java/org/elasticsearch/action/admin/indices/resolve/ResolveClusterActionRequest.java

@@ -73,6 +82,13 @@ public ResolveClusterActionRequest(StreamInput in) throws IOException {
        this.names = in.readStringArray();
        this.indicesOptions = IndicesOptions.readIndicesOptions(in);
        this.localIndicesRequested = localIndicesPresent(names);
+        if (in.getTransportVersion().onOrAfter(TransportVersions.RESOLVE_CLUSTER_NO_INDEX_EXPRESSION)) {
+            this.clusterInfoOnly = in.readBoolean();
+            this.isQueryingCluster = in.readBoolean();


I wonder though - is there any case where we get this over the wire and it's not false? I mean, it is true on querying cluster, but querying cluster does not get it over the wire, does it? And the only cluster which does get it over the wire is the remote cluster, which must always make this false. If so, is there any point in serializing this flag at all?

is there any point in serializing this flag at all?
Yes, the cluster receiving this request needs to know whether it is the primary cluster or a remote cluster that should not do additional remote cluster lookups (cross-cluster chaining). See the first part of TransportResolveClusterAction.doExecuteForked.

is there any case where we get this over the wire and it's not false?
If there was ever a case where a node could forward the request to a coordinating node on the same cluster it would be true. Not sure that happens, but I'd rather not have to worry about edge cases and just follow the general pattern.

smalyshev · 2025-01-16T00:41:35Z

...st/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1ResolveClusterIT.java

+            assertOK(response2);
+
+            Map<String, Object> responseMap2 = responseAsMap(response2);
+            System.err.println(">>> responseMap2: " + responseMap2);


Should this still be here?

Probably got left over after a debugging session. 😆

Thanks. Fixed in next push.

smalyshev · 2025-01-16T00:42:36Z

...st/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1ResolveClusterIT.java

+            assertNotNull(remoteClusterResponse.get("version"));
+        }
+        // {
+        // // TEST CASE 12: Resolution against wildcarded remote cluster expression that matches no remotes


Why this is commented out? If it's waiting for some other patch or development, it probably should be specified what it is waiting for.

Yeah, I'll add a code comment.

While my code change in this PR fixes the GET _resolve/cluster/*nosuchcluster:foo problem (where it no longer queries the local cluster instead, even though that is clearly NOT what was asked for) for unsecured clusters, the security layer resolution still hits this problem because it has it's own index resolution pathways :-(

So I plan to figure that out in a follow-on PR and left this test as a reminder that needs doing.

smalyshev · 2025-01-16T00:51:11Z

LGTM in general, some nitpicks.

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

…ter-info-only

slobodanadamovic

LGTM (from security perspective)

slobodanadamovic · 2025-01-21T14:56:22Z

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

+                                     * ElasticsearchSecurityException you can't tell whether the remote cluster is available or not, so mark
+                                     * it as connected=false
+                                     */
+                                    resolveClusterInfo = new ResolveClusterInfo(false, skipUnavailable, ese.getMessage());


++ on treating security exception as not connected.

slobodanadamovic · 2025-01-21T15:42:02Z

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java

+                 *    just "*" since that could be an expensive operation on clusters with thousands of indices/aliases/datastreams
+                 */
+                String[] dummyIndexExpr = new String[] { DUMMY_INDEX_FOR_OLDER_CLUSTERS };
+                remoteClusterIndices = remoteClusterService.groupIndices(request.indicesOptions(), dummyIndexExpr, false);


Depending on the request.indicesOptions(), this may result in security exception or index not found.

allow_no_indices=false -> results in IndexNotFoundException

expand_wildcards=none-> will results in ElasticsearchSecurityException if user does not have privileges for dummy* index, because dummy* is now treated as an index name and not expression

Meaning, the response of these calls will differ:

GET _resolve/cluster?allow_no_indices=true&expand_wildcards=none

GET _resolve/cluster?allow_no_indices=false&expand_wildcards=all

GET _resolve/cluster

This is not new behaviour and I'm not sure if this is relevant for this API, but just wanted to raise it since I did not see integration tests that pass any of the indices options.

Thanks very much for flagging this. I think I'll adjust the impl based on that and add some tests.

…ter-info-only

…info-only API variant

…ter-info-only

javanna · 2025-01-22T14:52:24Z

server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestResolveClusterAction.java

+        if (request.hasParam("ignore_throttled")) {
+            indexOptions.add("ignore_throttled");
+        }
+        return indexOptions;


drive-by comment: do we need to add these to a set?

Sure. I'll add that in the next push.

…ter-info-only

elasticsearchmachine · 2025-01-22T17:18:59Z

💚 Backport successful

Status	Branch	Result
✅	8.x

…ssion required) (elastic#119898) Resolve/cluster allows querying for cluster-info-only (no index expression required) This enhancement provides users with the ability to query the _resolve/cluster API endpoint without specifying an index expression to match against. This allows users to quickly test what remote clusters are configured on a cluster and whether they are available for querying. The new endpoint takes no index expression: ``` GET _resolve/cluster ``` and returns the same information as before except for the "matching_indices" field. Example response: ``` { "remote1": { "connected": false, "skip_unavailable": true }, "remote2": { "connected": true, "skip_unavailable": false, "version": { "number": "8.17.0", "build_flavor": "default", "minimum_wire_compatibility_version": "7.17.0", "minimum_index_compatibility_version": "7.0.0" } } } ``` For backwards compatibility, this new endpoint works with clusters from older versions by querying with the index expression `dummy*` on those older clusters and ignoring the matching_indices value in the response they return.

…ssion required) (#119898) (#120650) Resolve/cluster allows querying for cluster-info-only (no index expression required) This enhancement provides users with the ability to query the _resolve/cluster API endpoint without specifying an index expression to match against. This allows users to quickly test what remote clusters are configured on a cluster and whether they are available for querying. The new endpoint takes no index expression: ``` GET _resolve/cluster ``` and returns the same information as before except for the "matching_indices" field. Example response: ``` { "remote1": { "connected": false, "skip_unavailable": true }, "remote2": { "connected": true, "skip_unavailable": false, "version": { "number": "8.17.0", "build_flavor": "default", "minimum_wire_compatibility_version": "7.17.0", "minimum_index_compatibility_version": "7.0.0" } } } ``` For backwards compatibility, this new endpoint works with clusters from older versions by querying with the index expression `dummy*` on those older clusters and ignoring the matching_indices value in the response they return.

quux00 added >enhancement auto-backport Automatically create backport pull requests when merged Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch :Search Foundations/CCS v9.0.0 v8.18.0 labels Jan 9, 2025

quux00 requested review from smalyshev and pawankartik-elastic January 9, 2025 20:08

smalyshev reviewed Jan 9, 2025

View reviewed changes

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java Outdated Show resolved Hide resolved

smalyshev reviewed Jan 9, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestResolveClusterAction.java Show resolved Hide resolved

pawankartik-elastic reviewed Jan 10, 2025

View reviewed changes

quux00 force-pushed the resolve-cluster/cluster-info-only branch 2 times, most recently from 05284eb to 5386ee3 Compare January 14, 2025 20:12

quux00 added 7 commits January 14, 2025 15:14

Bug fix and added additional test to ResolveClusterIT

a23ba34

Update docs/changelog/119898.yaml

9e55c8b

PR feedback

9eacd5b

PR feedback

9c918d7

Fix compilation bug introduced in previous PR feedback commit

4507582

Changed to using index expression 'dummy*' rather than '*' for queryi…

46a7f28

…ng older clusters; updated end user docs

quux00 force-pushed the resolve-cluster/cluster-info-only branch from 5386ee3 to 46a7f28 Compare January 14, 2025 20:14

Updates to end user docs

81e0428

quux00 requested review from pawankartik-elastic, smalyshev and jakelandis January 15, 2025 13:43

smalyshev reviewed Jan 16, 2025

View reviewed changes

slobodanadamovic removed the request for review from jakelandis January 16, 2025 12:55

slobodanadamovic self-requested a review January 16, 2025 12:55

pawankartik-elastic reviewed Jan 16, 2025

View reviewed changes

.../main/java/org/elasticsearch/action/admin/indices/resolve/TransportResolveClusterAction.java Outdated Show resolved Hide resolved

quux00 added 3 commits January 17, 2025 12:42

Changes based on PR feedback

6c51f22

Merge remote-tracking branch 'elastic/main' into resolve-cluster/clus…

5c203fb

…ter-info-only

Fixed singleton String array instantiation

7ad8326

slobodanadamovic approved these changes Jan 21, 2025

View reviewed changes

quux00 added 6 commits January 21, 2025 12:12

Merge remote-tracking branch 'elastic/main' into resolve-cluster/clus…

75f31c8

…ter-info-only

Added restriction to no allow index options for clusterInfoOnly endpoint

d0f39ed

Merge remote-tracking branch 'elastic/main' into resolve-cluster/clus…

17251d2

…ter-info-only

Updated end user docs around no specifying index options for cluster-…

96c6318

…info-only API variant

Modifications to end user docs

64893c9

Merge remote-tracking branch 'elastic/main' into resolve-cluster/clus…

bab960c

…ter-info-only

javanna reviewed Jan 22, 2025

View reviewed changes

quux00 added 2 commits January 22, 2025 10:48

PR feedback: made a set of the INDEX_OPTIONS_PARAMS

1e8e520

Merge remote-tracking branch 'elastic/main' into resolve-cluster/clus…

e53f81c

…ter-info-only

quux00 merged commit b3a032c into elastic:main Jan 22, 2025
16 checks passed

quux00 mentioned this pull request Jan 22, 2025

[8.x] Resolve/cluster allows querying for cluster info only (no index expression required) (#119898) #120650

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve/cluster allows querying for cluster info only (no index expression required) #119898

Resolve/cluster allows querying for cluster info only (no index expression required) #119898

quux00 commented Jan 9, 2025 •

edited

Loading

elasticsearchmachine commented Jan 9, 2025

elasticsearchmachine commented Jan 9, 2025

smalyshev Jan 16, 2025

quux00 Jan 17, 2025

smalyshev Jan 16, 2025

quux00 Jan 17, 2025

smalyshev Jan 16, 2025

pawankartik-elastic Jan 16, 2025

quux00 Jan 17, 2025

smalyshev Jan 16, 2025

quux00 Jan 17, 2025

smalyshev commented Jan 16, 2025

slobodanadamovic left a comment

slobodanadamovic Jan 21, 2025

slobodanadamovic Jan 21, 2025 •

edited

Loading

quux00 Jan 21, 2025

javanna Jan 22, 2025

quux00 Jan 22, 2025

elasticsearchmachine commented Jan 22, 2025

Resolve/cluster allows querying for cluster info only (no index expression required) #119898

Resolve/cluster allows querying for cluster info only (no index expression required) #119898

Conversation

quux00 commented Jan 9, 2025 • edited Loading

elasticsearchmachine commented Jan 9, 2025

elasticsearchmachine commented Jan 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smalyshev commented Jan 16, 2025

slobodanadamovic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slobodanadamovic Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Jan 22, 2025

💚 Backport successful

quux00 commented Jan 9, 2025 •

edited

Loading

slobodanadamovic Jan 21, 2025 •

edited

Loading