Profile API should show node details as well as shard details

docs/reference/search/profile.asciidoc

@@ -71,7 +71,11 @@ The API returns the following result:
   "profile": {
     "shards": [
       {
-        "id": "[2aE02wS1R8q_QFnYu6vDVQ][my-index-000001][0]",
+        "id": "[q2aE02wS1R8qQFnYu6vDVQ][my-index-000001][0]",
+        "node_id": "q2aE02wS1R8qQFnYu6vDVQ",
+        "shard_id": 0,
+        "index": "my-index-000001",
+        "cluster": "(local)",
         "searches": [
           {
             "query": [
@@ -221,7 +225,8 @@ The API returns the following result:
 // TESTRESPONSE[s/"took": 25/"took": $body.took/]
 // TESTRESPONSE[s/"hits": \[...\]/"hits": $body.$_path/]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
-// TESTRESPONSE[s/\[2aE02wS1R8q_QFnYu6vDVQ\]\[my-index-000001\]\[0\]/$body.$_path/]
+// TESTRESPONSE[s/"id": "\[q2aE02wS1R8qQFnYu6vDVQ\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/]
+// TESTRESPONSE[s/"node_id": "q2aE02wS1R8qQFnYu6vDVQ",/"node_id": "$body.profile.shards.0.node_id",/]
 
 <1> Search results are returned, but were omitted here for brevity.
 
@@ -237,40 +242,52 @@ The overall structure of the profile response is as follows:
    "profile": {
         "shards": [
            {
-              "id": "[2aE02wS1R8q_QFnYu6vDVQ][my-index-000001][0]",  <1>
+              "id": "[q2aE02wS1R8qQFnYu6vDVQ][my-index-000001][0]",  <1>
+              "node_id": "q2aE02wS1R8qQFnYu6vDVQ",
+              "shard_id": 0,
+              "index": "my-index-000001",
+              "cluster": "(local)",             <2>
               "searches": [
                  {
-                    "query": [...],             <2>
-                    "rewrite_time": 51443,      <3>
-                    "collector": [...]          <4>
+                    "query": [...],             <3>
+                    "rewrite_time": 51443,      <4>
+                    "collector": [...]          <5>
                  }
               ],
-              "aggregations": [...],            <5>
-              "fetch": {...}                    <6>
+              "aggregations": [...],            <6>
+              "fetch": {...}                    <7>
            }
         ]
      }
 }
 --------------------------------------------------
 // TESTRESPONSE[s/"profile": /"took": $body.took, "timed_out": $body.timed_out, "_shards": $body._shards, "hits": $body.hits, "profile": /]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
-// TESTRESPONSE[s/\[2aE02wS1R8q_QFnYu6vDVQ\]\[my-index-000001\]\[0\]/$body.$_path/]
+// TESTRESPONSE[s/"id": "\[q2aE02wS1R8qQFnYu6vDVQ\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/]
+// TESTRESPONSE[s/"node_id": "q2aE02wS1R8qQFnYu6vDVQ",/"node_id": "$body.profile.shards.0.node_id",/]
 // TESTRESPONSE[s/"query": \[...\]/"query": $body.$_path/]
 // TESTRESPONSE[s/"collector": \[...\]/"collector": $body.$_path/]
 // TESTRESPONSE[s/"aggregations": \[...\]/"aggregations": []/]
 // TESTRESPONSE[s/"fetch": \{...\}/"fetch": $body.$_path/]
 <1> A profile is returned for each shard that participated in the response, and
 is identified by a unique ID.
-<2> Query timings and other debugging information.
-<3> The cumulative rewrite time.
-<4> Names and invocation timings for each collector.
-<5> Aggregation timings, invocation counts, and debug information.
-<6> Fetch timing and debug information.
+<2> If the query was run on the local cluster, the cluster name is left out of the
+composite id and is marked "(local)" here. For a profile running on a remote_cluster
+using cross-cluster search, the "id" value would be something like
+`[q2aE02wS1R8qQFnYu6vDVQ][remote1:my-index-000001][0]` and the "cluster"
+value would be `remote1`.
+<3> Query timings and other debugging information.
+<4> The cumulative rewrite time.
+<5> Names and invocation timings for each collector.
+<6> Aggregation timings, invocation counts, and debug information.
+<7> Fetch timing and debug information.
 
 Because a search request may be executed against one or more shards in an index,
 and a search may cover one or more indices, the top level element in the profile
 response is an array of `shard` objects. Each shard object lists its `id` which
 uniquely identifies the shard. The ID's format is
+`[nodeID][clusterName:indexName][shardID]`. If the search is run against the
+local cluster then the clusterName is not added and the format is
 `[nodeID][indexName][shardID]`.
 
 The profile itself may consist of one or more "searches", where a search is a
@@ -343,7 +360,7 @@ Using our previous `match` query example, let's analyze the `query` section:
     }
 ]
 --------------------------------------------------
-// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.$_path",\n"searches": [{\n/]
+// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.profile.shards.0.id",\n"node_id": "$body.profile.shards.0.node_id",\n"shard_id": $body.profile.shards.0.shard_id,\n"index": "$body.profile.shards.0.index",\n"cluster": "(local)",\n"searches": [{\n/]
 // TESTRESPONSE[s/]$/],"rewrite_time": $body.$_path, "collector": $body.$_path}], "aggregations": [], "fetch": $body.$_path}]}}/]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
 // TESTRESPONSE[s/"breakdown": \{...\}/"breakdown": $body.$_path/]
@@ -397,7 +414,7 @@ Lucene execution:
   "count_weight_count": 0
 }
 --------------------------------------------------
-// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.$_path",\n"searches": [{\n"query": [{\n"type": "BooleanQuery",\n"description": "message:get message:search",\n"time_in_nanos": $body.$_path,/]
+// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.profile.shards.0.id",\n"node_id": "$body.profile.shards.0.node_id",\n"shard_id": $body.profile.shards.0.shard_id,\n"index": "$body.profile.shards.0.index",\n"cluster": "(local)",\n"searches": [{\n"query": [{\n"type": "BooleanQuery",\n"description": "message:get message:search",\n"time_in_nanos": $body.$_path,/]
 // TESTRESPONSE[s/}$/},\n"children": $body.$_path}],\n"rewrite_time": $body.$_path, "collector": $body.$_path}], "aggregations": [], "fetch": $body.$_path}]}}/]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
 
@@ -498,7 +515,7 @@ Looking at the previous example:
   }
 ]
 --------------------------------------------------
-// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.$_path",\n"searches": [{\n"query": $body.$_path,\n"rewrite_time": $body.$_path,/]
+// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.profile.shards.0.id",\n"node_id": "$body.profile.shards.0.node_id",\n"shard_id": $body.profile.shards.0.shard_id,\n"index": "$body.profile.shards.0.index",\n"cluster": "(local)",\n"searches": [{\n"query": $body.$_path,\n"rewrite_time": $body.$_path,/]
 // TESTRESPONSE[s/]$/]}], "aggregations": [], "fetch": $body.$_path}]}}/]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
 
@@ -620,7 +637,7 @@ GET /my-index-000001/_search
 }
 --------------------------------------------------
 // TEST[setup:my_index]
-// TEST[s/_search/_search\?filter_path=profile.shards.id,profile.shards.searches,profile.shards.aggregations,profile.shards.fetch/]
+// TEST[s/_search/_search\?filter_path=profile.shards.id,profile.shards.node_id,profile.shards.shard_id,profile.shards.index,profile.shards.cluster,profile.shards.searches,profile.shards.aggregations,profile.shards.fetch/]
 
 
 This example has:
@@ -640,7 +657,11 @@ The API returns the following result:
   "profile": {
     "shards": [
       {
-        "id": "[P6-vulHtQRWuD4YnubWb7A][my-index-000001][0]",
+        "id": "[P6xvulHtQRWuD4YnubWb7A][my-index-000001][0]",
+        "node_id": "P6xvulHtQRWuD4YnubWb7A",
+        "shard_id": 0,
+        "index": "my-index-000001",
+        "cluster": "(local)",
         "searches": [
           {
             "query": [
@@ -739,7 +760,8 @@ The API returns the following result:
 // TESTRESPONSE[s/"fetch": \{\.\.\.\}/"fetch": $body.$_path/]
 // TESTRESPONSE[s/\.\.\.//]
 // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
-// TESTRESPONSE[s/"id": "\[P6-vulHtQRWuD4YnubWb7A\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/]
+// TESTRESPONSE[s/"id": "\[P6xvulHtQRWuD4YnubWb7A\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/]
+// TESTRESPONSE[s/"node_id": "P6xvulHtQRWuD4YnubWb7A",/"node_id": "$body.profile.shards.0.node_id",/]
 <1> The `"aggregations"` portion has been omitted because it will be covered in
 the next section.
 
@@ -1176,7 +1198,7 @@ One of the `dfs` sections for a shard looks like the following:
     }
 }
 --------------------------------------------------
-// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ "$body.$_path", {\n"id": "$body.$_path",\n/]
+// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ "$body.$_path", {\n"id": "$body.$_path",\n"node_id": "$body.$_path",\n"shard_id": "$body.$_path",\n"index": "$body.$_path",\n"cluster": "$body.$_path",\n/]
 // TESTRESPONSE[s/}$/}, "aggregations": [], "searches": $body.$_path}]}}/]
 // TESTRESPONSE[s/(\-)?[0-9]+/ $body.$_path/]
 
@@ -1290,7 +1312,7 @@ One of the `dfs.knn` sections for a shard looks like the following:
     }   ]
 }
 --------------------------------------------------
-// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.$_path",\n/]
+// TESTRESPONSE[s/^/{\n"took": $body.took,\n"timed_out": $body.timed_out,\n"_shards": $body._shards,\n"hits": $body.hits,\n"profile": {\n"shards": [ {\n"id": "$body.$_path",\n"node_id": "$body.$_path",\n"shard_id": "$body.$_path",\n"index": "$body.$_path",\n"cluster": "$body.$_path",\n/]
 // TESTRESPONSE[s/}$/}, "aggregations": [], "searches": $body.$_path, "fetch": $body.$_path}]}}/]
 // TESTRESPONSE[s/ (\-)?[0-9]+/ $body.$_path/]
 // TESTRESPONSE[s/"dfs" : \{/"dfs" : {"statistics": $body.$_path,/]

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/370_profile.yml

@@ -401,3 +401,30 @@ dfs profile for search with dfs_query_then_fetch:
 
   - match: { hits.total.value: 0 }
   - is_false: profile.shards.0.dfs
+---
+composite_id_parsed:
+  - skip:
+      version: " - 8.9.99"
+      reason: node_id, shard_id, index and cluster fields added in 8.10.0
+      features: "contains"
+
+  - do:
+      search:
+        index: test
+        body:
+          _source: false
+          profile: true
+          fields: [keyword]
+  - set:  { profile.shards.0.id: composite_id,
+            profile.shards.0.node_id: first_node_id,
+            profile.shards.0.index: first_index
+          }
+
+  - is_true: profile.shards.0.id
+  - is_true: profile.shards.0.node_id
+  - length: { profile.shards.0.node_id: 22 }
+  - gte: { profile.shards.0.shard_id: 0 }
+  - match: { profile.shards.0.index: "test" }
+  - is_true: profile.shards.0.cluster
+  - contains: { $composite_id: $first_node_id }
+  - contains: { $composite_id: $first_index }

server/src/main/java/org/elasticsearch/search/SearchShardTarget.java

@@ -23,7 +23,6 @@
  * The target that the search request was executed on.
  */
 public final class SearchShardTarget implements Writeable, Comparable<SearchShardTarget> {
-
     private final Text nodeId;
     private final ShardId shardId;
     private final String clusterAlias;
@@ -113,6 +112,11 @@ public int hashCode() {
         return Objects.hash(nodeId, shardId, clusterAlias);
     }
 
+    /**
+     * NOTE: this representation is used as the "id" for shards for the REST response
+     * when query profiling is requested. So changing this formulation may break
+     * systems that rely on the format, including the parser in SearchProfileResults.
+     */
     @Override
     public String toString() {
         String shardToString = "["

server/src/main/java/org/elasticsearch/search/profile/SearchProfileResults.java

@@ -8,11 +8,14 @@
 
 package org.elasticsearch.search.profile;
 
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
 import org.elasticsearch.TransportVersion;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.io.stream.Writeable;
+import org.elasticsearch.core.Nullable;
 import org.elasticsearch.search.profile.aggregation.AggregationProfileShardResult;
 import org.elasticsearch.search.profile.query.QueryProfileShardResult;
 import org.elasticsearch.xcontent.ToXContentFragment;
@@ -26,6 +29,8 @@
 import java.util.List;
 import java.util.Map;
 import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
 
 import static org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken;
 
@@ -34,10 +39,16 @@
  */
 public final class SearchProfileResults implements Writeable, ToXContentFragment {
 
+    private static final Logger logger = LogManager.getLogger(SearchProfileResults.class);
     private static final String ID_FIELD = "id";
+    private static final String NODE_ID_FIELD = "node_id";
+    private static final String CLUSTER_FIELD = "cluster";
+    private static final String INDEX_NAME_FIELD = "index";
+    private static final String SHARD_ID_FIELD = "shard_id";
     private static final String SHARDS_FIELD = "shards";
     public static final String PROFILE_FIELD = "profile";
 
+    // map key is the composite "id" of form [nodeId][(clusterName:)indexName][shardId] created from SearchShardTarget.toString
     private Map<String, SearchProfileShardResult> shardResults;
 
     public SearchProfileResults(Map<String, SearchProfileShardResult> shardResults) {
@@ -76,9 +87,24 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         for (String key : sortedKeys) {
             builder.startObject();
             builder.field(ID_FIELD, key);
-            shardResults.get(key).toXContent(builder, params);
+
+            ShardProfileId shardProfileId = parseCompositeProfileShardId(key);
+            if (shardProfileId != null) {
+                builder.field(NODE_ID_FIELD, shardProfileId.nodeId());
+                builder.field(SHARD_ID_FIELD, shardProfileId.shardId());
+                builder.field(INDEX_NAME_FIELD, shardProfileId.indexName());
+                String cluster = shardProfileId.clusterName();
+                if (cluster == null) {
+                    cluster = "(local)";
+                }
+                builder.field(CLUSTER_FIELD, cluster);
+            }
+
+            SearchProfileShardResult shardResult = shardResults.get(key);
+            shardResult.toXContent(builder, params);
             builder.endObject();
         }
+
         builder.endArray().endObject();
         return builder;
     }
@@ -170,4 +196,54 @@ private static void parseProfileResultsEntry(XContentParser parser, Map<String,
         result.getQueryPhase().setSearchProfileDfsPhaseResult(searchProfileDfsPhaseResult);
         searchProfileResults.put(id, result);
     }
+
+    /**
+     * Parsed representation of a composite id used for shards in a profile.
+     * The composite id format is specified/created via the {@code SearchShardTarget} method.
+     * @param nodeId nodeId that the shard is on
+     * @param indexName index being profiled
+     * @param shardId shard id being profiled
+     * @param clusterName if a CCS search, the remote clusters will have a name in the id. Local clusters will be null.
+     */
+    record ShardProfileId(String nodeId, String indexName, int shardId, @Nullable String clusterName) {}
+
+    private static Pattern SHARD_ID_DECOMPOSITION = Pattern.compile("\\[([^]]+)\\]\\[([^]]+)\\]\\[(\\d+)\\]");
+
+    /**
+     * Parse the composite "shard id" from the profiles output, which comes from the
+     * {@code SearchShardTarget.toString()} method, into its separate components.
+     * <p>
+     * One of two expected patterns is accepted:
+     * <p>
+     * 1) [nodeId][indexName][shardId]
+     * example: [2m7SW9oIRrirdrwirM1mwQ][blogs][1]
+     * <p>
+     * 2) [nodeId][clusterName:indexName][shardId]
+     * example: [UngEVXTBQL-7w5j_tftGAQ][remote1:blogs][0]
+     *
+     * @param compositeId see above for accepted formats
+     * @return ShardProfileId with parsed components or null if the compositeId has an unsupported format
+     */
+    static ShardProfileId parseCompositeProfileShardId(String compositeId) {
+        assert Strings.isNullOrEmpty(compositeId) == false : "An empty id should not be passed to parseCompositeProfileShardId";
+
+        Matcher m = SHARD_ID_DECOMPOSITION.matcher(compositeId);
+        if (m.find()) {
+            String nodeId = m.group(1);
+            String indexName = m.group(2);
+            int shardId = Integer.parseInt(m.group(3));
+            String cluster = null;
+            if (indexName.contains(":")) {
+                // index names and cluster names cannot contain a ':', so this split should be accurate
+                String[] tokens = indexName.split(":", 2);
+                cluster = tokens[0];
+                indexName = tokens[1];
+            }
+            return new ShardProfileId(nodeId, indexName, shardId, cluster);
+        } else {
+            assert false : "Unable to match input against expected pattern of [nodeId][indexName][shardId]. Input: " + compositeId;
+            logger.warn("Unable to match input against expected pattern of [nodeId][indexName][shardId]. Input: {}", compositeId);
+            return null;
+        }
+    }
 }