[Enhancement] Additional query stats page size and number of results per shard, index and node

[yaronp68]

Describe the feature:
In order to get a better understanding of the search load on each shard, index or node at a given time, it could be beneficial to collect page size specified in the query and the number of actual matches returned. If would be great if ES could collect it and calculate average and sd per second per shard and per index and per node and add that to the relevant stats APIs.

CC: @pickypg @ycombinator

[elasticmachine]

Pinging @elastic/es-core-infra

[polyfractal]

• Depending on how Do not compute hit counts by default #33028 turns out, returning the number of matches could be tricky, since we may not actually know the number of matches. I'm not sure how useful the metric would be if it were just showing the average of the "lower bound" of matches?

  OTOH, it may still be useful since it shows what work is actually being done on the shard in terms of matches.

• I'm on the fence about per-node / per-shard stats. E.g. what action would be taken by knowing that a particular shard or node had a high number of matches? Since searches are always against whole indices (excepting custom routing I guess), perhaps per-index is all we need?

[dakrone]

Internal monitoring has been deprecated for quite a while, and we're no longer doing any active development on it. I'm going to close this issue.