Improve thread pools

[jasontedor]

This issue is a meta-issue for collapsing several sub-issues for improving thread pools in core Elasticsearch:

• register all thread pool settings Register thread pool settings #18674
• provide hook for plugins to register a thread pool Register thread pool settings #18674
• thread pool settings should be node-level settings Register thread pool settings #18674
• remove unbounded queues Remove unbounded queues from thread pool executors #18491
• place a hard bound of 256 * max number of threads in pool for a bound on the queue Limit the maximum queue_size for fixed thread pools? #14448
• reduce the default maximum size of the generic thread pool
• bound the size of thread pools (e.g., as a function of the number of available processors)

The proposal here is that core Elasticsearch will no longer provide support for custom executors. Of course, a plugin can always create an executor on its own and so core Elasticsearch will provide an extension point for such plugins to provide stats back to core Elasticsearch for reading in node stats API and cat thread pools API.

Relates #17915, relates #18491, relates #14448, relates #15866

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

This issue has been stalled due to complexities of the general threadpool (and others) that are currently unbounded. The concern is that bounding them may cause contention, and could (in theory?) cause deadlocks. I don't think this will be theoretically resolved. We know that having unbounded queues has problems of its own, as outlined here. I propose that we move forward now by implementing the hard bound on queues suggested here (256 * max threads) in master for 8.0. This gives us sufficient time to test through our own daily performance tests, as well as users to test in the alphas/betas for 8.0.

[ldematte]

@rjernst can you comment on what is still missing and what we would need to do next (if anything?), please?
Thanks!

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)