Smarter request routing which takes recent node latency into account

[nirajpatel]

Hi! I was wondering if there was any ability to blacklist a node while it is recovering. Trying to route GETs, indexing and search to a recovering index can cause the node to fall out of the cluster or can cause the cluster to pause altogether. Is there a way to blacklist a node using the TransportClient that way no traffic is routed to a particular node (or anything similar)? Thanks!

[clintongormley]

Hi @nirajpatel

it's not possible, and I've been trying to think if it makes sense to add. It has complications (eg what happens if all nodes are recovering from each other?). Really, it shouldn't be necessary. Recovery should be throttled so there are sufficient resources on the box to handle search requests etc as well. (This also means providing enough hardware so that your nodes are not at breaking point already, and ensuring that there aren't abusive search requests - something we're trying help with in #11511)

[nirajpatel]

Thank you so much for responding @clintongormley! That certainly makes sense. I was thinking more on the transport client side while the cluster is recovering we could at least throttle requests.

Recovery should be throttled so there are sufficient resources on the box to handle search requests etc as well.
Recovery being throttled would mean higher search, GET, indexing times for an extended period of time correct? Wouldn't you almost want to favor recovering quickly over servicing incoming requests normally? I guess it's a personal tradeoff but if the option existed in the transport client the user could make that decision. What are your thoughts? Thank you!

[clintongormley]

We discussed this in FixItFriday - blacklisting a node is too black or white. Instead a better option would be to have a smarter request routing mechanism than the current round-robin that takes recent node latency into account.

[colings86]

This is the same idea as #3890 so will close in favour of it.