Discuss use of ServerList for Clients

[NiteshKant]

Pre 0.5.0 clients in RxNetty were tied to a specific host and port. With the API changes proposed in #303 and #317 a single HTTP/TCP client can be used for multiple host + port.
The existing API, forced higher level abstractions to be created on top of RxNetty like ribbon to make it usable in a real-world application. The changes in 0.5.0 makes it possible to use RxNetty as a top level abstraction instead of a low level network library.

This issue, is to discuss viability of providing a ServerList for a client, which is queried for every connect request on a client.

What is a ServerList?

Instead of initializing a client with a SocketAddress as the address for the target server, the client can be initialized with a virtual pool of servers. Then, for every connect() call, the client will query this virtual pool to give a target server back, which will be used for that connect.

What is the use?

This abstraction will provide a natural way to integrate various load balancers into RxNetty without changing the interaction model with the client. In absence of this abstraction, any library like ocelli had to create abstractions on top of RxNetty to be able to use a dynamic pool of servers instead of a well defined list of servers.

By providing this abstraction, users would be able to plug in any load balancer into RxNetty and still keep the same API for using a client.

[NiteshKant]

There are multiple approaches for designing this API and the primary decision point would be whether the interaction with the server list will be asynchronous or synchronous.

Before discussing the API, I think it is important to understand that getting a server from this list, should always have an immediate result. There isn't much reason for this list to wait for a server to be available. During the design of ocelli we have determined that a fast-fail behavior in case when no servers are available is the best way to design a load balancer.

Asynchronous API

For the purpose of discussion, this could be the API for the ServerList

public interface ServerList<T> {

    /**
     * Chooses the next best server on subscription.
     *
     * @return An observable, which when subscribed, will return a single item.
     */
    Observable<T> choose();
}

The advantage of having it designed as an Observable is that it can easily be composed in a connect chain.

Synchronous API

For the purpose of discussion, this could be the API for the ServerList

public interface ServerList<T> {

    /**
     * Returns the next best server.
     *
     * @return The next best server.
     *
     * @throws NoSuchElementException If no items are available.
     */
    T next();
}

This API is honest to the behavior of a ServerList i.e. the result is immediate.

Optimizations

If the ServerList retrieves a SocketAddress (target server) then it would mandate that we maintain a Map of SocketAddress -> Client which would mean that every connect will result in a map lookup. If we have ServerList maintain a form of the Client then we can eliminate the map lookup.
This is the reason ServerList in this proposal is parameterized.

[codefromthecrypt]

seems a little abstract though, right? Is there nothing that could tighten up this context? Originally, I was thinking a 3 tuple like alt-svc of (authority, protocol, socket) as an element of the list. also ServerList is probably not the best name as it isn't a list :)

next() ~ Iterator
choose() ~ Choice

[codefromthecrypt]

oh wait. serverlist scope is authority, so I suppose any structure doesn't need to also pass its authority.

[NiteshKant]

Apologies for the lack of context. So, here is how I intend this to be used:

RxNetty.newTcpClient(serverList).createConnectionRequest();

Inside the Tcp client, assuming ServerList returns a Observable<SocketAddress> with the async API , createConnectionRequest() will be implemented like:

serverList.choose().switchMap(socket -> ... bootstrap.connect(socket) ...)

(Above is just a representation, not a functional code.)

So, ServerList in this case is just the socket information and not related to protocol at all. It is used just to get a connection.

also ServerList is probably not the best name as it isn't a list :)

Agreed. I am looking for suggestions :)

There are other considerations here about how do we give feedback to the ServerList implementation around connection failures, latencies, etc. for it to correctly determine which is the best instance to choose for the next connection request. In order to do that we probably have to extend this interface to also be a MetricEventsListener. This issue is to discuss precisely those points, in order to get to a good API.

[NiteshKant]

\cc @elandau @benjchristensen

[NiteshKant]

PR #366 takes a different approach than discussed in this issue of providing an abstraction of a ConnectionProvider which can be used either to provide a connection pool implementation (provided out of the box) or to create a Load Balancer. I have added an example here to demonstrate simple round-robin load balancing using this abstraction.

This approach has these benefits over the approach discussed here:

• Provides complete flexibility to manage connections and hosts from a load balancer as the load balancer is consulted on every connection request.
• Eliminates a map lookup per request otherwise required if we only choose a host from the load balancer.
• Does not have one extra abstraction of a host selector over the already existing internal connection factory for providing connection pooling.