balancer: refactor package base into reusable components

[menghanl]

Example of components:

• Connection Manager
  • Sub connectivity state aggregator (like ConnectivityStateEvaluator. But should consider connecting after ready an error state, instead of connecting.
• Picker Builder

This helps decouple connection managing with RPC picking.
See more discussion in #3108

We should try to avoid adding features to the base balancer (thus causing API changes and/or complex code). The users are expected to build their own balancer with the reusable components, instead of based on the base balancer as a framework.

[MatthewDolan]

Recently, we were trying to write a balancer and ran into a few issues that I wanted to share in case they could be used to influence the design of the updates to the balancer/base package.

Background

We wanted to create a balancer that preferred instances in a local availability zone if they were present, then preferred instances in a local region if they were present, and ultimately failed back to any instance globally. This could be used to reduce the round trip time and cost of requests.

We used a custom resolver to set information on the resolver.Address.Attributes field about the region and zone of each address and then passed in a function to the balancer to indicate preference. (Not sure if it's useful but, for context, we are using kubernetes in Google Cloud with a custom resolver so we could get the zone and region by looking up the node labels when addresses were resolved for a service).

Problem

We wanted (ideally) this balancer to be composable with other balancers. So we could, for instance, indicate that a client should prefer local zones, but within a local zone, use round robin load balancing or consistent hashing, or whatever their current balancer is.

We found it difficult to compose two balancers together given the current api and split between base and balancer packages. This was mostly due to the fact that existing balancers don't expose their internals (e.g. they don't expose a builder picker if they use one).

Current workaround

Ultimately, we had to make a copy of any balancer we wanted to compose this balancer with and build a custom version of that balancer with exported internal types.

Additional details

A second example of wanting this composability is when we wanted to wrap our existing balancer with instrumentation. For example, we wanted to add spans around the balancer calls so we could more easily see when the addresses were updated in the balancer and how often addresses were being picked.

In the issue description you mentioned reusable components and that might have also enabled us to solve this problem, but I guess it would depend on what reusable components balancer implementations would chose to export.

[menghanl]

@MatthewDolan Thanks for the information. That's very helpful.

The first example you described sounds like a priority balancer.
This balancer manages a list of child balancers, instead of connections. It accepts a balancer config containing the priorities, and the child balancing policy for each. It also takes addresses that each tagged with the priority they belong to (in Attributes).
This is a feature I'm planning to work on soon.

There might be another option to build this priority balancer with the reusable components. The advantage of the previous balancer-wrapper approach is it works for any balancers. The advantage of the components approach probably will be flexibility.

The second example is something the reusable components can solve.

[jedrp]

Recently, we want to reduce the latency for internal communications between gRPC services. (before we are using linkerd as proxy to do the load balancing but it drops down the throughput)

as I can see now, the balancer package all APIs in this package are experimental. so what should I be careful when using this package on production-grade.

[zasweq]

This is one possible way addressing of addressing the concerns of #2909. Closing this issue, and will leave other issue open until we decide how we want to handle this.