Keep track of connections, introduce per-vhost limits

[michaelklishin]

This introduces (client) connection tracking that's aware of connection

• vhost
• username
• protocol

and so on across the cluster. We already collect much of this information
in node-local state but that's not sufficient for #500, #607, and related features.

This changes internal database schema and thus can only go into 3.7.0.

This also makes sure that when a vhost is deleted, we force close all connections in it.

This is a 3rd iteration on the design. All connections and counters are tracked using
node-specific tables which are only updated on the node itself but read by all cluster nodes.
This avoids concurrent counter updates without going all the way to CRDTs.

The implementation isn't perfect but I run out of practical test cases to add. Points of improvement for the future:

• Parallel folding over per-node connection tracking tables
• Management plugin extensions related to limit management
• rabbitmqctl status and report entries related to limits
• As with many other things, replacing Mnesia would open new possibilities but it's a long term goal

Fixes #500, #627.
Lays ground work for #501, #607.

Depends on rabbitmq/rabbitmq-common#121.

[michaelklishin]

Feel free to QA but I have one more test case to add => do not merge yet.

[michaelklishin]

Also: this has a rabbitmqctl(1) entry but still needs a doc guide.

[dumbbell]

You probably want to use ?PER_VHOST_COUNTER_TABLE here.

[michaelklishin]

This approach suffers from the lost updates problem when connections from a failed node are removed concurrently. We have evaluated a couple of approaches and the most prominent so far seems to be:

• Have a pair of connection tracking tables per node
• Every node updates its own tables
• Every node reads from all tables in order to come up with a total count

So, one writer, many readers. This should also make it possible to eliminate the connection re-registration mechanism in place today.

[Ayanda-D]

is a case statement is necessary here? if Cs = [], then nothing to evaluate in the list comprehension, and you can just return {ok, State}.

[Ayanda-D]

this might not lie on the critical path, but use of proplists:get_value/2 seems quite extensive. how about using rabbit_misc:pget/2 instead, considering that it was optimised in rabbitmq/rabbitmq-common#51

[michaelklishin]

Good idea, thank you.

[michaelklishin]

This affected connection tracking in the stats DB, now fixed: rabbitmq/rabbitmq-management#274.

[michaelklishin]

@dumbbell this is ready for another round.

[Ayanda-D]

@michaelklishin you're going through 2 lists just to calculate number of connections? imagine how much this will impact performance (per connection open request) as cluster size increases? can you use lists:foldl/3 only please, with connection count as your accumulator. there's no need for lists:map/2 here.

[michaelklishin]

Sorry but you are worrying about the wrong thing here. What really needs to happen is doing a parallel map but for small clusters it will likely be less efficient so I decided that it can be done at a later point.

It's not common to see clusters with more than maybe 7-9 nodes and it's unlikely to be a major contributor to how many connections/second can be accepted. Plus if you don't have any limits configured, none of this code will be executed.

[Ayanda-D]

you can also make Chunks your accumulator in lists:foldl/3 for both your listing functions (unless you were planning something else)