Connection churn and disk usage of Raft WAL segments

[dhxgit]

Hi,

I have found a reproduce-able issue with up to and including version 3.10.6 of rabbitmq.

What happens:

• Every Connection attempts seems to be stored/logged in quorum segment files
  • The needed storage grows and grows until full
    • ( Queue gets cleaned up if a message flows through the queue )

Reproduction Setup: ( see https://github.com/dhxgit/rmq-storage-issue )

• We start the official rabbitmq container (see docker-compose.yaml)
• We start a python test container (also in my repo) which in a loop with a lot of workers
  • connects to a queue with basic_consume
  • kills basic_consume after a short 2~3 second timeout
• After a few minutes of runtime we can restart the rabbitmq-container
  • this will write out the segments files into the quorum directory
  • this will also happen without restart when the wal-file gets rotated

Result:

• Given enough time * clients * frequency of connections * queues
  • -> Server storage will fill up, even though the queues are completely empty.
    ( The segments files will be cleaned up if any message flow through the queue, but if that does not happen the storage grows until completely full )

Even though this is a somewhat unusual case (un-used queues that are checked by consumers that constantly restart consuming), I believe that it should not fill up the storage with segments files which just seem to contain the consumer-tags.

[michaelklishin]

Only MQTT connections are tracked using Raft log (because client IDs must be unique).

Quorum queue Raft "system" is different from that of MQTT, so they use different storage directories.
Some WAL management settings can be controlled but in general, given a small enough disk,
there will be a scenario where either

1. The Raft log needs more space than is available, or
2. It cannot write out a commit successfully

[michaelklishin]

When you say "kill basic_consume", what exactly does that mean? A traffic capture would be very helpful to see what exactly the server receives (or does not receive) in this scenario.

It can be that your setup leaks consumers in a loop.

[glego]

kill basic_consume, basically means that the consumer process is stopped and restarted, which creates a new connection and consumer tag

[dhxgit]

my python reproduce-code sample uses process.terminate() on spawned consumer procresses after reaching the timeout (2~3 sec). I think this is sending SIGTERM to the process.

[kjnilsson]

Every basic consume request results in a state machine change and thus a Ra log entry. It is correct that we don't currently perform snapshotting for the scenario of basic consume against an empty queue then killing the connection. We do if the basic consume is ended with a basic cancel but for terminations we missed to do that.

I think we can promote this to a bug.