RabbitMQ crash under certain conditions after upgrading to Version 3.13.6 on Kubernetes in GCP

[radeth]

Describe the bug

Hi RabbitMQ Team,
I'm experiencing an issue with RabbitMQ deployed using the Cluster Operator version v0.15.0 on a Kubernetes cluster running in GCP. After upgrading RabbitMQ from version 3.10.2 to 3.13.6, the system started crashing under certain conditions, and I’m seeking your guidance to resolve this.
The error in the logs indicates:
[os_mon] memory supervisor port (memsup): Erlang has closed.
The crashes occur when I attempt to fetch more than 10,000 messages with the automatic ack option enabled, particularly when the queue size exceeds 100MB. However, the issue also happens sporadically without a clearly identifiable trigger. Interestingly, monitoring the RAM usage does not show any significant increase prior to the crash. When fetching fewer messages (below the threshold that causes a crash), the memory usage appears normal and aligns with expected behavior.
This problem was not present in the previous RabbitMQ version 3.10.2 and only started occurring after the upgrade to 3.13.6. My deployment uses the RabbitMQ Cluster Operator v0.15.0, and this setup has been stable prior to the upgrade.
Is this a known issue or compatibility problem between RabbitMQ 3.13.6 and the Cluster Operator v0.15.0? Could the error be related to the Erlang runtime, and are there any specific configurations that might prevent the memsup crash? If additional logs or diagnostic details would help, I am happy to provide them.
Thank you for your time and support in resolving this issue.
Best regards,

Reproduction steps

1. Upgrade rabbit in GKE to version 3.13.6
2. Read large amount of lagre messages, more than 100mb with automatic ack mode

Expected behavior

Rabbit should not crash

Additional context

No response

[mkuratczyk]

First of all:

• cluster operator 0.15.0 is 4.5 years old alpha version of the operator. There are no "known issues" with this version because no-one is using it and no-one is testing it. I'd recommend upgrading ASAP.
• RabbitMQ 3.13.6 is out of community support now (upgrade to 4.0 or purchase long term support for 3.13). Even if this discussion leads to a bug being identified and fixed, that fix will only be included in 4.0 (and perhaps back-ported to the 3.13 LTS version)

There's not enough details to suggest much. Most likely this particular operation uses a bit more memory than it used to but you didn't even mention what type of queue that is.

If you want further assistance, provide clear reproduction steps and your deployment YAML, so we can see your memory configuration. You can also play around with https://perftest.rabbitmq.com/ to reproduce and test different settings. If you end up using perf-test to reproduce, the commands will give us all the details we need (queue type and arguments, message size, prefetch size and so on and so forth).

[michaelklishin]

If "crash under certain conditions" means a node (pod) termination, the first suspect is usually an OOM killer, so start with Reasoning About Memory Use.

If it means a logged exception (a "crash" in Erlang terms), then we cannot suggest anything without seeing the logs (on 4.0.5) with at few hours of log entries before and after the event in question.

[radeth]

A few things I wanted to clarify:

1. We are using operator version 2.10.0.
2. Queue type: classic, but I also tested on quorum, durable: true, storage_version: 1.
3. On the RAM usage chart, when I use a lot of RAM but not enough to cause a crash, I can see the memory usage, but not before the crash.

[radeth]

I updated RabbitMQ to version 4.0.5 and the cluster operator to version 2.12.1. After the update, I fetched messages using the automatic ack option, which caused RabbitMQ to crash with the logs below.

I am also attaching a screenshot of the memory allocation in RabbitMQ during the crash, as well as information about the queue from which I fetched the messages.

[radeth]

Here is a summary of RabbitMQ alarms and crashes. It is evident that an alarm does not always cause a crash

[mkuratczyk]

What is the scale here? How much memory is used in that spike? Please share the deployment YAML.

Can you try to create a full reproduction case using https://perftest.rabbitmq.com/ ? If we have all the info required like:

1. YAML to deploy
2. perf-test command to load the queue with messages (this is likely the most important)
3. perf-test command to consume the message(s) (or perhaps you do this via the Management UI?)

[radeth]

The value on charts is on bytes. Should i do this test on local machine or on GKE ?

Here is my yaml confing for gke

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: dev-rabbitmqcluster
  namespace: rabbitmq-system
spec:
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 1
      memory: 4Gi
  persistence:
    storageClassName: rabbitmq-sc-cmek 
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq  
              image: rabbitmq:4.0.5-management
              securityContext:
                privileged: true
            initContainers:
            - name: setup-container
              image: rabbitmq:4.0.5-management

  rabbitmq:
          additionalConfig: |
                  vm_memory_high_watermark.absolute = 2460MiB
                  default_user = user 
                  default_pass = pass
                  default_user_tags.administrator = true
          additionalPlugins:
            - rabbitmq_shovel_management
  service:
    annotations: 
      cloud.google.com/load-balancer-type: "Internal"
    type: LoadBalancer            

[mkuratczyk]

As long as you can trigger a significant memory spike, it doesn't really matter where you try it.