HiveMQ CE from docker hub fails to start on Azure Kubernetes service with persistent volume, from version 2024.2 and forward.

[MikalJ]

Expected behavior

To be able to deploy a HiveMQ instance with persistent storage on Azure Kubernetes service

Actual behavior

Version 2024.2 and onward fails to initiate properly with an error.

2025-02-06 14:24:32,680 ERROR - Could not open data lock file.
java.nio.file.AccessDeniedException: /opt/hivemq/data/data.lock
        at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source)
        at java.base/java.nio.channels.FileChannel.open(Unknown Source)
        at java.base/java.nio.channels.FileChannel.open(Unknown Source)
        at com.hivemq.HiveMQServer$DataLock.lock(HiveMQServer.java:311)
        at com.hivemq.HiveMQServer.bootstrap(HiveMQServer.java:140)
        at com.hivemq.HiveMQServer.start(HiveMQServer.java:253)
        at com.hivemq.HiveMQServer.main(HiveMQServer.java:97)
2025-02-06 14:24:32,682 INFO  - HiveMQ start was cancelled. An error occurred while opening the persistence. Is another HiveMQ instance running?

To Reproduce

With a running Azure Kubernetes service ressource, and connected via a terminal using kubectl

Steps

1. Create a namespace to test in and set it to default:

❯ kubectl create namespace hivemq
namespace/hivemq created

❯ kubectl config set-context --current --namespace=hivemq
Context "xyz" modified.

2. Deploy hivemq using the below statefulset manifest

❯ kubectl apply -f statefulset.yaml
statefulset.apps/test-hivemq created

3. Check that objects have been created

❯ kubectl get pvc,pv,volumeattachments,pod
NAME                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/testdata-test-hivemq-0   Bound    pvc-d2108ccb-19a1-44b6-bdb1-892f6c973d0d   50Gi       RWO            managed        <unset>                 7s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                 STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-d2108ccb-19a1-44b6-bdb1-892f6c973d0d   50Gi       RWO            Delete           Bound    hivemq/testdata-test-hivemq-0         managed        <unset>                          4s

NAME                                                                                                   ATTACHER             PV                                         NODE                              ATTACHED   AGE
volumeattachment.storage.k8s.io/csi-e4de756450f4786074ea7067fc34e90608620f8e58f73cc0150ff293aaa5b7ec   disk.csi.azure.com   pvc-d2108ccb-19a1-44b6-bdb1-892f6c973d0d   aks-default-31429679-vmss000000   false      4s

NAME                READY   STATUS              RESTARTS   AGE
pod/test-hivemq-0   0/1     ContainerCreating   0          7s

4. Once container is created, check the log

❯ k logs test-hivemq-0
-------------------------------------------------------------------------

                  _    _  _              __  __   ____
                 | |  | |(_)            |  \/  | / __ \
                 | |__| | _ __   __ ___ | \  / || |  | |
                 |  __  || |\ \ / // _ \| |\/| || |  | |
                 | |  | || | \ V /|  __/| |  | || |__| |
                 |_|  |_||_|  \_/  \___||_|  |_| \___\_\

-------------------------------------------------------------------------

  HiveMQ Start Script for Linux/Unix v1.14

-------------------------------------------------------------------------

  HIVEMQ_HOME: /opt/hivemq

  JAVA_OPTS: -XX:+UnlockExperimentalVMOptions -XX:+UseNUMA -Djava.net.preferIPv4Stack=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED -Djava.security.egd=file:/dev/./urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Duser.language=en -Duser.region=US -XX:+CrashOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError

  JAVA_VERSION: 21

-------------------------------------------------------------------------

2025-02-06 14:47:15,187 INFO  - Starting HiveMQ Community Edition Server
2025-02-06 14:47:15,188 INFO  - HiveMQ version: 2025.1
2025-02-06 14:47:15,188 INFO  - HiveMQ home directory: /opt/hivemq
2025-02-06 14:47:15,284 INFO  - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2025-02-06 14:47:15,470 ERROR - Could not open data lock file.
java.nio.file.AccessDeniedException: /opt/hivemq/data/data.lock
        at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source)
        at java.base/java.nio.channels.FileChannel.open(Unknown Source)
        at java.base/java.nio.channels.FileChannel.open(Unknown Source)
        at com.hivemq.HiveMQServer$DataLock.lock(HiveMQServer.java:311)
        at com.hivemq.HiveMQServer.bootstrap(HiveMQServer.java:140)
        at com.hivemq.HiveMQServer.start(HiveMQServer.java:253)
        at com.hivemq.HiveMQServer.main(HiveMQServer.java:97)
2025-02-06 14:47:15,471 INFO  - HiveMQ start was cancelled. An error occurred while opening the persistence. Is another HiveMQ instance running?

This issue is present for the following versions of HiveMQ CE

• 2025.1
• 2024.9
• 2024.8
• 2024.7
• 2024.6
• 2024.5
• 2024.4
• 2024.3

Which can be confirmed by changing the image tag in the manifest file and executing kubectl apply -f statefulset.yaml, then waiting for the image to be pulled and the new version spun up.

Once version 2024.2 has succesfully started once. The statefulset kan be editted to use any of the otherwise afflicted versions, and they will then start as expected, without error.

Reproducer code

statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-hivemq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-hivemq
  serviceName: test-hivemq
  template:
    metadata:
      labels:
        app: test-hivemq
    spec:
      containers:
        - name: test-hivemq
          image: hivemq/hivemq-ce:latest
          volumeMounts:
            - name: testdata
              mountPath: /opt/hivemq/data
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: testdata
      spec:
        accessModes:
        - ReadWriteOnce
        storageClassName: managed
        resources:
          requests:
            storage: 50Gi
        volumeMode: Filesystem

Details

• Affected HiveMQ CE version(s): 2024.3 - 2025.1
• Used JVM version: The one included in the official docker hub images