Invalid volume permissions with CSI driver with kube-prometheus-stack

[ams0]

What happened:
Using kube-prometheus-stack chart with an CSI azure-disk managed disk doesn't work due to permission issues. The chart sets as defaults:

    securityContext:
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
      fsGroup: 2000

using a PV of type disk.csi.azure.com with the chart will lead to this error:

level=error ts=2021-08-06T23:46:59.473Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffc0f43fb76, 0xb, 0x14, 0x37603a0, 0xc000135720, 0x37603a0)
	/app/promql/query_logger.go:117 +0x4e8
main.main()
	/app/cmd/prometheus/main.go:479 +0x5aad

Setting the pod to run as root fixes the problem but it's not ideal:

          prometheusSpec:
            securityContext:
              runAsGroup: 0
              runAsNonRoot: false
              runAsUser: 0
              fsGroup: 0

What you expected to happen:

Prometheus running fine on managed disks with CSI drivers

How to reproduce it:

Anything else we need to know?:

Linked from prometheus-community/helm-charts#1189

Environment:

• CSI Driver version: 1.5.0
• Kubernetes version (use kubectl version): 1.21.2
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[andyzhangx]

would you follow guide here to workaround this issue?
#901 (comment)

[ams0]

Thanks, recreating the CSIDriver object worked. I forgot to say that I'm using a statically provisioned disk and just a PV referencing it, not a storageClass, and the PV definition already had fsType: ext4:

  csi:
    driver: disk.csi.azure.com
    volumeAttributes:
      fsType: ext4
    volumeHandle: /subscriptions/<subid>/resourceGroups/<rg>/providers/Microsoft.Compute/disks/prometheus-metrics
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

On 1.20.7, this did the trick for me. Thanks!