problem with multi node multi writer block device attaching ceph csi

[thatsk]

Nomad version

1.3.1

Operating system and Environment details

centos 7

Issue

trying to attach policy multi-node-multi-writer. for csi volume in ceph

Reproduction steps

test-volume.hcl

id = "testmysql[0]"
name = "testmysql[0]"
type = "csi"
plugin_id = "ceph-csi"
capability {
  access_mode     = "multi-node-multi-writer"
  attachment_mode = "block-device"
}
mount_options {}
secrets {
  userID  = "test"
  userKey = "AQC19ddi7WnlKBAALrO9uNWQAe3Ss4yGJr/wAA=="
}

parameters {
  clusterID = "11c21580-0430-11ed-b181-0050568ee646"
  pool = "myPool"
  imageFeatures = "layering"
}

Expected Result

it should mount to MySQL job.

Actual Result

Job file (if appropriate)

Nomad mysql job.

job "mysql-server" {
  datacenters = ["test"]
  type        = "service"
  constraint {
    attribute = "${node.unique.name}"
    operator  = "regexp"
    value     = "test"
  }
  group "mysql-server" {
    count = 1
    volume "ceph-mysql" {
      type      = "csi"
      access_mode     = "multi-node-multi-writer"
      attachment_mode = "block-device"
      read_only = false
      source    = "testmysql[0]"
    }
    network {
      port "db" {
        static = 3306
      }
    }
    restart {
      attempts = 10
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }
    task "mysql-server" {
      driver = "docker"
      volume_mount {
        volume      = "ceph-mysql"
        destination = "/srv"
        read_only   = false
      }
      env {
        MYSQL_ROOT_PASSWORD = "password"
      }
      config {
        image = "hashicorp/mysql-portworx-demo:latest"
        args  = ["--datadir", "/srv/mysql"]
        ports = ["db"]
      }
      resources {
        cpu    = 100
        memory = 1024
      }
      service {
        name = "mysql-server"
        port = "db"
        check {
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

[Amier3]

Hey @thatsk

Thanks for making this bug report, we'll take a look at this shortly. If you haven't seen it already, please read the comment I made on your other bug report

[tgross]

Hi @thatsk I tried to come up with a reproduction for this and wasn't able to; I'm not much of a Ceph expert and my Ceph isn't letting me create block device volumes.

That being said, it looks like this error is bubbling up from the mount syscall that runc is making, so it's way at the bottom of the stack in how runc is treating the block device. Do you still see this problem if you mount the volume with attachment_mode = "file-system"?

[tgross]

Haven't heard back on this one and there's not enough information for us to reproduce (and it feels like a Ceph config issue?). If you've got more information, we'll be happy to reopen. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.