-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ceph-csi pool parameter not passed correctly #8550
Comments
@tgross -> I tried to include enough spec to get a complete picture. let me know if more required. |
Thanks @ryanmickler. I'll take a look into this later in the week. |
I believe i've found the problem. First, here: Line 95 in 8f98ff2
parameters and context maps are part of the structure. in the hcl, I am passing in pool as a parameter .
Next, here:
context map is passed through to NodeStageVolumeRequest as VolumeContext , and the parameters block is ignored.
(Its not clear to me what should be a parameter, and what should be in And in the bug you fixed here: 901664f, context should now be passed properly into This was patched in https://github.com/hashicorp/nomad/releases/tag/v0.12.0-beta2.
|
Update: well, pushing to Nomad 0.12.1 and passing Now the issues I'm having are related to how strictly I'd be happy to close this, but there's probably some amount of documentation to preserve here as others are likely to hit this error. |
Darn, I didn't dig into this more quickly because I would have caught the version where we'd fixed that. Sorry about that.
Yeah, we don't currently have a great way of documenting the details of what each CSI plugin expects for inputs, and I'd worry about getting stale with what the upstream projects are doing as well. There's clearly some Nomad-specific documentation bits to be had that don't quite belong in Nomad docs but not anywhere else either. There's an I'm going to close this issue for now but rest assured we're tracking a discussion about what we can do around documentation improvements. |
@tgross I love the idea of having an Examples directory containing CSI deployment specs. If this could be something that could be contributed to by the public I would be happy to send over the Openstack spec we have along with any others we may use. |
@rigrassm do you mean using the Cinder CSI plugin https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md#csi-cinder-driver |
@ryanmickler yup, that's the one. It's currently broken until 0.12.2 is released but I've tested it with the patch that fixes it and it works great. |
Perhaps we could get started on a branch where we put our example configs together? |
Sure, I would happy to contribute to that! I have one created for the Cinder CSI Driver already and did some work getting the iscsi CSI driver working which I suspect will work after the 0.12.2 release. I'd like to hear from @tgross about the internal discussion re: the structure and location of these examples so that we can get this going as smoothly as possible. It's a good amount of work reverse engineering these CSI drivers to be deployed on Nomad (or any orchestrator not named K8's lol) so having documented examples of their configs coming in from the community would be fantastic. |
Absolutely. Lets wait for feedback. PS. I also use nomad on openstack, but ive been needing to use CephFS, which I dont think cinder can support. For that, we'll need to do that Manila CSI plugin (https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-manila-csi-plugin.md) which i assume your config will help with. |
Thanks folks! I've opened #8651 to add a directory at |
@tgross Awesome, appreciate it! @ryanmickler I Just opened #8662 adding an example for the Cinder CSI driver if you wanted to take a look at it. |
Great, i just added #8664 This should get people started. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
11.3
Operating system and Environment details
os:
ubuntu bionic
ceph-csi:
quay.io/cephcsi/cephcsi:v2.1.2-amd64
Issue
csi volume container parameter
pool
apparently not passed to ceph-csi node during volume mount. recievingmissing required parameter pool
Potentially problem related to
NodeStageVolume
in CSI spec.A quick inspection looks like
NodeStageVolume
(mounts the volume to a staging path on the node.)https://github.com/ceph/ceph-csi/blob/47d5b60af8d48574ff6d11ca37dbff5a6f56815b/internal/rbd/nodeserver.go#L116
is calling
genVolFromVolumeOptions
on line 171https://github.com/ceph/ceph-csi/blob/47d5b60af8d48574ff6d11ca37dbff5a6f56815b/internal/rbd/nodeserver.go#L171
Then inside that function, we are hitting our
missing required parameter pool
error here:https://github.com/ceph/ceph-csi/blob/be9e7cf956c378227ff43e0194410468919766b7/internal/rbd/rbd_util.go#L694
Reproduction steps
Job file
csi-test-job.hcl
using the terraform nomad_volume provider (had tried using just raw hcl upload to nomad cli, no change)
ceph-csi-volume.tf
ceph-csi-plugin-controller-job.hcl
ceph-csi-node.hcl
Nomad Client logs
from ceph-csi node container:
E0728 04:47:20.003465 1 utils.go:163] ID: 23 Req-ID: csi-test-0 GRPC error: rpc error: code = Internal desc = missing required parameter pool
split off from #8212
this specific error first mentioned here #7668 (comment)
The text was updated successfully, but these errors were encountered: