Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csi: release claims via csi_hook postrun unpublish RPC #8580

Merged
merged 4 commits into from
Aug 6, 2020
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
csi: release claims via csi_hook postrun unpublish RPC
Add a Postrun hook to send the `CSIVolume.Unpublish` RPC to the server. This
may forward client RPCs to the node plugins or to the controller plugins,
depending on whether other allocations on this node have claims on this
volume.
  • Loading branch information
tgross committed Aug 6, 2020
commit bc62c745808c2efd207dc651977cb55e334225de
46 changes: 40 additions & 6 deletions client/allocrunner/csi_hook.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"

hclog "github.com/hashicorp/go-hclog"
multierror "github.com/hashicorp/go-multierror"
"github.com/hashicorp/nomad/client/pluginmanager/csimanager"
"github.com/hashicorp/nomad/nomad/structs"
"github.com/hashicorp/nomad/plugins/drivers"
Expand All @@ -15,12 +16,13 @@ import (
//
// It is a noop for allocs that do not depend on CSI Volumes.
type csiHook struct {
ar *allocRunner
alloc *structs.Allocation
logger hclog.Logger
csimanager csimanager.Manager
rpcClient RPCer
updater hookResourceSetter
ar *allocRunner
alloc *structs.Allocation
logger hclog.Logger
csimanager csimanager.Manager
rpcClient RPCer
updater hookResourceSetter
volumeRequests map[string]*volumeAndRequest
}

func (c *csiHook) Name() string {
Expand All @@ -43,6 +45,7 @@ func (c *csiHook) Prerun() error {
if err != nil {
return fmt.Errorf("claim volumes: %v", err)
}
c.volumeRequests = volumes

mounts := make(map[string]*csimanager.MountInfo, len(volumes))
for alias, pair := range volumes {
Expand Down Expand Up @@ -73,6 +76,37 @@ func (c *csiHook) Prerun() error {
return nil
}

// Postrun sends an RPC to the server to unpublish the volume. This may
// forward client RPCs to the node plugins or to the controller plugins,
// depending on whether other allocations on this node have claims on this
// volume.
func (c *csiHook) Postrun() error {
if !c.shouldRun() {
return nil
}

var mErr *multierror.Error

for _, pair := range c.volumeRequests {
req := &structs.CSIVolumeUnpublishRequest{
VolumeID: pair.request.Source,
Claim: &structs.CSIVolumeClaim{
AllocationID: c.alloc.ID,
NodeID: c.alloc.NodeID,
Mode: structs.CSIVolumeClaimRelease,
},
WriteRequest: structs.WriteRequest{
Region: c.alloc.Job.Region, Namespace: c.alloc.Job.Namespace},
}
err := c.rpcClient.RPC("CSIVolume.Unpublish",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of side discussion: retries here are optional, and at best might speed up some cases of releasing the claim. A failure here will be retried in volume GC, when the terminal (or already GC'd nil) alloc is observed in the volume & claims. So this is cool

req, &structs.CSIVolumeUnpublishResponse{})
if err != nil {
mErr = multierror.Append(mErr, err)
}
}
return mErr.ErrorOrNil()
}

type volumeAndRequest struct {
volume *structs.CSIVolume
request *structs.VolumeRequest
Expand Down