Clean handling of driver restart

[okartau]

What is the state of provisioned volumes after driver restart?
Lvolumes may exist, and may be mounted or remain unmounted.
Host maintains that state, but driver may restart.
As driver does not have permanently stored volumes state,
it does not have context after restart, and that means trouble for next operations.
All existing volumes or mounts are orphaned in context of driver.
Driver should try to reach the same, or at least some reasonable state about existing volumes and mounts.
Here, similar to #11, it is also possible that some Lvolumes exist in host context which were not created by PMEM-CSI driver, so driver needs to recognize own vs foreign volumes as well.

[okartau]

I propose to close this. It does not contain a clear objective to solve. Driver anyway can not decide about the state of volumes, it executes orders from above, and is operating on top of device(s) state.

[pohly]

Let's keep this open. One possible test scenario is:

• create a PVC
• create an app that uses the PVC
• restart the cluster
• wait for app to restart
• delete app
• delete PVC

[okartau]

if, after restart, driver receives node:{stage-publish-unpublish-unstage} sequence for existing VolumeID, the resulting action is good, as Node operates on devices doing thes operations, without keeping in-memory context.
But Controller-Create/Delete are problematic after restart, as Controller operates on in-memory registry of Volumes which is not re-created after restart. Means, when receiving a next create with existing volume name, it will attempt new creation, and when receiving Delete, it responds with "not created by this controller"

[okartau]

Restoring complete state after driver restart is interesting topic.
Controller in-memory registry restore is not currently implemented, and would require some effort, as
Controller either needs to store it's registry in NV storage (complex as it's not fixed defined where it runs), or needs to recreate volumes information by collecting information from Nodes. That's probably doable, but poses some challenges, and some bits of data now in registry, are not stored in namespace information, like erasemode (added by us in this driver). Erasemode starts to look like troublemaker, would it be simpler just to forget about it and always wipe all data after volume deletion.

[avalluri]

I had in my mind that persisting driver state to a file on node, may be its the time to consider this.

[pohly]

Amarnath Valluri <[email protected]> writes:

I had in my mind that persisting driver state to a file on node,

That can't work for a pod that can move between nodes.

may be its the time to consider this.

ceph-csi had the same problem. They solved it by storing data in a confimap, see ceph/ceph-csi#113

[avalluri]

Yes i agree, the only thing in my mind is how far a csi driver can have Kubernetes dependency, in that case we could avoid RegistryServer itself completely :)

[pohly]

Amarnath Valluri <[email protected]> writes:

the only thing in my mind is how far a csi driver can have Kubernetes dependency,

ceph-csi made that dependency optional by using local files by default and a configmap only when explicitly enabled.

in that case we could avoid RegistryServer itself completely :)

True.

[okartau]

I made a "node reboot" test in running 4xVM cluster with Clear=28030 and Kubernetes 1.13.3.
Deployment is made according to README example steps (Devicemode:direct).
With my-csi-app running and using provisioned volume on node-1, I rebooted that node.
node-1 status changed to Error for some time, then back to Running.
Restarted pmem-csi on node-1 received NodeStageVolume, and NodePublishVolume,
then application reached the state using previously existing files system.

This happens because in NodeStageVolume (where mkfs is done for newly-created volume),
there is check for existing file system. If the existing file system type is same as requested,
mkfs is skipped.
(if fstype is different then error is returned, and only if there is no file system, we perform mkfs)

[okartau]

same result when rebooting cluster master, looking good

[okartau]

same result when forcibly stopping, starting entire cluster (make stop, make start)