Make raw block pvc creation from volume snapshot idempotent

[RaunakShah]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Previously, the dd command run at https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/hostpath.go#L334 for raw block volumes was taking longer than the timeout for CreateVolume. As a result, the first call to CreateVolume fails with DeadlineExceeded.

Subsequent calls to CreateVolume were failing at https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/controllerserver.go#L138 because this volume was added to the local cache in the first call.

This PR makes creation of raw block PVC's from volume snapshots idempotent, so that retries for the same volume succeed.

Testing:

1. Create raw block PVC
2. Create volume snapshot
3. Create new PVC with volume snapshot as source

Both PVC's are in Bound state:

$ kubectl get  pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pvc-raw           Bound    pvc-e8b2e03e-e2be-462e-aea4-4c6eab52ff95   1Gi        RWO            csi-hostpath-sc   3m59s
raw-pvc-restore   Bound    pvc-9fd5af15-ceb6-4fd9-b6d9-6e131b54b55b   1Gi        RWO            csi-hostpath-sc   2m56s

$ kubectl get volumesnapshot
NAME               READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
raw-pvc-snapshot   true         pvc-raw                             1Gi           csi-hostpath-snapclass   snapcontent-d2bac861-3ba2-4839-a85e-b06674041cd4   3m27s          4m11s

Logs showing CreateVolume is called twice:

I1111 17:57:25.570725       1 server.go:117] GRPC call: /csi.v1.Controller/CreateVolume
I1111 17:57:25.570775       1 server.go:118] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.hostpath.csi/node":"csi-prow-worker"}}],"requisite":[{"segments":{"topology.hostpath.csi/node":"csi-prow-worker"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-9fd5af15-ceb6-4fd9-b6d9-6e131b54b55b","volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"53ff081f-2447-11eb-bed8-1ec29f23ce4b"}}}}
I1111 17:57:25.650608       1 volume_path_handler_linux.go:41] Creating device for path: /csi-data-dir/5b2b7374-2447-11eb-bed8-1ec29f23ce4b
I1111 17:57:25.668637       1 controllerserver.go:165] created volume 5b2b7374-2447-11eb-bed8-1ec29f23ce4b at path /csi-data-dir/5b2b7374-2447-11eb-bed8-1ec29f23ce4b
I1111 17:57:26.591090       1 server.go:117] GRPC call: /csi.v1.Controller/CreateVolume
I1111 17:57:26.591117       1 server.go:118] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.hostpath.csi/node":"csi-prow-worker"}}],"requisite":[{"segments":{"topology.hostpath.csi/node":"csi-prow-worker"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-9fd5af15-ceb6-4fd9-b6d9-6e131b54b55b","volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"53ff081f-2447-11eb-bed8-1ec29f23ce4b"}}}}
I1111 17:57:26.593464       1 server.go:123] GRPC response: {"volume":{"capacity_bytes":1073741824,"content_source":{"Type":{"Snapshot":{"snapshot_id":"53ff081f-2447-11eb-bed8-1ec29f23ce4b"}}},"volume_id":"5b2b7374-2447-11eb-bed8-1ec29f23ce4b"}}

Get volumeHandle of PV's:

$ kubectl describe pv  | grep -i handle
    VolumeHandle:      5b2b7374-2447-11eb-bed8-1ec29f23ce4b
    VolumeHandle:      35df4fe4-2447-11eb-bed8-1ec29f23ce4b

Exec into the hostpath driver and verify that only 3 volumes exist (2 with PV volume handles):

/ # ls /csi-data-dir/
35df4fe4-2447-11eb-bed8-1ec29f23ce4b       53ff081f-2447-11eb-bed8-1ec29f23ce4b.snap  5b2b7374-2447-11eb-bed8-1ec29f23ce4b

Verify that only two volumes are attached:

/ # losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                          DIO LOG-SEC
/dev/loop1         0      0         0  0 /csi-data-dir/5b2b7374-2447-11eb-bed8-1ec29f23ce4b   0     512
/dev/loop0         0      0         0  0 /csi-data-dir/35df4fe4-2447-11eb-bed8-1ec29f23ce4b   0     512

Which issue(s) this PR fixes:

Fixes #219

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Make raw block PVC creation from VolumeSnapshot idempotent.

[xing-yang]

/assign

[xing-yang]

CC @prafull01

[xing-yang]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RaunakShah, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment