[openstack cinder] in-tree e2e failed with fsType = ""

[jichenjc]

What happened:

in order to fix kubernetes/cloud-provider-openstack#1362
I submitted kubernetes/cloud-provider-openstack#1366

however the case failed with in-tree test
https://logs.openlabtesting.org/logs/66/1366/6bd52d8f09cf9895e1baed6100f85daab3fef48f/cloud-provider-openstack-multinode-csi-migration-test/cloud-provider-openstack-multinode-csi-migration-test/58c19df/job-output.txt.gz

2021-01-14 06:42:42.106936 | k8s-master | [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
2021-01-14 06:42:42.107021 | k8s-master |   (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
2021-01-14 06:42:42.107106 | k8s-master |   /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:206
2021-01-14 06:42:42.107176 | k8s-master | [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
2021-01-14 06:42:42.107202 | k8s-master |   /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:130
2021-01-14 06:42:42.107271 | k8s-master | [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
2021-01-14 06:42:42.107298 | k8s-master |   /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:81
2021-01-14 06:42:42.107359 | k8s-master | [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
2021-01-14 06:42:42.107386 | k8s-master |   /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:174
2021-01-14 06:42:42.107443 | k8s-master | STEP: Creating a kubernetes client
2021-01-14 06:42:42.107509 | k8s-master | Jan 14 06:42:42.107: INFO: >>> kubeConfig: /etc/kubernetes/admin.conf
2021-01-14 06:42:42.108885 | k8s-master | STEP: Building a namespace api object, basename fsgroupchangepolicy
2021-01-14 06:42:42.134714 | k8s-master | STEP: Waiting for a default service account to be provisioned in namespace
2021-01-14 06:42:42.137612 | k8s-master | [It] (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
2021-01-14 06:42:42.137691 | k8s-master |   /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:206
2021-01-14 06:42:42.137808 | k8s-master | Jan 14 06:42:42.137: INFO: Creating resource for dynamic PV
2021-01-14 06:42:42.137968 | k8s-master | Jan 14 06:42:42.137: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(cinder) supported size:{ 1Mi}
2021-01-14 06:42:42.138096 | k8s-master | STEP: creating a StorageClass fsgroupchangepolicy-7600-cinder-sctqpgg
2021-01-14 06:42:42.141583 | k8s-master | STEP: creating a claim
2021-01-14 06:42:42.141713 | k8s-master | Jan 14 06:42:42.141: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
2021-01-14 06:42:42.146718 | k8s-master | Jan 14 06:42:42.146: INFO: Waiting up to 5m0s for PersistentVolumeClaims [cindervfrwd] to have phase Bound
2021-01-14 06:42:42.149105 | k8s-master | Jan 14 06:42:42.149: INFO: PersistentVolumeClaim cindervfrwd found but phase is Pending instead of Bound.
2021-01-14 06:42:44.155462 | k8s-master | Jan 14 06:42:44.155: INFO: PersistentVolumeClaim cindervfrwd found and phase=Bound (2.008454441s)
2021-01-14 06:42:44.162532 | k8s-master | STEP: Creating Pod in namespace fsgroupchangepolicy-7600 with fsgroup 1000
2021-01-14 06:42:54.296376 | k8s-master | Jan 14 06:42:54.296: INFO: Pod fsgroupchangepolicy-7600/pod-2b89220c-2ab3-4b4f-a910-c5b430fe8735 started successfully
2021-01-14 06:42:54.296568 | k8s-master | STEP: Creating a sub-directory and file, and verifying their ownership is 1000
2021-01-14 06:42:54.296677 | k8s-master | Jan 14 06:42:54.296: INFO: ExecWithOptions {Command:[/bin/sh -c touch /mnt/volume1/file1] Namespace:fsgroupchangepolicy-7600 PodName:pod-2b89220c-2ab3-4b4f-a910-c5b430fe8735 ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
2021-01-14 06:42:54.296734 | k8s-master | Jan 14 06:42:54.296: INFO: >>> kubeConfig: /etc/kubernetes/admin.conf
2021-01-14 06:42:54.469244 | k8s-master | Jan 14 06:42:54.469: INFO: ExecWithOptions {Command:[/bin/sh -c ls -l /mnt/volume1/file1] Namespace:fsgroupchangepolicy-7600 PodName:pod-2b89220c-2ab3-4b4f-a910-c5b430fe8735 ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
2021-01-14 06:42:54.469394 | k8s-master | Jan 14 06:42:54.469: INFO: >>> kubeConfig: /etc/kubernetes/admin.conf
2021-01-14 06:42:54.627771 | k8s-master | Jan 14 06:42:54.627: INFO: pod fsgroupchangepolicy-7600/pod-2b89220c-2ab3-4b4f-a910-c5b430fe8735 exec for cmd ls -l /mnt/volume1/file1, stdout: -rw-r--r--    1 root     root             0 Jan 14 06:42 /mnt/volume1/file1, stderr:
2021-01-14 06:42:54.628054 | k8s-master | Jan 14 06:42:54.627: INFO: stdout split: [-rw-r--r-- 1 root root 0 Jan 14 06:42 /mnt/volume1/file1], expected gid: 1000
2021-01-14 06:42:54.628469 | k8s-master | Jan 14 06:42:54.628: FAIL: Expected
2021-01-14 06:42:54.628510 | k8s-master |     <string>: root
2021-01-14 06:42:54.628535 | k8s-master | to equal
2021-01-14 06:42:54.628557 | k8s-master |     <string>: 1000
2021-01-14 06:42:54.628579 | k8s-master |

suspect it because the CSIMigration will use the new external provisioner (bump to v2.1.0) and without fsType it triggered the issue in kubernetes/cloud-provider-openstack#1362
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[k8s-ci-robot]

@jichenjc: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jichenjc]

/assign

[neolit123]

/sig cloud-provider storage

[jichenjc]

@ramineni can you help to provide some comments on this?

[ramineni]

@jichenjc I suppose providing default-fstype in provisioner did solve the issue, so we can close this issue?

[jichenjc]

yes, that's exactly why I hold and close 98053 , thanks :)