Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filesystem is not resized when restoring from snapshot/cloning to larger size than origin #972

Merged
merged 3 commits into from
Oct 25, 2022
Merged

filesystem is not resized when restoring from snapshot/cloning to larger size than origin #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
febd972
use mount-utils for resizing volumes
RomanBednar Oct 5, 2022
80c800d
fix NodeStageVolume tests
RomanBednar Apr 29, 2022
41b3dfd
disable pre-resize check of fs size
RomanBednar Sep 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions pkg/gce-pd-csi-driver/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,15 @@ func (ns *GCENodeServer) NodeStageVolume(ctx context.Context, req *csi.NodeStage
devicePath, stagingTargetPath, fstype, options, err))
}

// Part 4: Resize filesystem.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a reason why we have this call here in addition to the NodeExpandVolume resize call, I think we can trace it back through blame too though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a link to the original issue in k/k where the reasons are explained.

// https://github.com/kubernetes/kubernetes/issues/94929
resizer := resizefs.NewResizeFs(ns.Mounter)
_, err = resizer.Resize(devicePath, stagingTargetPath)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Windows this is https://github.com/kubernetes-csi/csi-proxy/blob/202a4e9cfa2c6277920c95f5325c9568bc520531/pkg/os/volume/api.go#L153 with size=0 which will resize the volume to occupy the entire disk, I think this should work in Windows too.

Do you know if we have a test for this scenario (restore from a snapshot to a larger volume size) in the storage e2e tests in k8s? I checked in https://testgrid.k8s.io/provider-gcp-compute-persistent-disk-csi-driver#ci-windows-2019-provider-gcp-compute-persistent-disk-csi-driver that we're not testing resizing in Windows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I have this PR open which is currently waiting for this PR to merge because the test would fail in CI for (any) driver that does not resize properly. To allow drivers to opt out from the test we've added the anti-capability.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I added a comment in that other PR, in the worst case we might need to add a skip for Windows or find a way to run the stat command in Windows too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you. Are we good for approval now @mattcary?

if err != nil {
return nil, status.Error(codes.Internal, fmt.Sprintf("error when resizing volume %s: %v", volumeID, err))

}

klog.V(4).Infof("NodeStageVolume succeeded on %v to %s", volumeID, stagingTargetPath)
return &csi.NodeStageVolumeResponse{}, nil
}
Expand Down
69 changes: 67 additions & 2 deletions pkg/gce-pd-csi-driver/node_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ package gceGCEDriver

import (
"context"
"fmt"
"io/ioutil"
"k8s.io/utils/exec"
testingexec "k8s.io/utils/exec/testing"
"os"
"path/filepath"
"testing"
Expand Down Expand Up @@ -63,6 +66,15 @@ func getTestBlockingGCEDriver(t *testing.T, readyToExecute chan chan struct{}) *
return gceDriver
}

func makeFakeCmd(fakeCmd *testingexec.FakeCmd, cmd string, args ...string) testingexec.FakeCommandAction {
c := cmd
a := args
return func(cmd string, args ...string) exec.Cmd {
command := testingexec.InitFakeCmd(fakeCmd, c, a...)
return command
}
}

func TestNodeGetVolumeStats(t *testing.T) {
gceDriver := getTestGCEDriver(t)
ns := gceDriver.ns
Expand Down Expand Up @@ -351,8 +363,6 @@ func TestNodeUnpublishVolume(t *testing.T) {
}

func TestNodeStageVolume(t *testing.T) {
gceDriver := getTestGCEDriver(t)
ns := gceDriver.ns
volumeID := "project/test001/zones/c1/disks/testDisk"
blockCap := &csi.VolumeCapability_Block{
Block: &csi.VolumeCapability_BlockVolume{},
Expand Down Expand Up @@ -436,6 +446,61 @@ func TestNodeStageVolume(t *testing.T) {
}
for _, tc := range testCases {
t.Logf("Test case: %s", tc.name)
actionList := []testingexec.FakeCommandAction{
makeFakeCmd(
&testingexec.FakeCmd{
CombinedOutputScript: []testingexec.FakeAction{
func() ([]byte, []byte, error) {
return []byte(fmt.Sprintf("DEVNAME=/dev/sdb\nTYPE=ext4")), nil, nil
},
},
},
"blkid",
),
makeFakeCmd(
&testingexec.FakeCmd{
CombinedOutputScript: []testingexec.FakeAction{
func() ([]byte, []byte, error) {
return []byte("1"), nil, nil
},
},
},
"blockdev",
),
makeFakeCmd(
&testingexec.FakeCmd{
CombinedOutputScript: []testingexec.FakeAction{
func() ([]byte, []byte, error) {
return []byte("1"), nil, nil
},
},
},
"blockdev",
),
makeFakeCmd(
&testingexec.FakeCmd{
CombinedOutputScript: []testingexec.FakeAction{
func() ([]byte, []byte, error) {
return []byte(fmt.Sprintf("DEVNAME=/dev/sdb\nTYPE=ext4")), nil, nil
},
},
},
"blkid",
),
makeFakeCmd(
&testingexec.FakeCmd{
CombinedOutputScript: []testingexec.FakeAction{
func() ([]byte, []byte, error) {
return []byte(fmt.Sprintf("block size: 1\nblock count: 1")), nil, nil
},
},
},
"dumpe2fs",
),
}
mounter := mountmanager.NewFakeSafeMounterWithCustomExec(&testingexec.FakeExec{CommandScript: actionList})
gceDriver := getTestGCEDriverWithCustomMounter(t, mounter)
ns := gceDriver.ns
_, err := ns.NodeStageVolume(context.Background(), tc.req)
if err != nil {
serverError, ok := status.FromError(err)
Expand Down
58 changes: 15 additions & 43 deletions pkg/resizefs/resizefs_linux.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ limitations under the License.
package resizefs

import (
"fmt"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"

"k8s.io/klog"
"k8s.io/mount-utils"
Expand All @@ -38,51 +39,22 @@ func NewResizeFs(mounter *mount.SafeFormatAndMount) *resizeFs {
}

// Resize perform resize of file system
func (resizefs *resizeFs) Resize(devicePath, deviceMountPath string) (bool, error) {
format, err := resizefs.mounter.GetDiskFormat(devicePath)
func (resizefs *resizeFs) Resize(devicePath, deviceMountPath string) (needResize bool, err error) {
resizer := mount.NewResizeFs(resizefs.mounter.Exec)

if err != nil {
formatErr := fmt.Errorf("ResizeFS.Resize - error checking format for device %s: %v", devicePath, err)
return false, formatErr
klog.V(4).Infof("Checking if filesystem needs to be resized. Device: %s Mountpoint: %s", devicePath, deviceMountPath)
if needResize, err = resizer.NeedResize(devicePath, deviceMountPath); err != nil {
err = status.Errorf(codes.Internal, "Could not determine if filesystem %q needs to be resized: %v", deviceMountPath, err)
return
}

// If disk has no format, there is no need to resize the disk because mkfs.*
// by default will use whole disk anyways.
if format == "" {
return false, nil
if needResize {
klog.V(4).Infof("Resizing filesystem. Device: %s Mountpoint: %s", devicePath, deviceMountPath)
if _, err = resizer.Resize(devicePath, deviceMountPath); err != nil {
err = status.Errorf(codes.Internal, "Failed to resize filesystem %q: %v", deviceMountPath, err)
return
}
}

klog.V(3).Infof("ResizeFS.Resize - Expanding mounted volume %s", devicePath)
switch format {
case "ext3", "ext4":
return resizefs.extResize(devicePath)
case "xfs":
return resizefs.xfsResize(deviceMountPath)
}
return false, fmt.Errorf("ResizeFS.Resize - resize of format %s is not supported for device %s mounted at %s", format, devicePath, deviceMountPath)
}

func (resizefs *resizeFs) extResize(devicePath string) (bool, error) {
output, err := resizefs.mounter.Exec.Command("resize2fs", devicePath).CombinedOutput()
if err == nil {
klog.V(2).Infof("Device %s resized successfully", devicePath)
return true, nil
}

resizeError := fmt.Errorf("resize of device %s failed: %v. resize2fs output: %s", devicePath, err, string(output))
return false, resizeError

}

func (resizefs *resizeFs) xfsResize(deviceMountPath string) (bool, error) {
args := []string{"-d", deviceMountPath}
output, err := resizefs.mounter.Exec.Command("xfs_growfs", args...).CombinedOutput()

if err == nil {
klog.V(2).Infof("Device %s resized successfully", deviceMountPath)
return true, nil
}

resizeError := fmt.Errorf("resize of device %s failed: %v. xfs_growfs output: %s", deviceMountPath, err, string(output))
return false, resizeError
return
}
7 changes: 1 addition & 6 deletions test/e2e/tests/resize_e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,17 +232,12 @@ var _ = Describe("GCE PD CSI Driver", func() {
}
}()

// Verify pre-resize fs size
sizeGb, err := testutils.GetFSSizeInGb(instance, publishDir)
Expect(err).To(BeNil(), "Failed to get FSSize in GB")
Expect(sizeGb).To(Equal(defaultSizeGb))

// Resize node
_, err = client.NodeExpandVolume(volID, publishDir, newSizeGb)
Expect(err).To(BeNil(), "Node expand volume failed")

// Verify disk size
sizeGb, err = testutils.GetFSSizeInGb(instance, publishDir)
sizeGb, err := testutils.GetFSSizeInGb(instance, publishDir)
Expect(err).To(BeNil(), "Failed to get FSSize in GB")
Expect(sizeGb).To(Equal(newSizeGb))

Expand Down