[FEATURE] Host local storage support - Experimental #5724

bk201 · 2024-05-02T10:09:41Z

Is your feature request related to a problem? Please describe.

Applications require large volume or high-performance local disks. Live-migration is not a concern.

Describe the solution you'd like

As a HCI solution, we can have a build-in local storage support like leveraging LVM or files.
We have experimented lvm-csi-driver in a POC environment.

Describe alternatives you've considered

Additional context

SlavikCA · 2024-06-11T02:54:18Z

I'm curious: how that feature will be different from simply using hostPath?

Is correct to say, that hostPath is file-level access and this feature will provide block-level access?

Or is it going to be file-level access, but the disc (volume group?) will be managed by the LVM CSI driver, so it will provide features such mirroring, striping, etc...?

Vicente-Cheng · 2024-06-12T10:17:07Z

I'm curious: how that feature will be different from simply using hostPath?

Is correct to say, that hostPath is file-level access and this feature will provide block-level access?

Or is it going to be file-level access, but the disc (volume group?) will be managed by the LVM CSI driver, so it will provide features such mirroring, striping, etc...?

Hi @SlavikCA,

This feature will provide the block device.
With LVM, we can easily use various LVM types like mirror and stripe. Also, we would benefit from various LVM features like expansion/snapshot.

abonillabeeche · 2024-06-19T07:51:57Z

FYI - I had the LVM CSI driver installed on 1.3.0 and upgraded to 1.3.1 without an issue. Perhaps worth investigating is the ability to have snapshots and backup, as it is not available out of the box.

harvesterhci-io-github-bot · 2024-06-24T08:55:19Z

GUI issue created #6057.

abonillabeeche · 2024-07-15T15:30:57Z

Or is it going to be file-level access, but the disc (volume group?) will be managed by the LVM CSI driver, so it will provide features such mirroring, striping, etc...?

Correct. The LVM CSI provides stripping which could be used for large volumes.

albinsun · 2024-07-18T07:09:11Z

Hi Guys,
Per discussed in HEP PR, Longhorn plans to support LVM-based Local Volume in v1.8 (Nov. 2024) or v1.9 (Mar. 2025) and we will use LH solution for local storage that time.

If so, why we need to make this feature by ourself now as LH will support it?
Or they are not completely overlapped?

Ref. https://github.com/longhorn/longhorn/wiki/Roadmap#longhorn-v19-march-2025

innobead · 2024-07-18T10:47:32Z

Hi Guys, Per discussed in HEP PR, Longhorn plans to support LVM-based Local Volume in v1.8 (Nov. 2024) or v1.9 (Mar. 2025) and we will use LH solution for local storage that time.

If so, why we need to make this feature by ourself now as LH will support it? Or they are not completely overlapped?

Ref. Wiki: Roadmap (longhorn v19 march 2025) (longhorn/longhorn)

@albinsun it's all about the priority and good question.

For Longhorn, local volume via LVM is a relatively low priority compared to ongoing v2 development. However, for Harvester, local volume support is an emerging requirement for data-intensive workloads in some cases. This is why we decided to develop an addon first to support an LVM storage driver with limited functions, allowing users to experience this and maximize the capacity of local disks before we implement complete functions such as snapshot, backup, trim, expansion, and encryption, which Longhorn supports. (I know @Vicente-Cheng is also experimenting with snapshot and expansion in the current development, which is good and okay.)

Whether we migrate to the version supported by Longhorn in the future will depend on how it will be used and adopted in Harvester first, and what features are exclusively supported by Longhorn.

cc @Vicente-Cheng @derekbit

harvesterhci-io-github-bot · 2024-09-10T06:51:29Z

Pre Ready-For-Testing Checklist

If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
The HEP PR is at: HEP: introduce the LVM CSI driver #5956
Where is the reproduce steps/test steps documented?
The reproduce steps/test steps are at:

Test Plan:

Enable CSI driver

create new addon with this PR: addons: add harvester-csi-driver-lvm experimental-addons#18
enable addon

Create Storage provisioner

Create Storage for lvm volumegroup (entry point: host//storage)
Select the disk -> select provisioner LVM -> input the vgName -> save

Create StorageClass

select provisioner LVM (we need to convert this to LVM on UI side)
select node -> select vgName -> save
select volume and use it as data volume for the VM.
NOTE:
If the VM is already running, it can only attach the LVM Volume within the same node.
If the VM is stopped, it will select the corresponding node of the LVM volume when VM starting

Is there a workaround for the issue? If so, where is it documented?
The workaround is at:

None

Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
The PR is at:
LVM Addon: addons: add harvester-csi-driver-lvm experimental-addons#18
Node-disk-Manager: Add lvm provisioner node-disk-manager#128
CSI driver: https://github.com/harvester/csi-driver-lvm <-- the whole repo is new introduced
- Does the PR include the explanation for the fix or the feature?
- Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:
If labeled: area/ui Has the UI issue filed or ready to be merged?
The UI issue/PR is at: Host local storage support dashboard#1076
If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
The documentation/KB PR is at: TBD

* [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
- The automation skeleton PR is at:
- The automation test case PR is at:

* [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
The compatibility issue is filed at:

harvesterhci-io-github-bot · 2024-09-10T06:51:30Z

Automation e2e test issue: harvester/tests#1515

TachunLin · 2024-10-22T14:29:01Z

Verifying on v1.4.0-rc3

Result

$\color{green}{\textsf{PASS}}$ Single node: Use lvm-csi-driver storage class volume on VM $~~$

After created the harvester-csi-driver-lvm, can enable the harvester-csi-driver-lvm on addons page

Can create Storage for LVM volume group on Host
Can create Storage Class for LVM provisioner

Check the storage class yaml details contains the allowedTopologies on specific node

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: false
allowedTopologies:
- matchLabelExpressions:
  - key: topology.lvm.csi/node
    values:
    - hp-164-seeder
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2024-10-22T13:38:55Z"
  name: lvm-sc
  resourceVersion: "3969422"
  uid: 12cd92ba-33bb-4e3e-bbb3-44115140a9d7
parameters:
  type: striped
  vgName: lvm-vg
provisioner: lvm.driver.harvesterhci.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Can select lvm storage volume and use it as data volume for the VM.
Check the VM is running and the lvm-sc volume is attached on it

$\color{red}{\textsf{Failed}}$ Multiple nodes: Use lvm-csi-driver storage class volume on VM $~~$

Given a disk on node1 have created by lvm provisioner

When we try to create the storage class with the lvm provisioner type, but can't find the node1 on the available list

vokoscreenNG-2024-10-23_18-38-54.mp4

After we create a storage class on node2
Then use this storage class to create a data volume in vm

The VM would failed to start with Unschedulable

0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

vokoscreenNG-2024-10-23_18-40-43.mp4

Test Information

Test Environment: Single node and three nodes baremetal machines
Harvester version: v1.4.0-rc3

Verify Steps

Single node: Use lvm-csi-driver storage class volume on VM

Create the harvester-csi-driver-lvm from yaml in Harvester

apiVersion: harvesterhci.io/v1beta1
kind: Addon
metadata:
  name: harvester-csi-driver-lvm
  namespace: harvester-system
  labels:
    addon.harvesterhci.io/experimental: "true"
spec:
  enabled: false
  repo: https://charts.harvesterhci.io
  version: 0.1.4
  chart: harvester-csi-driver-lvm
  valuesContent: ""

Enabled on Add-ons page
Create Storage for LVM volume group
First time create LVM volume group

Create StorageClass, select provisioner LVM
Select node -> Select volume group name -> select volume group type and save
Created the storage class of LVM provisioner
Check the storage class yaml details contains the allowedTopologies on specific node

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: false
allowedTopologies:
- matchLabelExpressions:
  - key: topology.lvm.csi/node
    values:
    - hp-164-seeder
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2024-10-22T13:38:55Z"
  name: lvm-sc
  resourceVersion: "3969422"
  uid: 12cd92ba-33bb-4e3e-bbb3-44115140a9d7
parameters:
  type: striped
  vgName: lvm-vg
provisioner: lvm.driver.harvesterhci.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Select volume and use it as data volume for the VM.

Check the VM is running and the lvm-sc volume is attached on it

Multiple nodes: Use lvm-csi-driver storage class volume on VM

Create the harvester-csi-driver-lvm from yaml in Harvester

apiVersion: harvesterhci.io/v1beta1
kind: Addon
metadata:
  name: harvester-csi-driver-lvm
  namespace: harvester-system
  labels:
    addon.harvesterhci.io/experimental: "true"
spec:
  enabled: false
  repo: https://charts.harvesterhci.io
  version: 0.1.4
  chart: harvester-csi-driver-lvm
  valuesContent: ""

Enabled on Add-ons page
Create Storage for LVM volume group on node0

Create StorageClass, select provisioner LVM
Select node -> Select volume group name -> select volume group type and save

Created the storage class of LVM provisioner

Check the storage class yaml details contains the allowedTopologies on specific node

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: false
allowedTopologies:
- matchLabelExpressions:
  - key: topology.lvm.csi/node
    values:
    - node2
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2024-10-23T08:09:31Z"
  name: lvm-sc2
  resourceVersion: "257300"
  uid: 6d3a3f00-004a-4f1b-be7c-27aaf7c2591a
parameters:
  type: dm-thin
  vgName: lvm-vg2
provisioner: lvm.driver.harvesterhci.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Create vm, select volume and use it as data volume for the VM.
Check the VM is running and the lvm-sc volume is attached on it

Vicente-Cheng · 2024-10-23T11:49:34Z

root cause:

WARNING: ext4 signature detected on /dev/sdb at offset 1080. Wipe it?

The vgcreate failure is due to the target device not being clean.

It should be fixed with that PR: harvester/node-disk-manager#159
You can use v1.4.0-rc4 to test again or wipefs the corresponding device manually.

TachunLin · 2024-10-24T05:32:02Z

Thanks for checking the root cause.
After using wipefs to cleanup the target disk device signature on the node machine.

$ sudo wipefs --all --force /dev/sdc

node2:~ # sudo wipefs --all --force /dev/sdc
/dev/sdc: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef

And retest in the following steps:

Delete the previously created storage class
Remove the /dev/sdc on node2
Add the /dev/sdc back on node2 given the lvm-vg2 group already created before
Create a new storage class on the disk
Create the new vm, add the second volume with the lvm storage class
Check the vm is running and can attach the volume well

The vm can be started in running and attach the lvm volume well

TachunLin · 2024-10-30T01:00:14Z

Update the retesting state of resting on three nodes v1.4.0-rc4

On a newly provisioned Harvester on kvm machines with extra disks.

After cleanup the signature with wipefs.

rancher@harvester-node-0:~> sudo wipefs /dev/sda
rancher@harvester-node-0:~> lsblk
...
rancher@harvester-node-0:~> sudo wipefs --all --force /dev/sda
rancher@harvester-node-0:~>

Then adding the lvm provisioner disk on host page. It prompt the error message

failed to activate volume group lvm-vg1, err: execute command 'vgchange' with args '[--activate y lvm-vg1]' failed: failed to execute: nsenter [--mount=/host/proc/4716/ns/mnt --net=/host/proc/4716/ns/net --ipc=/host/proc/4716/ns/ipc vgchange --activate y lvm-vg1], output , stderr File descriptor 7 (socket:[45102]) leaked on vgchange invocation. Parent PID 1: /usr/lib/systemd/systemd
Volume group "lvm-vg1" not found
Cannot process volume group lvm-vg1
: exit status 5

As per discussed with Vicente, it a side-effect we introduce in rc4 with provisioner: ensure the device is ready before adding into volume group node-disk-manager#159.
We also have the fix PR Add webhook to ensure the VGStatus is active node-disk-manager#161.
The next step is we will verify the add disk again on the v1.4.0-rc5.

TachunLin · 2024-11-07T11:46:40Z

Retest fixed on v1.4.0-rc5. Close this issue.

Test Result on single node

$\color{green}{\textsf{PASS}}$

NVME disk can be added as lvm-csi-driver disk to Host and be created as volume $~~$

After enabled harvester-csi-controller-lvm addons
Can correctly add the NVME disk as LVM provisioner to the host

Can correctly create the storage class with the selected node and volume group

VM created with LVM provisioner storage class running well

Volume create with the LVM provisioner

Can take a snapshot of the create vm

Disk state

v140rc5-test-lvm:~ # lsblk -o NAME,TYPE,TRAN
NAME                                                                   TYPE TRAN
loop0                                                                  loop 
loop1                                                                  loop 
loop2                                                                  loop 
...
nvme1n1                                                                disk nvme
├─lvm--vg1-pvc--812c99b3--a774--4877--8bf5--4bda9fe1edca-real          lvm  
│ ├─lvm--vg1-pvc--812c99b3--a774--4877--8bf5--4bda9fe1edca             lvm  
│ └─lvm--vg1-lvm--snapshot--d8215eca--79dd--4650--a4be--7426d643e7d5   lvm  
└─lvm--vg1-lvm--snapshot--d8215eca--79dd--4650--a4be--7426d643e7d5-cow lvm  
  └─lvm--vg1-lvm--snapshot--d8215eca--79dd--4650--a4be--7426d643e7d5   lvm

$\color{green}{\textsf{PASS}}$

SATA disk can be added as lvm-csi-driver disk to Host and be created as volume $~~$

After enabled harvester-csi-controller-lvm addons
Can correctly add the SATA disk as LVM provisioner to the host

v140rc5-test-lvm:~ # lsblk -o NAME,TYPE,TRAN
NAME                                                                   TYPE TRAN
...
sdc                                                                    disk   sata

Can correctly create the storage class with the selected node and volume group

VM created with LVM provisioner storage class running well

Volume create with the LVM provisioner

Can take a snapshot of the create vm

Disk state

v140rc5-test-lvm:~ # lsblk -o NAME,TYPE,TRAN
NAME                                                                   TYPE TRAN
...
sdc                                                                    disk sata
├─lvm--vg2-pvc--3ea4506c--260c--48e1--8528--af4235b765a1-real          lvm  
│ ├─lvm--vg2-pvc--3ea4506c--260c--48e1--8528--af4235b765a1             lvm  
│ └─lvm--vg2-lvm--snapshot--2a8acc9d--c8fb--4d55--83bc--bc54552fd97b   lvm  
└─lvm--vg2-lvm--snapshot--2a8acc9d--c8fb--4d55--83bc--bc54552fd97b-cow lvm  
  └─lvm--vg2-lvm--snapshot--2a8acc9d--c8fb--4d55--83bc--bc54552fd97b   lvm

Test Result on multiple nodes

$\color{green}{\textsf{PASS}}$ LVM volume can be created on specific node according to the LVM storage class $~~$

Can display all nodes contains LVM volume group while creating LVM storage class

Only the volume group on n2-140rc5 would be displayed
VM can be created and running ion the node specified in the lvm storage class

Test Information

Test Environment:
- Single node harvester on Equinix bare-metal machine (s3.xlarge.x86)
- Two nodes kvm machines
Harvester version: v1.4.0-rc5

Verify Steps

Test lvm-csi-driver on single node Harvester

Refer to document

Create the harvester-csi-driver-lvm from yaml in Harvester

apiVersion: harvesterhci.io/v1beta1
kind: Addon
metadata:
  name: harvester-csi-driver-lvm
  namespace: harvester-system
  labels:
    addon.harvesterhci.io/experimental: "true"
spec:
  enabled: false
  repo: https://charts.harvesterhci.io
  version: 0.1.4
  chart: harvester-csi-driver-lvm
  valuesContent: ""

Enabled harvester-csi-driver-lvm on Add-ons page

Edit Host page to add a disk in Storage
Select NVEM/SATA/SCSI type of disk
Select the LVM provisioner, and create a volume group lvm-vg1

Check the disk can be correct added to the host

Create StorageClass, select provisioner LVM
Select node -> Select volume group name -> select volume group type and save

Create a new storage class lvm-sc-nvme, select LVM provisioner
Select the Node, Volume Group Name and Volume Group Type

Check can correctly create the storage class

Create a vm and use the storage class lvm-sc-nvme

Check the vm can running well

Check the volume is created with the LVM provisioner

In the csi-driver-config section, select ⋮ > Edit Setting
Add an entry with the following settings and save

Provisioner: Select lvm.driver.harvesterhci.io.
Volume Snapshot Class Name: Select lvm-snapshot.
Backup Volume Snapshot Class Name: Select lvm-snapshot.

Take a snapshot of the created vm
Verify that snapshot is ready to us

Test lvm-csi-driver on multiple nodes Harvester

1. Create the harvester-csi-driver-lvm from yaml in Harvester
Enabled harvester-csi-driver-lvm on Add-ons page
Open the node1 host page -> Storage
Add a disk and select the LVM provisioner, create a new volume group: lvm-vg1

Open the node2 host page -> Storage
Add a disk and select the LVM provisioner, create a new volume group: lvm-vg2

Create a new storage class lvm-sc2 with LVM provisioner`
Select node n2-140rc5
Only the volume group on n2-140rc5 would be displayed

Create a vm and add a volume, select storage class with lvm-sc2

Check the vm can start in running state on node n2-140rc5

bk201 added the kind/feature Issues that represent larger new pieces of functionality, not enhancements to existing functionality label May 2, 2024

bk201 mentioned this issue May 2, 2024

[ENHANCEMENT] refine the node-disk-manager for more flexible to storage backend #5713

Closed

bk201 added this to the v1.5.0 milestone May 2, 2024

bk201 assigned Vicente-Cheng May 2, 2024

bk201 modified the milestones: v1.5.0, v1.4.0 May 14, 2024

innobead added area/storage highlight Highlight issues/features labels May 14, 2024

innobead added priority/0 Must be fixed in this release require/HEP Require Harvester Enhancement Proposal PR area/ui Harvester UI require/doc Improvements or additions to documentation labels May 28, 2024

Vicente-Cheng mentioned this issue Jun 4, 2024

HEP: introduce the LVM CSI driver #5956

Open

Vicente-Cheng mentioned this issue Jun 12, 2024

[ENHANCEMENT] Node Disk Manager refactor #6014

Closed

3 tasks

bk201 added require-ui/small Estimate 1-2 working days and removed area/ui Harvester UI require-ui/small Estimate 1-2 working days labels Jun 24, 2024

harvesterhci-io-github-bot mentioned this issue Jun 24, 2024

[GUI] [FEATURE] Host local storage support #6057

Closed

This was referenced Jul 10, 2024

oem: patch lvm.conf for the local path (lvm) support harvester/os2#121

Merged

misc improvement harvester/csi-driver-lvm#5

Merged

innobead added the experimental Experimental feature (alpha) label Jul 18, 2024

mergify bot mentioned this issue Jul 26, 2024

oem: patch lvm.conf for the local path (lvm) support (backport #121) harvester/os2#122

Merged

Vicente-Cheng mentioned this issue Sep 5, 2024

Add lvm provisioner harvester/node-disk-manager#128

Merged

harvesterhci-io-github-bot mentioned this issue Sep 10, 2024

[e2e] [FEATURE] Host local storage support harvester/tests#1515

Open

1 task

mergify bot mentioned this issue Sep 12, 2024

Add lvm provisioner (backport #128) harvester/node-disk-manager#131

Merged

tserong mentioned this issue Sep 16, 2024

bump(chart): node-manager to v0.3.2, node-disk-manager to v0.7.3 #6567

Merged

TachunLin self-assigned this Oct 22, 2024

innobead changed the title ~~[FEATURE] Host local storage support~~ [FEATURE] Host local storage support - Experimental Oct 24, 2024

Vicente-Cheng mentioned this issue Nov 5, 2024

docs: add local storage support harvester/docs#663

Merged

TachunLin closed this as completed Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Host local storage support - Experimental #5724

[FEATURE] Host local storage support - Experimental #5724

bk201 commented May 2, 2024

SlavikCA commented Jun 11, 2024

Vicente-Cheng commented Jun 12, 2024

abonillabeeche commented Jun 19, 2024

harvesterhci-io-github-bot commented Jun 24, 2024

abonillabeeche commented Jul 15, 2024

albinsun commented Jul 18, 2024

innobead commented Jul 18, 2024 •

edited

Loading

harvesterhci-io-github-bot commented Sep 10, 2024 •

edited by Vicente-Cheng

Loading

harvesterhci-io-github-bot commented Sep 10, 2024

TachunLin commented Oct 22, 2024 •

edited

Loading

Vicente-Cheng commented Oct 23, 2024

TachunLin commented Oct 24, 2024 •

edited

Loading

TachunLin commented Oct 30, 2024 •

edited

Loading

TachunLin commented Nov 7, 2024 •

edited

Loading

[FEATURE] Host local storage support - Experimental #5724

[FEATURE] Host local storage support - Experimental #5724

Comments

bk201 commented May 2, 2024

SlavikCA commented Jun 11, 2024

Vicente-Cheng commented Jun 12, 2024

abonillabeeche commented Jun 19, 2024

harvesterhci-io-github-bot commented Jun 24, 2024

abonillabeeche commented Jul 15, 2024

albinsun commented Jul 18, 2024

innobead commented Jul 18, 2024 • edited Loading

harvesterhci-io-github-bot commented Sep 10, 2024 • edited by Vicente-Cheng Loading

Pre Ready-For-Testing Checklist

harvesterhci-io-github-bot commented Sep 10, 2024

TachunLin commented Oct 22, 2024 • edited Loading

Result

Test Information

Verify Steps

Vicente-Cheng commented Oct 23, 2024

TachunLin commented Oct 24, 2024 • edited Loading

TachunLin commented Oct 30, 2024 • edited Loading

TachunLin commented Nov 7, 2024 • edited Loading

Test Result on single node

Test Result on multiple nodes

Test Information

Verify Steps

innobead commented Jul 18, 2024 •

edited

Loading

harvesterhci-io-github-bot commented Sep 10, 2024 •

edited by Vicente-Cheng

Loading

TachunLin commented Oct 22, 2024 •

edited

Loading

TachunLin commented Oct 24, 2024 •

edited

Loading

TachunLin commented Oct 30, 2024 •

edited

Loading

TachunLin commented Nov 7, 2024 •

edited

Loading