Storage Live migration for KVM

[OpenNebulaSupport]

Enhancement Request

Description

Live migration is now possible between hosts with shared storage. This feature implies enabling VM live migration between hosts with no shared storage, or from one DS to another within the same host

Use case

To load balance storage between datastores manually. In the future, a dynamic scheduler can optimize storage balancing.

Interface Changes

Changes in XMLRPC, CLI and Sunstone

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[tinova]

Splitting the issue in three:

#1540 - vCenter unmanaged live migration
#2875 - vCenter managed live migration
#1644 (Original issue) - Only for KVM

[vholer]

Forum discussion - community implementation:
https://forum.opennebula.org/t/ssh-live-migration-without-shared-storage-call-for-discussion/7354

[mzealey]

Using kvm -d migrate=migrate-live action with the script https://gist.github.com/mzealey/0cac6328c8bbc86c733a72e7a6af89a7 is working well for us.

[xorel]

Based on the comments and the proposal from the forum we have implemented the live migration for ssh TM. (f539f97)

Any comments are appreciated, (FYI: @atodorov-storpool)

[atodorov-storpool]

@xorel
Excuse me for the delay I was AFK for a while and still catching up with the stuff...

First some cosmetics - I thing it is better to have RSSH=rssh in scripts common and it should be used in f539f97#diff-520edf0754a9dc0780eebdcf667c19a0R169 . In this way someone could change the compression or the ssh options used by rssh :-)

Regarding --copy-storage-all While reading
https://wiki.libvirt.org/page/NBD_storage_migration
https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Virtualization_Deployment_and_Administration_Guide/virsh-migration-arguements.html
my understanding is that on a mixed environment like a VM with disks backed by both qcow files and other storage that could provide shared disks like StorPool, iSCSI, HPE 3PAR this will lead to unnecessary overwrite of the shared disks

That said if you follow https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.ldva/ldva_t_migratingprocess.html
and
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-kvm_live_migration-live_kvm_migration_with_virsh

There is an additional option --migrate-disks X,Y so IMHO the disk paths in the domainXML should be tested and only the non-shared disks (lets say only the disks that are of type files) should be added, something like --copy-storage-all --migrate-disks vda,vdc

Best Regards,
Anton Todorov

[atodorov-storpool]

(lets say only the disks that are of type files)

On a second thought this will include the persistent NFS backed disks too. So for example, if we have a VM with disk
sda -> qcow2 file on HV host filesystem(volatile disk on ssh TM_MAD)
sdb -> symlink -> qcow2 file on NFS (persistent image from NFS backed IMAGE datastore)
sdc -> /dev/sxx -> raw block device (iSCSI, StorPool, HPE 3PAR, etc)
hda -> qcow2 file contextualization cdrom

The virsh command should be --copy-storage-all --migrate-disks sda,hda (not sure for the CDROM because it is marked as RO and most likely excluded from the disk-copy process...
Also, it is possible that libvirt had some internal checks is the disk on a shared filesystem or not so the NFS in the above example could be irrelevant...

Creers
Anton Todorov

[mzealey]

Yes we had some similar issues to what you surmise - see this file for our approach migrate-live.txt which works with qcow2 type storage

[xorel]

Thank you @atodorov-storpool for your comments, I made few changes in #5119

Mainly

• changed rsync to tar|ssh, as rssh doesn't seem to be available for CentOS
• symlinks are re-created
• local qcow2 and raw disks are copied
• added the missing --migrate-disk

I tested it in an environment as you propose, having shared iscsi disks + other qcow2/raw either shared or local disks, seems working well.

Could you please comment on the pull req.?