Reset backup failed after restore individual disk on the VM #6741
Assignees
Milestone
Comments
|
We need to look at the hole picture here. If we are using OpenNebula as a public cloud, there will be many clients probably restoring individual disks, directly to VMs. You can not tell these clients that they need to poweroff their VMs, in order to execute a workaround, that will restablish the backing chain of these VMs. This is not feasible solution. Anyway, this is a workaround I have documented for this issue: INCREMENTAL BACKUP ISSUE: Bitmap not found in backing chainIf problem does not get fixed with a reset backup, then apply following procedure. On the orchestrator leader Define the following vars: VM_ID=10
DISK_ID=0
BACKUP_DS_ID=101
SYSTEM_DS_ID=102
BITMAP_ID=0Get qemu-img info /var/lib/one/datastores/${SYSTEM_DS_ID}/${VM_ID}/disk.${DISK_ID}The output looks as follow: image: /var/lib/one/datastores/0/10/disk.0
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.05 GiB
cluster_size: 65536
backing file: /var/lib/one//datastores/1/e4671f022ec4a6ac1abe89ee163a3ab0
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
bitmaps:
[0]:
flags:
[0]: auto
name: one-10-0
granularity: 65536
refcount bits: 16
corrupt: falseShutdown the VM: onevm poweroff ${VM_ID}Check the host from where the given VM is being running: HOST=$(onevm show ${VM_ID} | grep HOST | grep kvm | awk '{print $NF}')
echo $HOSTThe output was as follow: ON-LA-N1-kvmForce the removal of residual files from previous incremental backups: rm -rf /var/lib/one/datastores/${SYSTEM_DS_ID}/${VM_ID}/backup/ && \
rm -rf /var/lib/one/datastores/${SYSTEM_DS_ID}/${VM_ID}/tmp/ && \
rm -rf /var/lib/one/datastores/${BACKUP_DS_ID}/${VM_ID}/backup/ && \
rm -rf /var/lib/one/datastores/${BACKUP_DS_ID}/${VM_ID}/tmp/ && \
ssh $HOST "rm -rf /var/lib/libvirt/qemu/domain-*-one-${VM_ID}/" && \
ssh $HOST "rm -rf /var/lib/libvirt/qemu/checkpoint/one-${VM_ID}"Remove the bitmap and possible checkpoints: ssh $HOST "qemu-img bitmap --remove /var/lib/one/datastores/${SYSTEM_DS_ID}/${VM_ID}/disk.${DISK_ID} one-${VM_ID}-${BITMAP_ID}" && \
ssh $HOST "rm -rf /var/lib/libvirt/qemu/domain-*-one-${VM_ID}/"Resume VM: onevm resume ${VM_ID} |
|
I will keep an eye on this. Pretty good stuff!!! |
|
Thanks, @Franco-Sparrow. Thinking out loud, one feasible way could be removing all bitmaps of the remaining disk images during the restore-in-place process; in this way, when the VM resumes, all disks have the same information related to bitmaps. I will take a look at the new code! |
|
Hi We found this issue as part of the development of #6411 (Ceph incremental backup). We implemented a rest of a chain after a "in-place" restore. so if the VM is configured to do incremental backups, and do a restore, the next backup will start a new incremental chain. Same behavior as when a disk is attached to the VM, the backup chain is reset. Thanks for the feedback! (PS: This will be in 6.10.1, I'll close the issue together with #6411 when it is merged) |
rsmontero
added a commit
that referenced
this issue
Jan 13, 2025
Implementation overview: - Incremental points are saved as dedicated rbd snapshots under the "one_backup_<increment_id>" namespace. This snapshots are used to generate delta files in rbdiff format. - The rbdiff formats are stored in the backup server to restore the rbd volumes. - The restore process is performed directly on the Ceph cluster importing the base image (first full backup in the chain, rbd import) and then applying the increments (rbd import-diff) up to the target increment. - Two new pseudo-protocols has been implemented to adopt the restore pattern above (restic+rbd, rsync+rbd). This protocols bundle of the rbdiff files in a tarball for transfer from the backup server. Note: reconstruct process uses the Ceph BRIDGE_LIST and not the backup server (as opposed to qcow2 backups) Other bug fixes - This commit also fixes #6741, resetting the backup chain after a restore - The original ceph drivers do not receive the full action information, this now has been fixed by including VM information in the STDIN string sent to the driver. Compatibility note. - backup actions should return now the backup format used raw, rbd, ... If not provided oned (6.10.x) will use raw as a default to accommodate any third party driver implementation. It is recommended to include this third argument. Signed-off-by: Guillermo Ramos <[email protected]> Co-authored-by: Guillermo Ramos <[email protected]> (cherry picked from commit 5f7b370)
Re-Opening issueHi @rsmontero I am opening this issue again, because is still present on
These are the logs entries from the given VM: I restored an individual disk (vdb) which had some previous incremental backups from the original VM, before the restore. After restored the disk, we need to execute a Expected behaviorThe reset backup should works for a VM that has individual disks restored and is on RUNNING state. Details
|
rsmontero
pushed a commit
that referenced
this issue
Feb 4, 2025
Signed-off-by: Guillermo Ramos <[email protected]>
rsmontero
pushed a commit
that referenced
this issue
Feb 4, 2025
Signed-off-by: Guillermo Ramos <[email protected]> (cherry picked from commit bd1965e)
1gramos
added a commit
to OpenNebula/docs
that referenced
this issue
Feb 4, 2025
OpenNebula/one#6547, OpenNebula/one#6741, OpenNebula/one#6583, OpenNebula/one#2109 Signed-off-by: Guillermo Ramos <[email protected]>
2 tasks
rsmontero
pushed a commit
to OpenNebula/docs
that referenced
this issue
Feb 4, 2025
OpenNebula/one#6547, OpenNebula/one#6741, OpenNebula/one#6583, OpenNebula/one#2109 Signed-off-by: Guillermo Ramos <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is really nice to have a restore in place now, without the need of detatch or create new VMs for the use of these backups. Thanks to OpenNebula team for this new feature. Anyway, please, check this bug report:
Description
If you execute a restore in place (directly on the VM) for a given disk (suppose you restore the 2nd disk of the VM
disk.2), you are forced to apply aresetbackup for the next backups of the VM. When you execute theresetbackup, it fails, because the disk that was not restored and is the original disk of the VM (disk.0) still hasdirty-bitmapsthat the new restored disk doesnt have.To Reproduce
non-volatiledisks.BACKUP_VOLATILE: YES,INCREMENT_MODE: CBT,FS_FREEZE: AGENT,KEEP_LAST: 3,MODE: INCREMENT.inc0).inc1).disk.2) from theinc1.resetbackup with the VM inRUNNINGstate and see how it fails.vdb(disk.2) has samedirty-bitmapsandcheckpointsas thevda(disk.0), and is not the case.resetbackup with the VM inPOWEROFFstate, it works, probably because thedirty-bitmapsare converted into bitmaps and they are successfully removed.Expected behavior
The
resetbackup should works for a VM that has individual disks restored and is onRUNNINGstate.Details
Additional context
Add any other context about the problem here.
Progress Status
The text was updated successfully, but these errors were encountered: