ZFS crashes when reading from a block pointer with no valid DVAs #17077
Labels
Type: Defect
Incorrect behavior (e.g. crash, hang)
Comments
|
If we assume zero asize is illegal (it is definitely weird), then we could check for it in |
Yes, I have a patch that does exactly that, and I'll open a PR soon. |
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
If a block pointer isn't embedded yet doesn't have any valid DVAs, that's a data corruption bug. zdb should be able to handle the situation gracefully. Issue openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
If a block pointer isn't embedded yet doesn't have any valid DVAs, that's a data corruption bug. zdb should be able to handle the situation gracefully. Issue openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
Now instead of crashing when attempting to read the corrupt block pointer, ZFS will return ECKSUM, in a stack that looks like this: ``` none:set-error zfs.ko`arc_read+0x1d82 zfs.ko`dbuf_read+0xa8c zfs.ko`dmu_buf_hold_array_by_dnode+0x292 zfs.ko`dmu_read_uio_dnode+0x47 zfs.ko`zfs_read+0x2d5 zfs.ko`zfs_freebsd_read+0x7b kernel`VOP_READ_APV+0xd0 kernel`vn_read+0x20e kernel`vn_io_fault_doio+0x45 kernel`vn_io_fault1+0x15e kernel`vn_io_fault+0x150 kernel`dofileread+0x80 kernel`sys_read+0xb7 kernel`amd64_syscall+0x424 kernel`0xffffffff810633cb ``` This patch should hopefully also prevent such corrupt block pointers from being written to disk in the first place. Fixes openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
If a block pointer isn't embedded yet doesn't have any valid DVAs, that's a data corruption bug. zdb should be able to handle the situation gracefully. Issue openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
Now instead of crashing when attempting to read the corrupt block pointer, ZFS will return ECKSUM, in a stack that looks like this: ``` none:set-error zfs.ko`arc_read+0x1d82 zfs.ko`dbuf_read+0xa8c zfs.ko`dmu_buf_hold_array_by_dnode+0x292 zfs.ko`dmu_read_uio_dnode+0x47 zfs.ko`zfs_read+0x2d5 zfs.ko`zfs_freebsd_read+0x7b kernel`VOP_READ_APV+0xd0 kernel`vn_read+0x20e kernel`vn_io_fault_doio+0x45 kernel`vn_io_fault1+0x15e kernel`vn_io_fault+0x150 kernel`dofileread+0x80 kernel`sys_read+0xb7 kernel`amd64_syscall+0x424 kernel`0xffffffff810633cb ``` This patch should hopefully also prevent such corrupt block pointers from being written to disk in the first place. Fixes openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
asomers
added a commit
to asomers/zfs
that referenced
this issue
Feb 20, 2025
Now instead of crashing when attempting to read the corrupt block pointer, ZFS will return ECKSUM, in a stack that looks like this: ``` none:set-error zfs.ko`arc_read+0x1d82 zfs.ko`dbuf_read+0xa8c zfs.ko`dmu_buf_hold_array_by_dnode+0x292 zfs.ko`dmu_read_uio_dnode+0x47 zfs.ko`zfs_read+0x2d5 zfs.ko`zfs_freebsd_read+0x7b kernel`VOP_READ_APV+0xd0 kernel`vn_read+0x20e kernel`vn_io_fault_doio+0x45 kernel`vn_io_fault1+0x15e kernel`vn_io_fault+0x150 kernel`dofileread+0x80 kernel`sys_read+0xb7 kernel`amd64_syscall+0x424 kernel`0xffffffff810633cb ``` This patch should hopefully also prevent such corrupt block pointers from being written to disk in the first place. Fixes openzfs#17077 Sponsored by: ConnectWise Signed-off-by: Alan Somers <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
Describe the problem you're observing
Somehow ZFS wrote a block pointer to an L0 block in such a way that the only DVA has an asize of 0. Yet the block pointer is neither embedded nor a hole. As far as I understand, that should be impossible. Yet it happened. Now ZFS crashes whenever it tries to read that block. And zdb crashes whenever it tries to display the block pointer.
Describe how to reproduce the problem
Now that the block pointer is corrupt on disk, the problem can be easily reproduced simply by trying to read the corrupt block with
dd. It is 100% reproducible. But I have no idea how to reproduce the corruption in the first place.The zdb crash can be reproduced with a command like
zdb -vv -bbbb -O <DATASET> -U <CACHEFILE> <PATH/TO/FILE>Include any warning/errors/backtraces from the system logs
The precise stack trace depends on whether INVARIANTS are enabled. If they are not, then the crash looks like this, in FreeBSD 14.1-RELEASE:
However, if INVARIANTS are enabled, the crash happens sooner. This is what it looks like on 15.0-CURRENT with INVARIANTS.
The zdb crash looks like this:
The text was updated successfully, but these errors were encountered: