Linux 6.6.x(2.2.5) -> 6.12.x(2.2.7): WRITE FPDMA QUEUED (ATA bus error) #16873
Comments
IvanVolosyuk
changed the title
|
@IvanVolosyuk Can you try |
Doesn't look like it helps: Nothing in |
|
I reproduced that on linux 6.12, zfs 2.2.7 without PREEMPT_RT (zfs_vdev_disk_classic=1) :( |
IvanVolosyuk
changed the title
|
I saw the same thing in my Gentoo 6.12.4 with ZFS 2.2.7 during a zfs scrub.
|
|
@RinCat what kernel / zfs version you had before you started experiencing this? |
|
I think my last linux 6.11.11 with zfs 2.2.6 works fine. Only saw this when I bump to 6.12 with zfs 2.2.7. |
|
@RinCat Another question, do you use rcu_nocbs, nohz_full and/or realtime priority tasks? My qemu runs at realtime priority with vCPU pinning to rcu offloaded CPUs. |
|
@IvanVolosyuk no, it just normal "Preemptible Kernel (Low-Latency Desktop)". I did not try the new RT config. |
|
I have a bad feeling about this. |
|
Maybe it's a coincidence but I'm having the same issue after upgrading to zfs 2.2.7. A ton of errors under high I/O pressure. Switched back to 2.2.6 and the errors went away. |
@RichardBelzer Did you change the kernel version as well with ZFS 2.2.7? I went back to Linux 6.6.66 keeping ZFS 2.2.7 and I don't see the errors. What kernel version do you use? |
|
Ok, there are 2 issues here it seems. I figured out '[WRITE FPDMA QUEUED]' part. As of linux 6.11 (5433f0e7427ae4f5b128d89ec16ccaafc9fef5ee) the default sata As for CONFIG_PREEMT_RT, I'll give up for now. It seems I get multiple lockups with no fs reads coming through and I'm not sure if it ZFS related on not. |
IvanVolosyuk
changed the title
|
@IvanVolosyuk I changed |
|
You were on Linux 6.11.x which already changed link_power_management_policy, so it should be something different in your case. And this issue is specially about "ATA bus error" subclass of the errors. In general the WRITE FPDMA QUEUED errors can be caused literally by anything from failing disk to power delivery problems. @RinCat in your case 'SError: { UnrecovData 10B8B BadCRC }' means the disk reports that it is dying or at least has unreadable sectors. |
|
on debian PACKAGE_NAME="zfs"
PACKAGE_VERSION="2.2.7"
PACKAGE_CONFIG="/etc/dkms/zfs.conf"
NO_WEAK_MODULES="yes"
if [ -f $kernel_source_dir/.config ]; then
. $kernel_source_dir/.config
if [ "$CONFIG_PREEMPT_RT" = "y" ]; then
BUILD_EXCLUSIVE_KERNEL="NOT ON PREEMPT_RT"
fi
fiprevious maybe related ref #11097 referencing rw_sempahore so there may be a hint there even if the snippets from back then have entirely changed in current tree |
|
Just hit this error on Arch when testing out kernel 6.12.11, coming from 6.6.72. Disks are HGST 14 TB SATA and ZFS is 2.3.0. Problem disappears on 6.6.74 but here CONFIG_SATA_MOBILE_LPM_POLICY=3 is already set. Is there a final judgement what is causing this? Is it the new preempt, a SATA driver bug with preempt, or ZFS and 6.12, or something else? |
Something else might been changing the link pm policy. Check It should say Also, do you have |
|
Thank you for taking the time to reply! Appreciate it. Yes, ATA bus error, on multiple disks. link_power_management_policy says indeed max_performance right now. Will check the kernel in a couple days again. How did you tweak it, with kernel config switch during compilation or udev or similar? I would like the kernel to just do the right thing from get go, without needing a tweak in initramfs.
|
|
I just set it in kernel config |
|
Arch Linux has CONFIG_SATA_MOBILE_LPM_POLICY=3 for 6.6 (old linux-lts package) which I am using, as well as 6.12 (linux-lts package since a few days, they rolled). Checking makes me think perhaps the AHCI_HFLAG_USE_LPM_POLICY check in 6.6 prevented SATA links to the hard disks from being set to Device Initiated Power Management. And nobody noticed because who uses hard drives any more, right. Another workaround is ahci.mobile_lpm_policy=1 kernel parameter to nail the setting to max performance. If you run spinners, link power saving might not be the most important issue in overall power usage. |
|
Addendum: Observe the lpm-pol 4 on vanilla 6.12 for the mainboard (Supermicro X11SAT, C236 chipset) AHCI ports. This looks to be ATA_LPM_MIN_POWER_WITH_PARTIAL and is one step too much for my HGST HC530 14 TB drives: So newer kernels seem to take link power management suggestions from the AHCI chip or wherever and this trips up hard drives. Question remains is where this target_lpm_policy 4 is set or coming from, because kernel's CONFIG_SATA_MOBILE_LPM_POLICY is set to 3. Of note, this is the second time I see a non-benign regression introduced by Niklas Cassel, after shooting udisks in the kneecaps storaged-project/udisks#732.
|
|
Addendum: ahci.mobile_lpm_policy=1 is without effect, lpm-pol stays on a Xeon Cascade Lake system at 0. I am stopping here to crawl around in this rabbit hole. Will use 6.6 for longer it seems, showstopper bug. |
|
While this issue is closed (and for good reason, since not connected to ZFS), here are some more links about the issue. It is on front page of all search machines for this bug and Perplexity and Grok have started quoting it.
My own patch for my type of drives (WDC WD140 "shucked" white-labels) and kernel 6.12.11: I haven't bothered to submit this upstream because I think they can figure it out themselves in the brief moments when they are actually working on the kernel and aren't busy setting 97% of the budget on fire in non-Linux projects. |
I tried to boot ZFS 2.2.7 with 6.12.4 kernel with new PREEMPT_RT.
Running local
zfs send ... | zfs recv ...while running 3DMark / Port Royal in Windows 11 VM in kvm/vfio.I ran this workload perfectly fine on 6.6.xx and ZFS 2.2.5, but now it actually failed badly with WRITE FPDMA QUEUED, hang zfs kthreads, zed errors. Somehow, the system was writing logs fine on another pool and
zpool statusshowed zero errors. All my disks are parts of zpools, so no writes can happen outside of ZFS.System information
Describe the problem you're observing
zpool status. The smart data from disks show UDMA_CRC_Error_Count, Reallocated_Sector_Ct, Offline_Uncorrectable all at zero.Describe how to reproduce the problem
Just got it once before pulling the plug. I believe it may be bad interaction of kernel compiled with PREEMPT_RT and ZFS while having a lot of disk activity from
zpool sendand kvm.Run qemu/kvm/vfio with vCPU pinning and run 3dmark benchmark with PREEMPT_RT kernel.
zfs receiverunning in background storing around 50G into that pool.Kernel params:
Include any warning/errors/backtraces from the system logs
log.txt
kernel-config.txt
But
zpool statusreporting 0 errors. The logs were successfully stored by a different ssd-only pool. Interesting bits:The text was updated successfully, but these errors were encountered: