zfs-2.1.7 dkms fails to install on Fedora 36

[mabachel]

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	36
Kernel Version	6.0.10-200.fc36
Architecture	x86_64
OpenZFS Version	2.1.7

Describe the problem you're observing

Installing and or reinstalling does not work because during the installation of zfs-2.1.7 /var/lib/dkms/zfs/2.1.5/source/dkms.conf is missing. Note the different versions (2.1.5 vs. 2.1.7)

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.5/source/dkms.conf does not exist.

Describe how to reproduce the problem

sudo dnf reinstall zfs zfs-dkms

Include any warning/errors/backtraces from the system logs

$ ls /var/lib/dkms/zfs/
total 12
drwxr-xr-x. 1 root root 248  2. Dez 11:20 .
drwxr-xr-x. 1 root root  34 28. Okt 11:06 ..
drwxr-xr-x. 1 root root  58  4. Okt 08:17 2.1.5
drwxr-xr-x. 1 root root  58  4. Okt 08:16 2.1.6
drwxr-xr-x. 1 root root  54  2. Dez 11:20 2.1.7
lrwxrwxrwx. 1 root root  36 10. Jul 16:36 kernel-5.17.13-300.fc36.x86_64-x86_64 -> 2.1.5/5.17.13-300.fc36.x86_64/x86_64
lrwxrwxrwx. 1 root root  36  4. Okt 08:16 kernel-5.19.12-200.fc36.x86_64-x86_64 -> 2.1.6/5.19.12-200.fc36.x86_64/x86_64
lrwxrwxrwx. 1 root root  34  2. Dez 11:20 kernel-6.0.7-200.fc36.x86_64-x86_64 -> 2.1.7/6.0.7-200.fc36.x86_64/x86_64
[relentless@machine ~]$ sudo dnf reinstall zfs zfs-dkms
Last metadata expiration check: 0:13:51 ago on Fr 02 Dez 2022 11:15:28 CET.
Dependencies resolved.
================================================================================================================================================================
 Package                                Architecture                         Version                                    Repository                         Size
================================================================================================================================================================
Reinstalling:
 zfs                                    x86_64                               2.1.7-1.fc36                               zfs                               659 k
 zfs-dkms                               noarch                               2.1.7-1.fc36                               zfs                                30 M

Transaction Summary
================================================================================================================================================================

Total download size: 30 M
Installed size: 64 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): zfs-2.1.7-1.fc36.x86_64.rpm                                                                                              297 kB/s | 659 kB     00:02    
(2/2): zfs-dkms-2.1.7-1.fc36.noarch.rpm                                                                                         836 kB/s |  30 MB     00:36    
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                           854 kB/s |  30 MB     00:36     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                        1/1 
  Reinstalling     : zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           1/4 
  Running scriptlet: zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           1/4 
Removing old zfs-2.1.7 DKMS files...
Module zfs-2.1.7 for kernel 6.0.7-200.fc36.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

zavl.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

znvpair.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zunicode.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zcommon.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zfs.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

icp.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zlua.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

spl.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zzstd.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.7-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod......
Deleting module zfs-2.1.7 completely from the DKMS tree.
Loading new zfs-2.1.7 DKMS files...
Building for 6.0.10-200.fc36.x86_64
Building initial module for 6.0.10-200.fc36.x86_64
Done.

zavl.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

znvpair.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

zunicode.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

zcommon.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

zfs.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

icp.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

zlua.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

spl.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/

zzstd.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.0.10-200.fc36.x86_64/extra/
depmod......

  Reinstalling     : zfs-2.1.7-1.fc36.x86_64                                                                                                                2/4 
  Running scriptlet: zfs-2.1.7-1.fc36.x86_64                                                                                                                2/4 
  Running scriptlet: zfs-2.1.7-1.fc36.x86_64                                                                                                                3/4 
  Cleanup          : zfs-2.1.7-1.fc36.x86_64                                                                                                                3/4 
  Running scriptlet: zfs-2.1.7-1.fc36.x86_64                                                                                                                3/4 
  Running scriptlet: zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           4/4 
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.5/source/dkms.conf does not exist.

Uninstall of zfs module (zfs-2.1.7-1) beginning:
Module zfs-2.1.7 for kernel 6.0.10-200.fc36.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

zavl.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

znvpair.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zunicode.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zcommon.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zfs.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

icp.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zlua.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

spl.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

zzstd.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/6.0.10-200.fc36.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod.....
Deleting module zfs-2.1.7 completely from the DKMS tree.

  Cleanup          : zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           4/4 
  Running scriptlet: zfs-2.1.7-1.fc36.x86_64                                                                                                                4/4 
  Running scriptlet: zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           4/4 
  Verifying        : zfs-2.1.7-1.fc36.x86_64                                                                                                                1/4 
  Verifying        : zfs-2.1.7-1.fc36.x86_64                                                                                                                2/4 
  Verifying        : zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           3/4 
  Verifying        : zfs-dkms-2.1.7-1.fc36.noarch                                                                                                           4/4 

Reinstalled:
  zfs-2.1.7-1.fc36.x86_64                                                      zfs-dkms-2.1.7-1.fc36.noarch                                                     

Complete!

[SirFritz]

I also have this issue on Fedora 36. Although for me it's 2.1.6 not 2.1.5.

File: /var/lib/dkms/zfs/2.1.6/source/dkms.conf does not exist.

[almereyda]

Confirmed:

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.5/source/dkms.conf does not exist.

Uninstall of zfs module (zfs-2.1.7-1) beginning:
Module zfs-2.1.7 for kernel 6.0.10-200.fc36.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

Here it is still possible to build the kernel module on one's own:

$ dkms install zfs/2.1.7
Sign command: /lib/modules/6.0.10-200.fc36.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Creating symlink /var/lib/dkms/zfs/2.1.7/source -> /usr/src/zfs-2.1.7
…
Building module:
Cleaning build area...
make -j8 KERNELRELEASE=6.0.10-200.fc36.x86_64..............................
Signing module /var/lib/dkms/zfs/2.1.7/build/module/avl/zavl.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/nvpair/znvpair.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/unicode/zunicode.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zcommon/zcommon.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zfs/zfs.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/icp/icp.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/lua/zlua.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/spl/spl.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zstd/zzstd.ko
…
$ modprobe zfs
$ zfs version
zfs-2.1.7-1
zfs-kmod-2.1.7-1

[chenxiaolong]

I've run into this several times in the past. It seems to happen if zfs-dkms and the kernel are updated in the same transaction. Maybe the RPM hooks are running in the wrong order?

I've been able to prevent it from happening by doing updates in two transactions:

# All updates, including zfs, but excluding the kernel
sudo dnf update --exclude kernel\*
# Just the kernel
sudo dnf update

If it's already in the broken state, I've taken the nuclear option in the past:

sudo rpm -e --nodeps zfs-dkms
sudo rm -r /var/lib/dkms/zfs/<leftover old version>
sudo dnf install zfs-dkms

[almereyda]

Am I mistaken, or can DKMS only build against the current Kernel? Because this is what happens: The recompilation hook fires during the transaction. Yesterday during the update, a lovely module for 2.1.7/6.0.9 was created, while the Kernel prepped itself to be at 6.0.10. with the next reboot. Before uninstall, this module version was ACTIVE on this kernel.

As the DKMS build ultimately worked, this feels like a packaging question around Fedoras RPMs and how DNF resolves the dependencies during the module build triggers, some kind of race condition of active and newly installed Kernels, ZFS package versions, and at what time a compilation is fired. We've seen that kind of behaviour with ZFS RPM package updates many times over the last years. Would there be some kind of dnf trigger, like the system upgrade code path, which could rebuild the module during the first boot into the new Kernel?

For Fedora, similar behaviours are also observed on Silverblue:

• https://discussion.fedoraproject.org/t/silverblue-and-zfs/18000

[arch-user-france1]

Confirmed:

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.5/source/dkms.conf does not exist.

Uninstall of zfs module (zfs-2.1.7-1) beginning:
Module zfs-2.1.7 for kernel 6.0.10-200.fc36.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

Here it is still possible to build the kernel module on one's own:

$ dkms install zfs/2.1.7
Sign command: /lib/modules/6.0.10-200.fc36.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Creating symlink /var/lib/dkms/zfs/2.1.7/source -> /usr/src/zfs-2.1.7
…
Building module:
Cleaning build area...
make -j8 KERNELRELEASE=6.0.10-200.fc36.x86_64..............................
Signing module /var/lib/dkms/zfs/2.1.7/build/module/avl/zavl.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/nvpair/znvpair.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/unicode/zunicode.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zcommon/zcommon.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zfs/zfs.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/icp/icp.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/lua/zlua.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/spl/spl.ko
Signing module /var/lib/dkms/zfs/2.1.7/build/module/zstd/zzstd.ko
…
$ modprobe zfs
$ zfs version
zfs-2.1.7-1
zfs-kmod-2.1.7-1

I can not confirm that. I have tried:

sudo dkms install zfs/2.1.7

Sign command: /lib/modules/6.1.18-200.fc37.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.7/source/dkms.conf does not exist

[almereyda]

I believe we can close here, now that subsequent versions with Fedora 36 support had been released.