error: Committing: Adding composefs metadata: No data available

[Szwendacz99]

Host system details

State: idle
Deployments:
● fedora:fedora/38/x86_64/kinoite
                  Version: 38.20231009.0 (2023-10-09T01:14:47Z)
               BaseCommit: c86269550b6843d4a7e2b27f8ced25079abf6391c07653c47753e5e6be7742d8
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
      RemovedBasePackages: firefox firefox-langpacks 118.0.1-7.fc38 plasma-discover-rpm-ostree plasma-discover plasma-discover-notifier plasma-discover-flatpak 5.27.8-1.fc38
          LayeredPackages: conda conntrack-tools dejavu-fonts-all fzf git gnu-free-mono-fonts ksshaskpass libvirt libvirt-daemon-kvm lm_sensors virt-manager waydroid wireguard-tools

  fedora:fedora/38/x86_64/kinoite
                  Version: 38.20231012.0 (2023-10-12T01:02:34Z)
                   Commit: 77b5fb9cfa654d48839617a237798fa9c2cd91d31e5aa4a86f48f910abd526c4
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

Expected vs actual behavior

Cannot do anything other than rpm-ostree reset. Upgrading, removing packages, installing new packages does not work. After rpm-ostree reset upgrading works, but changing the packages still does not.

rpm-ostree remove waydroid
Inactive requests:
  podman (already provided by podman-5:4.7.0-1.fc38.x86_64)
  nftables (already provided by nftables-1:1.0.5-2.fc38.x86_64)
Checking out tree c862695... done
Resolving dependencies... done
Applying 6 overrides and 197 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
error: Committing: Adding composefs metadata: No data available

Expected:

# rpm-ostree remove waydroid
.....
Success!

Steps to reproduce it

Would you like to work on the issue?

Can only provide more info about the setup.

[cgwalters]

Hmm, must be a regression from #4594
Downgrading to the previous rpm-ostree should fix it, can you confirm?

"No data available" is odd, will look.

What filesystem are you using? i.e. paste the output of stat -f /

[Szwendacz99]

~ # stat -f /
  File: "/"
    ID: 74b629b43ff7215b Namelen: 255     Type: btrfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 249630976  Free: 116566620  Available: 115966172
Inodes: Total: 0          Free: 0

I also have another machine that was following the upgrades similarly, and have the same additional packages installed, but itdoes not have any default packages removed. On that machine upgrade works.

[Szwendacz99]

Damn, I forgot before, but I need to add that lately (cannot tell if after latest successfull update or before) I have canceled the ongoing upgrade (very early in the process) with Ctrl+c

[cgwalters]

I am not totally sure where this error is coming from, but the latest git ostree (soon to show up in https://copr.fedorainfracloud.org/coprs/g/CoreOS/continuous/ will help debug - want to try installing that via usroverlay?)

Is this reproducible 100% of the time?

[Szwendacz99]

The error is happening 100% of the time, but I was able to do rpm-ostree reset and then rpm-ostree upgrade, but even then in the upgraded system I could not uninstall or add new package, so i did rollback (which also worked) to not lose the layered packages which I need. When I tried to deploy older commit it also did not work, but have not tested it together with rpm-ostree reset yet.
Tommorow I might do more tests, including the debug version of rpm-ostree.

[Szwendacz99]

I found out that:

~ # ostree fsck -a
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 213 commit objects...
fsck objects (31035/155646) [==           ]  19%In commits ba4338e09c76b76d46bcb8c970f4a3f46acdcae2a3c32c03ffb70ff423795477, b3fb087d0c2103f9296f575ab1f9914bab4f0e7ce8422d42e3fe536f4ab72753, c86269550b6843d4a7e2b27f8ced25079abf6391c07653c47753e5e6be7742d8: fsck content object 1677f333fd976173d257f3983aa96408302f1b0d01fdeff32a5938deb1acba84: Corrupted file object; checksum expected='1677f333fd976173d257f3983aa96408302f1b0d01fdeff32a5938deb1acba84' actual='73a00af44ecb0706519c27968e7dbbef09a8bc2073e6679341dcf07c38215102'
fsck objects (56318/155646) [====         ]  36%In commits ba4338e09c76b76d46bcb8c970f4a3f46acdcae2a3c32c03ffb70ff423795477, b3fb087d0c2103f9296f575ab1f9914bab4f0e7ce8422d42e3fe536f4ab72753, c86269550b6843d4a7e2b27f8ced25079abf6391c07653c47753e5e6be7742d8: fsck content object 5ffff84f608ab26386b4d883aeba1ac0bfce1aa7f029fe7ec80a9329afea8c54: Corrupted file object; checksum expected='5ffff84f608ab26386b4d883aeba1ac0bfce1aa7f029fe7ec80a9329afea8c54' actual='18f65042bd148628cec694a80de5e4c025e1885c1ec39c15cb24355210cd721c'
fsck objects (126067/155646) [==========   ]  80%In commits ba4338e09c76b76d46bcb8c970f4a3f46acdcae2a3c32c03ffb70ff423795477, b3fb087d0c2103f9296f575ab1f9914bab4f0e7ce8422d42e3fe536f4ab72753, c86269550b6843d4a7e2b27f8ced25079abf6391c07653c47753e5e6be7742d8: fsck content object 73da4c33ba83d5c53de84a82d95d0c39615aa378c907fdbc729f5f7dc7365694: Error reading from file descriptor: Input/output error
fsck objects (155646/155646) [=============] 100%

In the end I managed to fix the system with steps:

• rpm-ostree reset
• rpm-ostree rebase <commit from ~2 weeks ago>
• reboot
• rpm-ostree install <my stuff>
• rpm-ostree upgrade
  and here I am on latest commit of fedora/38/x86_64/kinoite

the problematic commit path is still on my system, since I pinned it.

Not sure if this helps with the mystery (I guess some file/commit was corrupted ), if there is need for some debug info, I could revert temporarily to the problematic deployment, to test stuff, but need info what and how.
Closing for now.

[cgwalters]

Hi, you likely have filesystem corruption. Check dmesg. You may need to reinstall.

[Szwendacz99]

I was checking dmesg before and now, no errors there, btrfs scrub shows no error as well, but btrfsck found something. So this might be thee case.

[moacirrf]

Same error
error: Committing: Adding composefs metadata: Checking out composefs: Processing 55e418cdba0a2e6968669e95bbd3017d40992572ef15199751e648d3f412da3d: No data available

[Szwendacz99]

Same error error: Committing: Adding composefs metadata: Checking out composefs: Processing 55e418cdba0a2e6968669e95bbd3017d40992572ef15199751e648d3f412da3d: No data available

For me it was indeed broken filesystem, but if I was able to fix it with btrfs tools, and then, thanks to rpm-ostree powers I was able to restore system fully, without reinstall, and I am still using the same system installation right now.
Right after that I made small set of tips related to managing the system with ostree/rpm-ostree: https://github.com/Szwendacz99/fedora-immutable-cheatsheet

[moacirrf]

• I did a cleanup as you explain in your cheatsheet
  rpm-ostree cleanup -p -b -r -m
• Rebase from fedora:fedora/39/x86_64/kinoite to fedora:fedora/40/x86_64/kinoite
  rpm-ostree rebase fedora:fedora/40/x86_64/kinoite

Thanks

[J3RN]

I had the same issue and running sudo ostree fsck -a --delete seems to have resolved it for me 👍