37->38 upgrade after previous rollback yields loss of SSH access

[fifofonix]

Describe the bug

SSHD fails to start because of too public host keys.

Reproduction steps

1. Commission an F37 host.
2. Upgrade to F38
3. Rollback to F37
4. Upgrade Again to F38

Expected behavior

SSH access should be retained.

Actual behavior

SSH access is lost.

System details

• vSphere
• Next F37/F38

Butane or Ignition config

No response

Additional information

After step #3 in reproduction steps the /var/lib/.ssh-host-keys-migration stamp is still present causing the key migration service not to run in step #4. Removal of stamp file prior to step #4 allows a clean upgrade and SSH access is available.

Note that although it is fairly evident that SSHD fails in the boot console it is hard to capture details of the reason for failure from the journals when downgrading to F37 because of journal log version incompatibilities. journalctl provides a warning to this effect but its recommended solution did not work for me on the now F37 node. I ended up capturing the journal file it listed and scp'ing to a F38 machine and then used the command given to examine the file there.

[bgilbert]

Thanks for the report. While this issue does involve migration code that was implemented by the Fedora CoreOS WG, Fedora doesn't generally support downgrades between major versions, so you're trying to do something that won't necessarily work. What led you to discover this problem?

[travier]

Related to #1394

[fifofonix]

Thanks for the report. While this issue does involve migration code that was implemented by the Fedora CoreOS WG, Fedora doesn't generally support downgrades between major versions, so you're trying to do something that won't necessarily work. What led you to discover this problem?

I think the CIFS related issues with F38 were the main reason I rolled back a few nodes. The rollback feature is awesome of course and it is a shame that it is not officially supported for major versions because intuitively that is where you would think the most types of issues are likely to be discovered.

[dustymabe]

@bgilbert
Thanks for the report. While this issue does involve migration code that was implemented by the Fedora CoreOS WG, Fedora doesn't generally support downgrades between major versions, so you're trying to do something that won't necessarily work. What led you to discover this problem?

Indeed. Fedora traditionally hasn't supported major downgrades (or even really any downgrades IIUC). The OSTree tech powering FCOS/Silverblue/IoT does give us the ability to roll back to the previous version and it is a feature we often highlight as a benefit of running one of these variants.

While it would be hard to support major downgrades generally (i.e. there is a lot of software we don't control) it would be nice if we could do our best to make it as reliable as possible and/or at least collect known issues.. I can think of two ways for us to try to catch something like this in the future:

• In the very least we could test an upgrade->rollback->upgrade cycle in our Fedora N test day days.
• We could create an automated test that did this when we did new releases.

@fifofonix
I think the CIFS related issues with F38 were the main reason I rolled back a few nodes. The rollback feature is awesome of course and it is a shame that it is not officially supported for major versions because intuitively that is where you would think the most types of issues are likely to be discovered.

Rolling back for the CIFS related issue is a great example of OSTree working to solve a problem for you (the user).

I think what @bgilbert was highlighting was that generally the people who put software in Fedora don't anticipate major downgrades in packages to occur, so it can invalidate assumptions and expose problems. In this particular case the files that were stored in /etc/ were "rolled back" while there was state that was stored in /var/ that prevented the migration script from running on the re-upgrade.

I'm trying to think of a way we can account for this case and maybe we'll ship a fix before F38 hits stable.

[jlebon]

I agree it's hard in the general case to flawlessly support rollbacks if the rest of Fedora doesn't (barring moving to btrfs and snapshotting /var on upgrades :)). I'm comfortable not trying to work against the current and having to deal with e.g. database migration issues manually. In this case though, SSH itself is concerned which might be the only way you have to get into the system, so you don't even have a fighting chance to get out of this manually even if you wanted to.

Hmm, one simple thing we could do I think is to have a systemd unit that runs before ssh-host-keys-migration.service and just rm -f the stamp file? The unit is idempotent, so it's fine if it reruns on an already-migrated system. Then drop it in a barrier where the likelihood that you'd rollback to f37 again is very low (could even be the very next release).

[dustymabe]

We discussed this in the community meeting today.

12:56:35  dustymabe | #agreed we will ship a systemd dropin to remove the stamp file  
                    | ConditionPathExists= on the migration unit so the idempotent
                    | migration code will run on every boot until we remove it after a
                    | barrier release

[dustymabe]

The fix for this went into testing stream release 38.20230414.2.1. Please try out the new release and report issues.

[dustymabe]

The fix for this went into next stream release 38.20230430.1.0. Please try out the new release and report issues.

[dustymabe]

This issue never affected our stable stream.