[RFE] Support upgrade from RHEL 9 to 10

[swapdisk]

As announced here, RHEL 10 Beta is now available and "mid-2025" is the target date for GA release.

To prepare for this, let's start working on making the roles support upgrades from RHEL 9 to 10. For now, these upgrades would target RHEL 10 Beta just for testing and demonstration use, but the ultimate goal is to be ready on day one after the RHEL 10 GA release.

This work will happen under the rhel_9_to_10 branch of my infra.leapp fork. If you are interested in contributing, I'll happily review your pull requests there!

We'll be posting updates on the progress under this issue. The first big milestone with be to document here how to try it out when we have something reasonably ready for testing.

Cheers!

[swapdisk]

Amazingly, only very minor changes required to make the collection roles work for upgrading from RHEL 9 to RHEL 10 Beta. See the diff under commit 63de955 on my rhel_9_to_10 branch.

What follows are the steps that worked for me to get to RHEL 10 Beta using the rhel_9_to10 branch of the collection. There were a some minor issues requiring a couple of sneaky extra tasks in my test playbooks, but we have demonstrated a successful RHEL 10 upgrade using Ansible automation and the infra.leapp collection.

The first step is we need to get a RHEL 9 instance to upgrade. I used Insights image builder to create a VM on my laptop with a vanilla 9.5 install automatically subscribed to Red Hat repos. You could also install RHEL 9 using an ISO, but it will need to be registered using subscription-manager for this to work.

When you have your RHEL 9 host ready to test, the next step is to run an analysis job. Unfortunately, there are no Leapp packages yet in Red Hat's RHEL 9 GA repos. So to get around that, we are using the "beta" Leapp packages available from the public upstream COPR repo for Leapp. You can see how I configured that repo in the playbook I used to automate my analysis job:

---
- name: Analysis
  hosts: sweet-bmader
  strategy: free
  gather_facts: true
  become: true
  force_handlers: true
  vars:
    leapp_upgrade_type: cdn
    leapp_env_vars:
      LEAPP_UNSUPPORTED: 1
      LEAPP_DEVEL_DATABASE_SYNC_OFF: 1
      LEAPP_DEVEL_TARGET_PRODUCT_TYPE: beta

  pre_tasks:
    - name: Configure upstream COPR repository for Leapp
      ansible.builtin.get_url:
        url: https://copr.fedorainfracloud.org/coprs/g/oamg/leapp/repo/epel-9/group_oamg-leapp-epel-9.repo
        dest: /etc/yum.repos.d/oamg.repo
        mode: '0644'

  roles:
    - infra.leapp.analysis
...

You can see we also configured some env vars for the Leapp run. Setting LEAPP_DEVEL_DATABASE_SYNC_OFF=1 is convenient for testing as it makes Leapp run faster. The LEAPP_DEVEL_TARGET_PRODUCT_TYPE=beta setting is required to inform Leapp to use the RHEL 10 Beta repos for the upgrade. Finally, LEAPP_UNSUPPORTED=1 is required to acknowledge that using LEAPP_DEVEL_* vars is not supported. Continue at your own risk!

Here's the log output from the new task added in the rhel_9_to_10 branch of the analysis role to install the RHEL 9 Leapp packages:

TASK [infra.leapp.analysis : analysis-leapp | Install packages for preupgrade analysis on RHEL 9] ***
changed: [sweet-bmader] => {
  "changed": true,
  "msg": "",
  "rc": 0,
  "results": [
    "Installed: leapp-0.18.0-100.20250108212821781602.main.7.g52cd433.el9.noarch",
    "Installed: leapp-upgrade-el9toel10-0.21.0-100.20250109205304713680.main.44.gc92dbb41.el9.noarch",
    "Installed: leapp-deps-0.18.0-100.20250108212821781602.main.7.g52cd433.el9.noarch",
    "Installed: libdb-utils-5.3.28-54.el9.x86_64",
    "Installed: leapp-upgrade-el9toel10-deps-0.21.0-100.20250109205304713680.main.44.gc92dbb41.el9.noarch",
    "Installed: systemd-container-252-46.el9_5.2.x86_64",
    "Installed: python3-leapp-0.18.0-100.20250108212821781602.main.7.g52cd433.el9.noarch"
  ]
}

The analysis job finished successfully, so let's check /var/log/leapp/leapp-report.txt on the host. Much to my surprise, there were no inhibitors listed. Having no inhibitors, the next step is to try an upgrade job. Remember, "fail fast" and then have coffee!

Here we hit an issue where the SHA1 rpm signing public keys in the RHEL 9 image are incompatible with upgrading to RHEL 10. This is a known issue and the Leapp team is working in fixing it under private bug RHEL-71517. In the meantime, we can get around the issue by manually importing the SHA256 generation of the keys. Of course, we can easily automate this fix as seen in my test playbook for running the upgrade:

---
- name: Upgrade
  hosts: sweet-bmader
  strategy: free
  gather_facts: true
  become: true
  force_handlers: true
  vars:
    leapp_upgrade_type: cdn
    leapp_env_vars:
      LEAPP_UNSUPPORTED: 1
      LEAPP_DEVEL_DATABASE_SYNC_OFF: 1
      LEAPP_DEVEL_TARGET_PRODUCT_TYPE: beta

  pre_tasks:
    - name: Workaround for rpm gpg-pubkey issue
      ansible.builtin.shell: |
        /bin/cp /var/lib/leapp/el10userspace/etc/pki/rpm-gpg/* /etc/pki/rpm-gpg/
        /bin/rpm -e gpg-pubkey-fd431d51-4ae0493b
        /bin/rpm -e gpg-pubkey-f21541eb-4a5233e8
        /bin/rpmkeys --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
        /bin/rpmkeys --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta

  roles:
    - infra.leapp.upgrade
...

Using this playbook, the run successfully executed the Leapp upgrade and our host came up running a RHEL 10 kernel. These commands provides our proof of the successful upgrade:

[root@sweet-bmader ~]# last -w reboot
reboot   system boot  6.11.0-0.rc5.23.el10.x86_64 Fri Jan 10 12:19   still running
reboot   system boot  6.11.0-0.rc5.23.el10.x86_64 Fri Jan 10 12:19 - 12:19  (00:00)
reboot   system boot  5.14.0-503.21.1.el9_5.x86_64 Fri Jan 10 12:06 - 12:15  (00:09)
reboot   system boot  5.14.0-503.19.1.el9_5.x86_64 Fri Jan 10 10:31 - 12:06  (01:35)

wtmp begins Fri Jan 10 10:31:00 2025
[root@sweet-bmader ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 10.0 Beta (Coughlan)
[root@sweet-bmader ~]# dnf repolist
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

repo id                                         repo name
copr:copr.fedorainfracloud.org:group_oamg:leapp Copr repo for leapp owned by @oamg
rhel-10-for-x86_64-appstream-beta-rpms          Red Hat Enterprise Linux 10 for x86_64 - AppStream Beta (RPMs)
rhel-10-for-x86_64-baseos-beta-rpms             Red Hat Enterprise Linux 10 for x86_64 - BaseOS Beta (RPMs)

So where does that leave us. Yes, we had to fiddle with a couple fun Leapp issues, but hey, it's beta. The good news is we learned that with only some minor changes, the infra.leapp collection will be able to handle Leapp upgrades from RHEL 9 to 10 when Red Hat makes the support for that GA.

Thanks for riding along.

Cheers!

[swapdisk]

We're following PR oamg/leapp-repository#1325 which should be fixing the gpg-pubkey issue discussed in the previous comment.