RPMOSTreePayload

[Promaethius]

Feature Request

Ignition API for a Similar OSTree deployment flow to Anaconda

Environment

Any/All outside of containers.

Desired Feature

Right now, Fedora CoreOS is heading in a direction to use kickstart files with Fedora Everything and Anaconda to deploy OSTrees and configure the bootloader. The API for that process is documented over at https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/payload/rpmostreepayload.py

To cut out the middleman, and reduce installer artifacts, would it be useful to program a similar logic into Ignition? rpmostreepayload.py logic is very well documented and might not take major work to integrate. Additionally, since Ignition runs during initramfs, it'd be the perfect time to init a new filesystem with an OSTree repo.

Other Information

The physical host portion of this corresponds to coreos/coreos-assembler#91 and to the general assembler flow of https://github.com/coreos/coreos-assembler

[ajeddeloh]

This is on my mind as well (I was literally looking at the rpmostreepayload.py file earlier today to see how anaconda handles ostree). We want to move away from using anaconda for image creation for *COS systems. We started with anaconda because it was the easiest path to get started. However, I don't think baking all of the things anaconda does for *COS into Ignition is the right path.

My vision for image creation is to let Ignition handle the partitioning and filesystem creation and then set up the ostree, bootloader, and anything else that needs to happen (shouldn't be much if anything) via a simple bash script. This could all happen in a supermin vm or using loop devices (depending on how much you trust loop devices, what your isolation concerns are, etc). If we want to include a default user we could even use Ignition for that.

One of Ignition's goals is to remain distro agnostic. Adding support for rpm-ostree would break that.

Note that image creation and ostree initialization in the initramfs are separate events that can be implemented independently. The original plan was *COS to ship with a populated root, but detect if Ignition's disk stage blew it away and recreate it only if necessary (see coreos/fedora-coreos-tracker#34).

[dustymabe]

Does coreos/fedora-coreos-tracker#34 overlap with what you're asking for?

[dustymabe]

oops didn't see andrew's comment before I replied.

[bgilbert]

For context, since I was confused at first: I think this bug is talking about how the OS image is built, and not how it's deployed? FCOS will not support deployment via Anaconda or kickstart; provisioning will consist of running Ignition on the first boot of a prebuilt image.

[Promaethius]

Does coreos/fedora-coreos-tracker#34 overlap with what you're asking for?

It does look similar. For a while, I had been crawling through code because of that issue I referenced in coreos-assembler, I kept thinking there must be an easier way to create disk images for ostree other than anaconda/ks. Found answers in the ostree man pages and the rest from how anaconda actually handles the ostreesetup command.

This is on my mind as well (I was literally looking at the rpmostreepayload.py file earlier today to see how anaconda handles ostree). We want to move away from using anaconda for image creation for *COS systems. We started with anaconda because it was the easiest path to get started. However, I don't think baking all of the things anaconda does for *COS into Ignition is the right path.

My vision for image creation is to let Ignition handle the partitioning and filesystem creation and then set up the ostree, bootloader, and anything else that needs to happen (shouldn't be much if anything) via a simple bash script. This could all happen in a supermin vm or using loop devices (depending on how much you trust loop devices, what your isolation concerns are, etc). If we want to include a default user we could even use Ignition for that.

Note that image creation and ostree initialization in the initramfs are separate events that can be implemented independently. The original plan was *COS to ship with a populated root, but detect if Ignition's disk stage blew it away and recreate it only if necessary (see coreos/fedora-coreos-tracker#34).

Totally agree with this since after looking at lorax image creator, it just seemed like a lot of overhead and artifacts to go through extra build steps. After reading your issue, it looks like we both have the same idea of iPXE serving only the vmlinuz and initramfs images and allowing ignition to part and init root with remote ostree repos.

One of Ignition's goals is to remain distro agnostic. Adding support for rpm-ostree would break that.

Even in the rpmostreepayload code, it does not reference ostree specifically as rpm-ostree. In Ignition, the API can go agnostic and either force some compliance or use logic with the efi and grub2 (similar to how rpmostreepayload performs it L133 )

I think this would drastically reduce the work that coreos-assembler has to do since it'll be working mainly from the ostree repo boot images and filesystem. After partitioning is done, it'll pull the repo, and create users. Kinda falls within the use case of Ignition: "Even though Ignition only runs once, it packs a powerful punch. Because Ignition runs so early in the boot process (in the initramfs, to be exact), it is able to accomplish configuration all before the userspace has begun booting, which enables advanced features that Container Linux admins require."

[bgilbert]

After reading your issue, it looks like we both have the same idea of iPXE serving only the vmlinuz and initramfs images and allowing ignition to part and init root with remote ostree repos.

The initramfs will include a full root filesystem image, as CL does now. The ability to replace the ostree from inside the initramfs is useful in a couple cases, such as reformatting the root filesystem, but isn't required for FCOS to work.

[Promaethius]

The initramfs will include a full root filesystem image, as CL does now. The ability to replace the ostree from inside the initramfs is useful in a couple cases, such as reformatting the root filesystem, but isn't required for FCOS to work.

True. Performing ostreesetup during initramfs does mean that FCOS can ship with a smaller initramfs and pull from an upstream repo for the latest distribution at init.

[ajeddeloh]

Well, not necessarily the latest. You need to ensure that rootfs matches the initramfs and kernel.

I'm gunna close this since it seems like we're all on the same page that Ignition should not be handling ostree in any way. For discussion on FCOS in a more general sense see https://github.com/coreos/fedora-coreos-tracker

[bgilbert]

Performing ostreesetup during initramfs does mean that FCOS can ship with a smaller initramfs and pull from an upstream repo for the latest distribution at init.

Conversely, it means the ostree would be downloaded once per boot per machine, rather than once per cluster per release.

[Promaethius]

Conversely, it means the ostree would be downloaded once per boot per machine, rather than once per cluster per release.

Within the same ostree operation, I was imagining it'd also setup the ostree remote for updates cause Ignition is only supposed to run once during a machine's lifetime.

I'm gunna close this since it seems like we're all on the same page that Ignition should not be handling ostree in any way. For discussion on FCOS in a more general sense see https://github.com/coreos/fedora-coreos-tracker

I guess so... there's a lot I'm still learning about this process and it's utterly fascinating so thank you for addressing my questions.