Add Afterburn

[cgwalters]

We're going to focus on Ignition, but today CL includes coreos-metadata. Unlike Ignition, it runs every boot; it's obviously very useful, but IMO this blurs the story of "immutable infastructure".

(I personally vote to include it, but we should think about it)

[bgilbert]

@cgwalters coreos-metadata is primarily for fetching metadata such as IP addresses from cloud providers. We need to do that on every boot because e.g. stopping and restarting an EC2 instance will cause new IP addresses to be assigned. Responding to changes in the network environment doesn't seem to me to conflict with the idea of immutable infrastructure. Are you referring to the sshkeys functionality specifically?

It's worth noting that coreos-metadata is currently part of the Ignition ecosystem (via CT dynamic data), unless we decide to change that.

[cgwalters]

Are you referring to the sshkeys functionality specifically?

Yeah.

We need to do that on every boot because e.g. stopping and restarting an EC2 instance will cause new IP addresses to be assigned.

Right, though (AIUI) that functionality was added as a dependency of etcd/locksmith right?

Anyways...my vote is to ship it! But I just want to have some discussion for each bit of functionality.

[jlebon]

Even if it were just for the SSH keys, I'd vote to include it. Those rarely change from boot to boot, so we could argue that changing SSH keys isn't supported, but without coreos-metadata, it won't even work on the first boot IIUC, which is bound to surprise people. So not including it would at least require teaching it to Ignition.

[cgwalters]

It definitely works to install SSH keys with Ignition, without having coreos-metadata. What doesn't work is using the ssh-keys endpoint in cloud metadata APIs. Or to expand on this, picking a SSH key in the EC2 launcher web interface definitely doesn't work as that doesn't know about Ignition.

[jlebon]

Yep, I'm talking about the provider SSH keys too.

(Though playing with it in OpenStack right now, it seems like that's actually handled by coreos-cloudinit instead of coreos-metadata?)

[bgilbert]

@jlebon In CL, [email protected] is enabled on a platform-by-platform basis.

@cgwalters Yeah, if we require that users provide an Ignition config, lots of people will trip over that.

Right, though (AIUI) that functionality was added as a dependency of etcd/locksmith right?

It can be used to configure etcd, but it can also be used to configure arbitrary other services.

[dcode]

coreos-metadata.service is pretty important for me to support testing across baremetal and various cloud systems. I merely change the provider flag of ct to generate the appropriate ignition config. I've written my other systemd units passed through to use the same variable construct. Without it, self-discovery of a public-facing IP may not be easily possible.

[jlebon]

I changed the title of this issue to reflect the outcome in openshift/os#125 (comment).

[dustymabe]

related work items we need to complete as part of this:

• packaging up coreos-metadata as an rpm (some work has been done by @lucab already for this)
• Add mechanism for reporting instance health to cloud platforms

[dustymabe]

It seems like we aren't too far off from having all source rpms needed to push this over the edge and not bundle.

• base64 https://src.fedoraproject.org/rpms/rust-base64
• byteorder https://src.fedoraproject.org/rpms/rust-byteorder
• hyper https://src.fedoraproject.org/rpms/rust-hyper
• ipnetwork not in fedora
• mockito not in fedora
• nix https://src.fedoraproject.org/rpms/rust-nix
• openssh-keys - on us to package
• openssl https://src.fedoraproject.org/rpms/rust-openssl
• reqwest not in fedora
• serde_json https://src.fedoraproject.org/rpms/rust-serde_json
• users https://src.fedoraproject.org/rpms/rust-users

[rfairley]

Updated title for coreos-metadata -> Afterburn rename.

[rfairley]

Fedora package review request for Afterburn: https://bugzilla.redhat.com/show_bug.cgi?id=1703542

[rfairley]

Opened coreos/fedora-coreos-config#93 to add this into FCOS, which also enables the afterburn-checkin.service and afterburn-firstboot-checkin.service units.

For [email protected], we want this enabled in FCOS, but I think we'd want to enable this somewhere in the COSA buildextend-* scripts here https://github.com/coreos/coreos-assembler/tree/master/src? (with it being enabled/disabled depending on the platform)

Enabling it on FCOS and cosa run gives the following:

$ journalctl --no-pager -u [email protected] 
-- Logs begin at Tue 2019-05-14 15:16:19 UTC, end at Tue 2019-05-14 15:16:33 UTC. --
May 14 15:16:26 localhost.localdomain systemd[1]: Starting Afterburn (SSH Keys)...
May 14 15:16:26 localhost.localdomain afterburn[954]: Error: fetching metadata from provider
May 14 15:16:26 localhost.localdomain afterburn[954]: Caused by: unknown provider 'qemu'
May 14 15:16:26 localhost.localdomain systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
May 14 15:16:26 localhost.localdomain systemd[1]: [email protected]: Failed with result 'exit-code'.
May 14 15:16:26 localhost.localdomain systemd[1]: Failed to start Afterburn (SSH Keys).

I notice in coreos-overlay for CL this gets enabled through an Ignition base config, e.g. https://github.com/coreos/coreos-overlay/blob/master/coreos-base/oem-digitalocean/files/base/base.ign.

[dustymabe]

For [email protected], we want this enabled in FCOS, but I think we'd want to enable this somewhere in the COSA buildextend-* scripts here

could we just use ConditionKernelCommandLine= in the unit to only enable it on certain values of ignition.platform.id

[rfairley]

could we just use ConditionKernelCommandLine= in the unit to only enable it on certain values of ignition.platform.id

Ahh yes that'd make sense, just like the *checkin.service units are doing.

[rfairley]

With coreos/fedora-coreos-config#97 merged, this completes enabling Afterburn functionality in FCOS.

Marking as done (please re-open if you think of something else missing).

[dustymabe]

With coreos/fedora-coreos-config#97 merged, this completes enabling Afterburn functionality in FCOS.

Thanks @rfairley