Support updating from local mirrors

[bgilbert]

Many Fedora CoreOS nodes will update directly from our Cincinnati server and ostree repo, but there are two other use cases we should support:

1. Nodes updating from a local mirror on an internal network. The mirror has access to the Internet but the nodes may not.
2. As above, but the mirror doesn't have Internet access either. Updates are shuttled to the mirror via other means.

We'll need tooling and docs for each.

EDIT(lucab): related but slightly different, #261 covers the "manual updates to air-gapped instances" scenario.

[lucab]

I'm starting to add notes here, so we can more easily collect them into docs.

To use a customized/mirrored updates backend, at least all of the following topics have to be covered:

1. running an rpm-ostree mirror, and how to mirror/proxy ostree content for our releases
2. running a Cincinnati backend, and how to mirror/proxy our update-graph
3. overriding the default repository in rpm-ostree on each node
4. overriding the default Cincinnati endpoint in Zincati on each node

[cgwalters]

xref https://discussion.fedoraproject.org/t/zincati-version-control/19850

[3d-pro]

Hi, any updates on this? thanks.

[dustymabe]

I don't have any updates. Luca may have made some progress on the items he mentioned in #240 (comment). Will let him respond when he gets back.

[dkliban]

Pulp (https://pulpproject.org) supports mirroring repositories locally. Pulp also supports exporting repositories to a physical media and then importing them from that media in a disconnected environment. While Pulp does not currently support OSTree repositories, we would like to add OSTree support in the near future. Please file a story at https://pulp.plan.io/issues/new/ outlining the OSTree repository mirroring use case. Having clear use cases will help us plan and execute this effort.

[cverna]

Linking #812 which should make it easier to support updating from a local registry

[cgwalters]

As of recently, https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md is the successor to #812

One thing I'd note here is that as of rpm-ostree v2021.14 in Fedora, one can use ex-container encapsulate to mirror from the current production ostree repository into a container image, push that to an internal registry, and have a separate process which de-encapsulates for clients to use as an update source.

But eventually, the goal of coreos layering is that we just update by default from a container image - and that the container-registries config for mirroring just works for OS updates too.

[jlebon]

For the subcase where the nodes have access to the Internet, but a local mirror is desired to reduce bandwidth, a simple OSTree strategy is:

1. On the node doing the mirroring, have a systemd timer + service which does e.g. ostree pull fedora:fedora/x86_64/coreos/stable into an archive repo.
2. On nodes that want mirrored content, modify the contenturl key of the OSTree remote to point to the archive repo on the mirror. (Or add a separate remote with that modification and rpm-ostree rebase to it.)

(I.e. this is 1 and 3 of #240 (comment))

[buckaroogeek]

Suggest adding some discussion on the effect of different --depth values

[htcosta]

Any update on this ?

Best regards.