Fetch via container-image-proxy #98
Conversation
cgwalters
changed the title
| /// The first return value is an `AsyncRead` of that tar file. | ||
| /// The second return value is a background worker task that | ||
| /// owns stream processing. | ||
| pub async fn find_layer_tar( |
There was a problem hiding this comment.
Good riddance to this code!
|
So what I am currently having a huge hellish pain of a time debugging is why after exactly 32768 bytes reading from the child process, the stream gets corrupted. We get exactly the correct length. I haven't figured out yet whether it's the child process outputting the wrong thing, or we're reading the wrong thing. But I think it's the latter. My instinct was to blame something going wrong in this weird manual "hyper over socketpair" that we have going on - I tried extracting that into a standalone thing https://github.com/cgwalters/hyper-self-socketpair but it works there. |
|
It was cgwalters/container-image-proxy@b64f3a0 |
cgwalters
force-pushed
the
import-via-proxy
branch
2 times, most recently
from
56c804b to
9e47e34
Compare
|
OK, this depends on #95 so keeping draft, but this should otherwise be in a merge-able state. Actually shipping this in e.g. rpm-ostree is probably going to depend on cgwalters/container-image-proxy#1 |
cgwalters
force-pushed
the
import-via-proxy
branch
from
9e47e34 to
91cb1e0
Compare
cgwalters
changed the title
|
OK lifting draft! |
cgwalters
force-pushed
the
import-via-proxy
branch
from
91cb1e0 to
c4afae4
Compare
|
Oops, had forgotten we had two manifest fetch paths - changed the other one to stop using skopeo too. (Now that I look there's more cleanup to do here - right now we're tearing down and recreating the proxy, which is inefficient) |
https://github.com/cgwalters/container-image-proxy is prototype code to expose containers/image via a HTTP API suitable for use in non-Go programs. This has many advantages over us forking skopeo; the key one being on-demand layer fetching.
cgwalters
force-pushed
the
import-via-proxy
branch
from
c4afae4 to
678a442
Compare
| #!/bin/bash | ||
| set -xeuo pipefail | ||
|
|
||
| yum -y install skopeo |
There was a problem hiding this comment.
Shouldn't we be able to drop this now that we no longer fork it?
There was a problem hiding this comment.
Right now our CI only runs in the buildroot container which doesn't have skopeo.
At some point though we clearly need to go to the next level and split buildroot and runtime CI flow, whether that's Prow or CoreOS Jenkins (or both), and that leads to things like reusing ostree/rpm-ostree CI etc.
| set -xeuo pipefail | ||
|
|
||
| yum -y install skopeo | ||
| yum -y --enablerepo=updates-testing update ostree-devel |
There was a problem hiding this comment.
Should be able to drop this now.
There was a problem hiding this comment.
Hmm true but OTOH I think let's keep it because I've actually had a new desire to add new API to ostree again, and while this still requires an ostree release, we don't need to block on bodhi at least.
| } | ||
|
|
||
| impl ImageProxy { | ||
| pub(crate) async fn new(imgref: &ImageReference) -> Result<Self> { |
There was a problem hiding this comment.
Hmm, why spawn separate instances per image ref, vs just making the image ref part of the protocol?
There was a problem hiding this comment.
This is more an issue for https://github.com/cgwalters/container-image-proxy/ right? So it actually did originally work that way, but when I generalized it to support all containers/image backends (e.g. supporting oci-archive: and containers-storage: and not just a registry) it became much nicer to have it do only one image at a time.
And the reason for that is HTTP is inherently stateless, but interacting with a containers/image reference is somewhat stateful (e.g. it will transparently convert an oci-archive: to an oci: directory in the background in a tempdir).
There was a problem hiding this comment.
https://github.com/cgwalters/container-image-proxy
is prototype code to expose containers/image via a HTTP API
suitable for use in non-Go programs.
This has many advantages over us forking skopeo; the key one being
on-demand layer fetching.
Closes: #18