Skip to content
This repository has been archived by the owner on Jan 15, 2025. It is now read-only.

container: Work on new skopeo backend #18

Closed
cgwalters opened this issue Apr 26, 2021 · 4 comments · Fixed by #98
Closed

container: Work on new skopeo backend #18

cgwalters opened this issue Apr 26, 2021 · 4 comments · Fixed by #98

Comments

@cgwalters
Copy link
Member

The way we fork skopeo as a subprocess has a lot of flaws.

See: containers/image#1209
@cgwalters
Copy link
Member Author

I think as a PoC, we can make a new sub-project like ostreedev/ostree-container-fetcher or something that vendors containers/image and does what we need.

The ideal then would be to fold that code back into skopeo; something like e.g. skopeo registry-proxy that can speak HTTP over a unix domain pipe. We still want it to do things like signature verification and layer digest verification, but otherwise fetching and parsing the blobs is via us doing pulls.

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Sep 20, 2021
@cgwalters
lib/container: Convert to oci-archive, not docker-archive:
44bff1b 
The use of `docker-archive:` originally came from
ostreedev#18
but it turns out I was wrong, skopeo ends up staging everything
anyways.

In practice, what we really want to do here is land custom code
in skopeo to do what we need, instead of treating this as a `copy`
operation, because it isn't.

Relatedly, I also noticed in our `docker-archive:` parsing that we weren't
verifying the relationship of the layer symlink to the sha256 digest;
we were only verifying the *presence* of the symlink.
In practice, skopeo will already be doing the layer verification for
us, but this mechanism is clearer.

Prep for derivation.
@cgwalters cgwalters mentioned this issue Sep 20, 2021
Closed
@cgwalters
Copy link
Member Author

https://github.com/cgwalters/container-image-proxy now exists

@cgwalters
Copy link
Member Author

Some extremely hacky WIP on this in https://github.com/cgwalters/ostree-rs-ext/commits/import-via-proxy

@cgwalters
Copy link
Member Author

cgwalters commented Sep 22, 2021 •
edited
Loading

OK I thought this didn't work, but it turns out it's racy. The race appears to be somewhere in the Rust async code around buffering, but I haven't figured out where yet. Debugging this has been extremely painful. I'm getting burned by lack of unit tests at different places in the pipeline, so working on that now.

@cgwalters cgwalters mentioned this issue Sep 22, 2021
Merged
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 6, 2021
@cgwalters
Add new experimental-image-proxy hidden command
091cb6d 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
@cgwalters cgwalters mentioned this issue Oct 6, 2021
Merged
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 6, 2021
@cgwalters
Add new experimental-image-proxy hidden command
88c692d 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 9, 2021
@cgwalters
Add new experimental-image-proxy hidden command
8d51fcb 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 12, 2021
@cgwalters
Add new experimental-image-proxy hidden command
e799dd6 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 12, 2021
@cgwalters
Add new experimental-image-proxy hidden command
df0a8a4 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 12, 2021
@cgwalters
Add new experimental-image-proxy hidden command
bd41021 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 12, 2021
@cgwalters
Add new experimental-image-proxy hidden command
11737ee 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 12, 2021
@cgwalters
Add new experimental-image-proxy hidden command
d714718 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 13, 2021
@cgwalters
Add new experimental-image-proxy hidden command
de8fbe6 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 13, 2021
@cgwalters
Add new experimental-image-proxy hidden command
8bf95e6 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 13, 2021
@cgwalters
Add new experimental-image-proxy hidden command
34e67e5 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 13, 2021
@cgwalters
Add new experimental-image-proxy hidden command
786c178 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 13, 2021
@cgwalters
Add new experimental-image-proxy hidden command
16171ce 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
cgwalters added a commit to cgwalters/skopeo that referenced this issue Oct 14, 2021
@cgwalters
Add new experimental-image-proxy hidden command
11b5989 
This imports the code from https://github.com/cgwalters/container-image-proxy

First, assume one is operating on a codebase that isn't Go, but wants
to interact with container images - we can't just include the Go containers/image
library.

The primary intended use case of this is for things like
[ostree-containers](ostreedev/ostree-rs-ext#18)
where we're using container images to encapsulate host operating system
updates, but we don't want to involve the [containers/image](github.com/containers/image/)
storage layer.

Vendoring the containers/image stack in another project is a large lift; the stripped
binary for this proxy standalone weighs in at 16M (I'm sure the lack
of LTO and the overall simplicity of the Go compiler is a large factor).
Anyways, I'd like to avoid shipping another copy.

This command is marked as experimental, and hidden.  The goal is
just to use it from the ostree stack for now, ideally shipping at least
in CentOS 9 Stream relatively soon.   We can (and IMO should)
change and improve it later.

A lot more discussion in cgwalters/container-image-proxy#1
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fetch via container-image-proxy cgwalters/ostree-rs-ext
1 participant
@cgwalters