docker/internal/tarfile has incorrect media type in manifests; copy layer format is not manifest-format dependent #1270
Labels
kind/bug
A defect in an existing functionality (or a PR fixing it)
Comments
|
Thanks for your report.
No, using the existing one is what the code currently intends to do. Note to self: It looks like Either way, please collect the full debug log to show what exactly happens here. |
|
@mtrmac Full Debug Logs: |
|
Thanks, so this is the direct “blob already exists at destination” path with no |
mtrmac
changed the title
mtrmac
changed the title
|
Hum, so runs hard against the fact that the uncompressed MIME type is not specified as valid in schema2 manifests, and at least Quay.io validates that. E.g. using a digest reference to ensure the uncompressed version % bin/skopeo docker-archive:foo.tar docker://quay.io/mitr/test-archive@sha256:881940c6a398ffeb50f36cb9ce070109e8c069eb67b6adaafa0de890df697d51
… (successfully uploads the uncompressed data)
FATA[0097] Error writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":1316,\"digest\":\"sha256:abec9a7a7dc640768e6dc51b6a5728e470411615c62e9ff46215206bde816772\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":183715840,\"digest\":\"sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586\"}]}": Error uploading manifest sha256:881940c6a398ffeb50f36cb9ce070109e8c069eb67b6adaafa0de890df697d51 to quay.io/mitr/test-archive: manifest invalid: manifest invalid (where the digest is the digest of the artificial manifest, obtained by and afterwards % bin/skopeo copy docker-archive:foo.tar docker://quay.io/mitr/test-archive:after
…
DEBU[0000] Checking /v2/mitr/test-archive/blobs/sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586
DEBU[0001] ... already exists
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/mitr/test-archive/manifests/after
DEBU[0002] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":1316,\"digest\":\"sha256:abec9a7a7dc640768e6dc51b6a5728e470411615c62e9ff46215206bde816772\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":183715840,\"digest\":\"sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586\"}]}": Error uploading manifest after to quay.io/mitr/test-archive: manifest invalid: manifest invalid
DEBU[0002] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws…
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/mitr/test-archive/manifests/after
…i.e. Quay.io rejects the uncompressed value, and that causes a fallback to schema1. On balance, it seems distinctly more useful to use schema2 with an incorrect MIME type (which is, in practice, quite interoperable) than to fall back back to schema1, where the digest changes with each registry/repository change. OTOH it’s extremely unclean that we currently achieve the more useful result by such an indirect approach. The fundamental difficulty is that we need to express a Docker schema2 image (because the config in docker/internal/tarfile is a schema2 config), but with uncompressed layers — which just doesn’t exist in the spec; but the rest of the copy code makes a presumption that the input representation is the ~preferred one and should not be changed frivolously, and that layer reuse is always better than layer upload. Implementation artifacts involved:
I can’t think of a simple fix. We’ll probably in some vague future have to teach the copy code about manifest-format-dependent layer formats, but that’s non-trivial work and it has serious downsides as well — if we upload the layer in one format, then try uploading the manifest, and that fails, deleting blobs might not even be possible. I’ll keep this open for how, hoping that someone else can come up with a good solution, or just to keep this in mind for the future. |
|
@mtrmac Looks like there is no easy solution. But are there any workarounds to force it to upload the compressed layer or have the manifest be correct? |
|
Depending on the registry, pushing to a different repository (where the uncompressed layer does not yet exist) might work — or it might find it anyway; an inter-repo copy would then be easy. Or use an intermediate |
|
Would it help to do |
|
Yes, to the extent that the MIME type will be “uncompressed” (but Quay.io will reject that and cause a downgrade to schema1.) But the uncompressed version will be preferred if it already exists on the registry, if the process is not already using |
|
… but that’s a fair point; if c/storage uses the uncompressed manifests, it’s more attractive for the tarfile transports ( |
mtrmac
added
kind/feature
|
After #2068 , users should be able to force layer compression instead of reuse of uncompressed layers. The underlying design issue remains outstanding. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When Skopeo copies an image to our Artifactory Docker repo the first layer
5f70bf18a086is not compressed but has type gzip in the manifests. I noticed during the copy it sees that the first layer is already in Artifactory so doesn't copy it again but it's the uncompressed SHA, not sure if that is related? But I'd expect it to compress the layer and upload that layer instead of using the already existing uncompressed one.Skopeo Version:
skopeo version 1.3.0Skopeo copy command:
skopeo copy --debug --retry-times 3 --dest-creds ****:**** --sign-by <redacted> docker-daemon:<local_image> docker://<image>Logs:
--
$ skopeo inspect --raw --creds "${SKOPUSER}":"${SKOPPASS}" docker://<image> | jq{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 4686, "digest": "sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 1024, "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 15192304, "digest": "sha256:1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 556, "digest": "sha256:5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 180, "digest": "sha256:15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 176, "digest": "sha256:06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 403, "digest": "sha256:044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 315, "digest": "sha256:ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82" } ] }Downloaded Manifest file:
{ "schemaVersion": 2, "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:d1f06d5ae426406b9a4828dca9987c92b65b3fbb76b979cea0bb35bdad3b085d", "size": 2922 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", "size": 1024 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259", "size": 15192304 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421", "size": 556 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede", "size": 180 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293", "size": 176 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc", "size": 403 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82", "size": 315 } ] }5f70bf18a086is not compressedThe text was updated successfully, but these errors were encountered: