pushing to a Gitlab registry fails with Unauthorized

[mschrupp]

Hi everyone,

I'm very interested in this project (thanks for your work) and I'm currently testing it with gitlab.

Now I provided a .docker/config.json that would allow me to login.
This works fine for pulling the image referenced in FROM in the Dockerfile.
When kaniko tries to push, it fails:
ERRO[0001] UNAUTHORIZED: "authentication required"

I've tracked the issue down to this line:

kaniko/pkg/executor/executor.go

Line 185 in d2fc49f

return remote.Write(destRef, image, pushAuth, http.DefaultTransport, wo)

where remote.Write is called to push the image.

Now, in remote.Write, the scopes are assembled:

https://github.com/google/go-containerregistry/blob/ee5a6c257df843b47a2666ff0fff3d31d484ebda/v1/remote/write.go#L45

I logged the scopes and received the following:
[repository:MY_GROUP/MY_PROJECT:push,pull repository:MY_GROUP/MY_PROJECT:pull]

I first thought it looks OK, but I think the second scope, which only has pull permissions, somehow overwrites the first.
The second scope comes from the executor again, where a single MountPath is set:

kaniko/pkg/executor/executor.go

Line 179 in d2fc49f

wo.MountPaths = []name.Repository{ref.Context()}

After commenting that line out, it worked like a charm.

I'm not sure why that MountPath is neccessary or what it is for.
Thanks for any help! :)

[jonjohnsonjr]

The MountPaths are useful for doing cross-repo mounting, which allows fast intra-registry copies. See here:
https://docs.docker.com/registry/spec/api/#cross-repository-blob-mount

I believe this is failing because gitlab assumes a single scope per token exchange, but there can be multiple. See here:
https://github.com/docker/distribution/blob/1503fa210962ee023c09a319b96b86ebf2d92991/docs/spec/auth/scope.md#authorization-server-use

Multiple resource scopes may be provided using multiple scope fields on the GET request.

And here:
https://github.com/docker/distribution/blob/1503fa210962ee023c09a319b96b86ebf2d92991/docs/spec/auth/token.md#requesting-a-token

[mschrupp]

@jonjohnsonjr thanks for the explanation!

This seems to be a gitlab issue, I will try to find the right people there

[azazel75]

@jesusofsuburbia did you open an issue on gitlab that I can follow?

[mschrupp]

@azazel75 not yet, I wanted to research a bit where this problem happens in the gitlab code first, but didn't have the time...

[azazel75]

@jesusofsuburbia They surely know their code better and can help us spot and fix the problem

[mschrupp]

@azazel75 feel free to open an issue

[azazel75]

Of course, of course

[pao]

Have either of you opened the issue? I don't see a relevant one over there yet.

[pao]

Cross-reference for this specific issue (multiple authentication scopes): https://gitlab.com/gitlab-org/gitlab-ce/issues/48968