error checking push permissions when pushing to insecure registry #702

CeadeS · 2019-06-20T07:05:21Z

Actual behavior
When pushing to an insecure registry, process fails with the message:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again
: checking push permission for "adress:port/name:tag": Get https://server:port/v2/: http: server gave HTTP response to HTTPS client

Expected behavior
Kaniko builds and pushes Image to registry. Still working at v0.9.0

To Reproduce
Steps to reproduce the behavior:

config.yaml
...
args: ["--dockerfile=Dockerfile",    
            "--insecure-registry=YOUR_OWN_INSECURE_REGISTRY",
            "--destination=AN_EXISTING_DESTINATION",            
            "--insecure"]
...

The text was updated successfully, but these errors were encountered:

vijaySamanuri · 2019-06-20T07:58:18Z

I hit with the same issue when using docker image by tag latest (updated 13 hours ago).
I noticed issue with hostname resolution.

cn-sunletian · 2019-07-02T08:26:57Z

Look at
Duplicate of #683

priyawadhwa · 2019-09-27T17:57:31Z

Hey @CeadeS could you try again with gcr.io/kaniko-project/executor: 51734fc3a33e04f113487853d118608ba6ff2b81 ? I'm wondering if #685 will fix your issue.

tejal29 · 2019-09-27T22:19:09Z

@CeadeS and @CeadeS Can you please confirm?

tejal29 · 2020-03-18T05:46:34Z

This is now fixed. thank you for your patience.

pvgbabu · 2020-07-20T16:15:48Z

@tejal29 , how did you fixed this issue, could you let us know

just4est · 2020-09-17T08:32:29Z

So??? How to fix this sh... ? There is no answer #1179 to

tejal29 · 2020-09-18T03:35:43Z

@just4est and @pvgbabu #685 should respect --insecure flag even if the --insecure-registries does not contain the destination image registry.
#663 should skip tls verficaition for your insecure registry.
Can you try --insecure and --skip-tls-verify ?

tchellomello · 2021-01-01T20:05:56Z

Hey @CeadeS could you try again with gcr.io/kaniko-project/executor: 51734fc3a33e04f113487853d118608ba6ff2b81 ? I'm wondering if #685 will fix your issue.

I was hitting the same issue and that image 51734fc3a33e04f113487853d118608ba6ff2b81 fixed the problem!

   steps:
     - name: build-and-push
       # https://github.com/GoogleContainerTools/kaniko/issues/702
       image: gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81
       #image: gcr.io/kaniko-project/executor:latest
       #image: gcr.io/kaniko-project/executor:debug
       # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
       env:
         - name: "DOCKER_CONFIG"
           value: "/tekton/home/.docker/"
       command:
         - /kaniko/executor
       args:
         - --dockerfile=$(params.pathToDockerFile)
         - --destination=$(params.registryURL)/$(params.imageOwner)/$(params.imageName):$(params.imageTag)
         - --context=$(params.pathToContext)
         - --insecure
         - --skip-tls-verify
         - --skip-tls-verify-pull
         - --insecure-pull

luisgc93 · 2021-10-20T13:58:35Z

Just a heads up that the latest kaniko version (v1.7.0 ) seems to be causing this problem for my team:
https://github.com/GoogleContainerTools/kaniko/releases

Downgrading to v1.6.0 fixed the problem.

hamko · 2021-10-20T16:58:31Z

@tejal29 Internally at Google, I see multiple high-tier GCP users report the same issue with the latest version. Could you find someone to take a look at this?

peter-dough · 2021-10-20T18:50:26Z

I saw some references in the release notes mentioning that some authentication binaries were removed for the slim executor images? Could it be that it was removed across the board?

vitdolphin · 2021-10-20T19:15:02Z

i have the same issue now on the pipeline that worked few days ago.

austinpray-mixpanel · 2021-10-20T22:31:49Z

I'm seeing this as well. And downgrading to 1.6.0 (gcr.io/kaniko-project/executor@sha256:0fae223f496525e31226cde473ec77ed15abfa8cbabff5ff5bf1c5268355bbb0) restored functionality.

sigitisme · 2021-10-22T09:19:52Z

yes using 1.6.0, solved my issue

imjasonh · 2021-10-22T13:48:45Z

In case it helps, here's a diff of the files in v1.6.0 and 1.7.0, using crane:

$ diff <(crane export gcr.io/kaniko-project/executor:v1.6.0-debug - | tar -tvf - | sort) <(crane export gcr.io/kaniko-project/executor:v1.7.0-debug - | tar -tvf - | sort)
1,4c1,2
< -rw-r--r--  0 root   root       25 Apr 26 12:54 kaniko/.config/gcloud/docker_credential_gcr_config.json
< -rw-r--r--  0 root   root      782 Apr 26 12:46 etc/nsswitch.conf
< -rw-r--r--  0 root   root   641760 Apr 26 12:47 kaniko/ssl/certs/ca-certificates.crt
< -rwxr-x---  0 root   root  6435607 Mar  5  2020 kaniko/docker-credential-gcr
---
> -rw-r--r--  0 root   root      782 Oct 19 22:29 etc/nsswitch.conf
> -rw-r--r--  0 root   root   641760 Oct 19 22:30 kaniko/ssl/certs/ca-certificates.crt
7,22c5,20
< -rwxr-xr-x  0 root   root 10223616 Apr 26 12:49 kaniko/docker-credential-ecr-login
< -rwxr-xr-x  0 root   root 13836465 Apr 26 12:50 kaniko/docker-credential-acr
< -rwxr-xr-x  0 root   root 33923072 Apr 26 12:54 kaniko/warmer
< -rwxr-xr-x  0 root   root 50630656 Apr 26 12:54 kaniko/executor
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 busybox/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 dev/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 etc/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.config/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.config/gcloud/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.docker/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/ssl/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/ssl/certs/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 proc/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 sys/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 workspace/
---
> -rwxr-xr-x  0 root   root  9829384 Oct 19 22:31 kaniko/docker-credential-gcr
> -rwxr-xr-x  0 root   root 10342400 Oct 19 22:32 kaniko/docker-credential-ecr-login
> -rwxr-xr-x  0 root   root 13836465 Oct 19 22:33 kaniko/docker-credential-acr
> -rwxr-xr-x  0 root   root 33959936 Oct 19 22:35 kaniko/warmer
> -rwxr-xr-x  0 root   root 50647040 Oct 19 22:35 kaniko/executor
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 bin/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 busybox/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 dev/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 etc/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/.docker/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/ssl/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/ssl/certs/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 proc/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 sys/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 workspace/

It looks like kaniko/.config/gcloud/docker_credential_gcr_config.json has been removed since v1.6.0.

The contents of that file at v1.6.0:

$ crane export gcr.io/kaniko-project/executor:v1.6.0-debug - | tar -Oxf - kaniko/.config/gcloud/docker_credential_gcr_config.json
{"TokenSources":["env"]}

That seems to be caused by the the removal of this line:

RUN ["docker-credential-gcr", "config", "--token-source=env"]

...in this change, part of #1780, 3 days ago.

I don't know if this is related, but it seems like a possible cause. That flag's behavior is documented here.

imjasonh · 2021-10-22T14:07:36Z

This comment on the other issue seems to indicate that re-adding docker-credential-gcr config --token-source=env might be (part of?) the fix, so that line in deploy/Dockerfile might be load-bearing.

I'm happy to send a PR to re-add this line, but I'd love to have some test to have more confidence that it is indeed the fix, and to prevent cases like this in the future.

bygui86 · 2021-12-15T13:26:02Z

@imjasonh when do you release this fix?

imjasonh · 2021-12-15T13:28:16Z

Maintainers rolled back the 1.7.0 release to unbreak users, but I have no idea what the plan is for rolling it back forward or for finding or fixing the bug.

@tejal29 is there any update on this?

bygui86 · 2021-12-15T13:33:01Z

@imjasonh thanks a lot

unfortunately I'm affected by #1667 so v1.6.0 is not suitable for me at the moment and I'm forced to use v1.6.0-debug :(

edtshuma · 2022-05-16T08:53:59Z

Hello @ALL.
I am facing the same issue and I have tried most of the fixes to no avail :

I have tried using gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81 :

`before_script:
- 'dotnet restore --packages $NUGET_PACKAGES_DIRECTORY'

build_job:
tags:
- xxxx
only:
- develop
stage: build
script:
- dotnet build --configuration Release --no-restore

publish_job:
tags:
- xxxx
only:
- develop
stage: publish
artifacts:
name: "$CI_COMMIT_SHA"
paths:
- ./$PUBLISH_DIR
script:
- dotnet publish ./src --configuration Release --output $(pwd)/$PUBLISH_DIR

docker_build_dev:
tags:
- xxxx
image:
name: gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81
entrypoint: [""]
only:
- develop
stage: docker
before_script:
- echo "Docker build"
script:
- echo "${CI_PROJECT_DIR}"
- cp ./src/Dockerfile /builds/xxx/xxx/xxx/Dockerfile
- mkdir -p /kaniko/.docker
- echo "{"auths":{"${CI_REGISTRY}":{"auth":"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')"}}}" > /kaniko/.docker/config.json
- >-
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--insecure
--skip-tls-verify
--skip-tls-verify-pull
--insecure-pull
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"`

I have also tried using gcr.io/kaniko-project/executor:v1.6.0-debug :

`before_script:
- 'dotnet restore --packages $NUGET_PACKAGES_DIRECTORY'

build_job:
tags:
- xxxx
only:
- develop
stage: build
script:
- dotnet build --configuration Release --no-restore

publish_job:
tags:
- xxxx
only:
- develop
stage: publish
artifacts:
name: "$CI_COMMIT_SHA"
paths:
- ./$PUBLISH_DIR
script:
- dotnet publish ./src --configuration Release --output $(pwd)/$PUBLISH_DIR

docker_build_dev:
tags:
- xxxx
image:
name: gcr.io/kaniko-project/executor:v1.6.0-debug
entrypoint: [""]
only:
- develop
stage: docker
before_script:
- echo "Docker build"
script:
- echo "${CI_PROJECT_DIR}"
- cp ./src/Dockerfile /builds/xxx/xxx/xxx/Dockerfile
- mkdir -p /kaniko/.docker
- echo "{"auths":{"${CI_REGISTRY}":{"auth":"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')"}}}" > /kaniko/.docker/config.json
- >-
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--insecure
--skip-tls-verify
--skip-tls-verify-pull
--insecure-pull
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG`

When using gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81 the pipeline fails with error :

`[32;1mSkipping Git submodules setup[0;m
section_end:1652535765:get_sources
[0Ksection_start:1652535765:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m[0;m
[32;1mDownloading artifacts for publish_job (33475)...[0;m
Downloading artifacts from coordinator... ok [0;m id[0;m=33475 responseStatus[0;m=200 OK token[xxxxxxxxxxx

section_end:1652535769:download_artifacts
[0Ksection_start:1652535769:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
section_end:1652535769:step_script
[0Ksection_start:1652535769:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m

section_end:1652539354:cleanup_file_variables
[0K[31;1mERROR: Job failed: execution took longer than 1h0m0s seconds
[0;m`

When using gcr.io/kaniko-project/executor:v1.6.0-debug the pipeline fails with error :

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: a repository name must be specified Cleaning up project directory and file based variables 00:00 ERROR: Job failed: command terminated with exit code 1

Is there something else obvious that I am missing?

hades0932 · 2022-05-30T17:28:13Z

@ed87
I have same error in minikube.
If you use rancher desktop create cluster and use the same k8s version, maybe no problem.
I don't know why。。。

grid-dev · 2022-08-30T12:29:58Z

Having the same error on EKS pushing cross account to ECR.

Using the following versions:

Kubernetes: 1.22
AMIs 1.22.6-20220526
Kaniko: gcr.io/kaniko-project/executor:v1.9.0@sha256:1f982af0b54be748221d9a35dcfa608660ab3d51229aa56bde5416f75aff7561

Error message

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/PLACEHOLDER2/manager:latest](http://PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/PLACEHOLDER2/manager:latest)": Post "https://PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/v2/PLACEHOLDER2/manager/blobs/uploads/": EOF

quyuancheng · 2023-02-02T07:41:56Z

Can --insecure and --skip-tls-verify be configured at the same time? What is the impact?

TK777777 · 2023-06-21T11:10:27Z

So how can we solve this problem in the end?

hadrien-toma · 2023-06-21T18:40:40Z

On my side I decided to use a secured registry because since a long time this issue was troubling my code and the workarounds were too costly to update along the releases...

TK777777 · 2023-06-25T08:10:08Z

I have found the problem and need to set the variable of Gitlab as a non protected variable!!!!!

nastacio mentioned this issue Aug 19, 2019

Fails to push image to registry appsody/tekton-example#6

Closed

tejal29 added this to the GA Release v1.0.0 milestone Jan 10, 2020

tejal29 closed this as completed Mar 18, 2020

tejal29 reopened this Sep 18, 2020

This was referenced Oct 22, 2021

Push Permission Error on latest #1786

Closed

recipe: Document diffing filesystem contents google/go-containerregistry#1155

Merged

imjasonh mentioned this issue Dec 21, 2021

Revert "Run docker_credentials_gcr in warmer (#1780)" #1852

Closed

4 tasks

aaron-prindle added platform/tekton and removed platform/tekton labels Jun 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error checking push permissions when pushing to insecure registry #702

error checking push permissions when pushing to insecure registry #702

CeadeS commented Jun 20, 2019

vijaySamanuri commented Jun 20, 2019

cn-sunletian commented Jul 2, 2019

priyawadhwa commented Sep 27, 2019

tejal29 commented Sep 27, 2019

tejal29 commented Mar 18, 2020

pvgbabu commented Jul 20, 2020

just4est commented Sep 17, 2020

tejal29 commented Sep 18, 2020

tchellomello commented Jan 1, 2021 •

edited

Loading

luisgc93 commented Oct 20, 2021 •

edited

Loading

hamko commented Oct 20, 2021

peter-dough commented Oct 20, 2021

vitdolphin commented Oct 20, 2021

austinpray-mixpanel commented Oct 20, 2021 •

edited

Loading

sigitisme commented Oct 22, 2021

imjasonh commented Oct 22, 2021

imjasonh commented Oct 22, 2021 •

edited

Loading

bygui86 commented Dec 15, 2021

imjasonh commented Dec 15, 2021

bygui86 commented Dec 15, 2021

edtshuma commented May 16, 2022

hades0932 commented May 30, 2022

grid-dev commented Aug 30, 2022

quyuancheng commented Feb 2, 2023

TK777777 commented Jun 21, 2023

hadrien-toma commented Jun 21, 2023

TK777777 commented Jun 25, 2023

error checking push permissions when pushing to insecure registry #702

error checking push permissions when pushing to insecure registry #702

Comments

CeadeS commented Jun 20, 2019

vijaySamanuri commented Jun 20, 2019

cn-sunletian commented Jul 2, 2019

priyawadhwa commented Sep 27, 2019

tejal29 commented Sep 27, 2019

tejal29 commented Mar 18, 2020

pvgbabu commented Jul 20, 2020

just4est commented Sep 17, 2020

tejal29 commented Sep 18, 2020

tchellomello commented Jan 1, 2021 • edited Loading

luisgc93 commented Oct 20, 2021 • edited Loading

hamko commented Oct 20, 2021

peter-dough commented Oct 20, 2021

vitdolphin commented Oct 20, 2021

austinpray-mixpanel commented Oct 20, 2021 • edited Loading

sigitisme commented Oct 22, 2021

imjasonh commented Oct 22, 2021

imjasonh commented Oct 22, 2021 • edited Loading

bygui86 commented Dec 15, 2021

imjasonh commented Dec 15, 2021

bygui86 commented Dec 15, 2021

edtshuma commented May 16, 2022

hades0932 commented May 30, 2022

grid-dev commented Aug 30, 2022

quyuancheng commented Feb 2, 2023

TK777777 commented Jun 21, 2023

hadrien-toma commented Jun 21, 2023

TK777777 commented Jun 25, 2023

tchellomello commented Jan 1, 2021 •

edited

Loading

luisgc93 commented Oct 20, 2021 •

edited

Loading

austinpray-mixpanel commented Oct 20, 2021 •

edited

Loading

imjasonh commented Oct 22, 2021 •

edited

Loading