Missing files in Kaniko-built image

[LinzerToertchen]

Actual behavior
When building from Red Hat's UBI 8 and installing nginx a library is missing in the resulting image preventing docker from pulling it.

$ docker pull registry.example.com/nginx:v0.2
v0.2: Pulling from it/docker-images/nginx
03e56b46bf0b: Already exists
3a13cc2f5d65: Already exists
d42cbf90e64a: Pull complete
b4427cf0b409: Extracting [==================================================>] 
 22.93MB/22.93MB
97d00bd6afb4: Download complete
failed to register layer: Error processing tar file(exit status 1): open /lib64/libtirpc.so.3.0.0: no such file or directory

Expected behavior
The files should be present

To Reproduce

$ docker run --rm -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug

Additional Information

$ docker run --rm gcr.io/kaniko-project/executor:debug version  
Kaniko version :  v0.17.1

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:26:10 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:24:42 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Build Context:

• Dockerfile

  FROM registry.access.redhat.com/ubi8/ubi-minimal
  
  # Copy nginx repo
  COPY nginx.repo /etc/yum.repos.d/nginx.repo
  
  # TODO: Add --nodocs or --setopt=tsflags=nodocs once
  # https://bugzilla.redhat.com/show_bug.cgi?id=1769831
  # is closed
  RUN microdnf install nginx \
    && microdnf clean all
  
  EXPOSE 8080/tcp
  
  VOLUME /usr/share/nginx/html
  
  # Run nginx
  CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
  

• nginx.repo

  [nginx-stable]
  name=nginx stable repo
  baseurl=http://nginx.org/packages/rhel/$releasever/$basearch/
  gpgcheck=1
  enabled=1
  gpgkey=https://nginx.org/keys/nginx_signing.key
  module_hotfixes=true
  

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:debug@sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[stephanf]

Could be related to:
#1039
#1038
#1024
#1028

I had a working build by moving from AmazonLinux:2018.03 to AmazonLinux:2, but pip3 was not working with that image. Yesterday I fixed the issue with pip3, and the images didn't work anymore (build successful, but pulling images doesn't work).

It's always something different, sometimes it is git, or zip, or java ... tried different approaches including not uninstalling installed software.

[tbonfort]

Also getting this on a go build with 0.17.1.

error building image: error building stage: failed to take snapshot: unable to add file /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a to layered map: error creating hash for /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a: lstat /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a: no such file or directory

[tejal29]

Thanks @HaehnleinMar Sorry for the regression. Please use v0.16.0 untill we fix this issue.

[LinzerToertchen]

Thank you @tejal29 for that suggestion. Using v0.16.0 it works like a charm 👍

[binnythomas-1989]

guys I need an equivalent version of v0.16.0 for executor:debug. What would that be?

[tejal29]

@binnythomas-1989 please use the tag v0.16.0-debug

[stephanf]

debug-v0.16.0

[cvgw]

I believe this is the same issue as #1039 as is fixed in a1af057.

Tags a1af057f997316bfb1c4d2d82719d78481a02a79 and debug-a1af057f997316bfb1c4d2d82719d78481a02a79 have the new code

[tejal29]

I verfied this on latest image gcr.io/tejal-test/executor

docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_1045 '

Complete.
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving paths                              
INFO[0016] EXPOSE 8080/tcp                              
INFO[0016] cmd: EXPOSE                                  
INFO[0016] Adding exposed port: 8080/tcp                
INFO[0016] VOLUME /usr/share/nginx/html                 
INFO[0016] cmd: VOLUME                                  
INFO[0016] adding volume /usr/share/nginx/html to whitelist 
INFO[0016] CMD ["/usr/sbin/nginx", "-g", "daemon off;"] 
tejaldesai@@kaniko (r-v0.18.0)$ 

tejaldesai@@kaniko (r-v0.18.0)$ docker pull gcr.io/tejal-test/test_1045
Using default tag: latest
latest: Pulling from tejal-test/test_1045
0a63a759fe25: Pull complete 
9d6c79b335fa: Pull complete 
8975e1c4bc3c: Pull complete 
43d749462e9b: Pull complete 
Digest: sha256:ddc9bc6d6d819618ff871e1b04414ab36088d14c3dbf292b0fa39dac24038b9f
Status: Downloaded newer image for gcr.io/tejal-test/test_1045:latest
gcr.io/tejal-test/test_1045:latest

[gmmephisto]

Facing the same issue with build image usingdebug-v0.19.0 executor, but inside running container:

Unable to find image '<somename>' locally
latest: Pulling from <somename>
ac9208207ada: Pull complete
4f3f99a61312: Pull complete
fd34dc44f01c: Pull complete
f4efdfd25de7: Pull complete
2fcdbdc9c4c7: Pull complete
5a4f61ccd968: Pull complete
4c85c679d9e1: Pull complete
9c1482cd1d98: Pull complete
f37097ffe8b0: Pull complete
e0ff42de1c82: Pull complete
c344f69900f3: Pull complete
b67f438365c2: Pull complete
0e551847c7df: Pull complete
3fc55cba6164: Pull complete
Digest: sha256:e81612d107dc045d0736b9c9f750d42cec3d97cee50d7edcd8068832709f1375
Status: Downloaded newer image for <somename>
[root@86f72346de6b devops]#
[root@86f72346de6b devops]# yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   libelf.so.1: cannot open shared object file: No such file or directory

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Oct 30 2018, 23:45:53)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


[root@86f72346de6b devops]#

Workaround - run ldconfig

[gmmephisto]

And another fail on pulling image:

$ docker run -ti --rm <somename>
Unable to find image '<somename>' locally
ac9208207ada: Already exists
4f3f99a61312: Already exists
fd34dc44f01c: Already exists
f4efdfd25de7: Already exists
2fcdbdc9c4c7: Already exists
5a4f61ccd968: Already exists
4c85c679d9e1: Already exists
9c1482cd1d98: Already exists
f37097ffe8b0: Already exists
06e1d006bd86: Pull complete
c599036dfe85: Pull complete
1094881658c8: Pull complete
655e5f9566d6: Pull complete
8306954b60d9: Pull complete
d545b9e2c7a9: Extracting [==================================================>]  151.7MB/151.7MB
docker: failed to register layer: Error processing tar file(exit status 1): file exists.
See 'docker run --help'.

Was successfully build with existing cache. I'll try to figure out how to stable reproduce this error.

[evgkrsk]

Same issue with debug-v0.19.0

[yashbhutwala]

Is this issue fixed? I think I'm facing a variant of this with the latest gcr.io/kaniko-project/executor:debug

The weird thing for me is that the image functions fine when I run it locally, but when I run it via gitlab-runner, I see the following issue:

python3: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

[llech]

When running microdnf update on the ubi8-minimal based image, I'm getting the error:

ERRO[0036] couldn't eval /usr/lib/libssl.so.1.1 with link /usr/lib/libssl.so.1.1

I don't know if it's related, but at the beginning, I'm having error

E0601 07:08:11.044500 113 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

It is both with 'latest' and with 'v0.16.0' kaniko images.

[peter-englmaier]

The problem has to do with symbolic links, as has been written here and elsewhere a couple of times.
In ubi8 there are 4 such links:

[root@c17cc86c6c20 /]# ls -l /
lrwxrwxrwx   1 root root    7 Apr 23  2020 bin -> usr/bin
lrwxrwxrwx   1 root root    7 Apr 23  2020 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Apr 23  2020 lib64 -> usr/lib64
lrwxrwxrwx   1 root root    8 Apr 23  2020 sbin -> usr/sbin

Inside the lib folders are links like

[root@c17cc86c6c20 /]# ls -l usr/lib64/*ssl*
lrwxrwxrwx 1 root root     16 Mar 25 16:49 usr/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x 1 root root 615576 Mar 25 16:49 usr/lib64/libssl.so.1.1.1g

Inside kaniko they look like this, however:

lrwxrwxrwx  1 root root     23 Apr 14 10:25 libssl.so.1.1 -> ../../lib/libssl.so.1.1

And do not work.

A simple workaround is, to add the following line at the top of the Dockerfile for ubi8 derived images:

FROM quay.io/keycloak/keycloak:11.0.3
# Fix resolving path when building with kaniko:
ENV LD_LIBRARY_PATH=/usr/lib64:/usr/lib

With this, the kaniko build succeeds!

[MartinSadovy]

Similar problem with postgres package, kaniko 0.17.1 produces invalid image. 0.18.0 is OK 🎉

[mareksuscak]

This is happening to us too. An empty folder created in a RUN instruction using mkdir -p is missing from the final image. We're using Kaniko 1.8.0 and tried full snapshot mode as well as redo snapshot mode. Here's the instruction:

RUN set -eux; \
  mkdir -p var/cache var/log var/sessions /var/run/php; \
  composer dump-autoload --classmap-authoritative --optimize --no-dev --no-interaction; \
  composer dump-env $APP_ENV; \
  rm .env; \
  rm .env.prod; \
  composer patch:apply --no-interaction --no-dev; \
  chown -R www-data:www-data var; \
  chown -R www-data:www-data public; \
  chmod +x bin/*; sync

All folders are created successfully but /var/run/php is missing for some reason. This image is based on the official php:7.4.24-fpm-alpine3.13 image. Building via plain docker fixes the issue.

[JanSchutteAdyen]

@mareksuscak That is to be expected, see the --ignore-var-run option, it defaults to true.