error building image: error building stage: error removing sbin to make way for new symlink: remove /sbin/docker-init: device or resource busy

[hanlaipeng]

I run the kaniko in kubernetes, the k8s node's docker version is 19.03. When i run kaniko in k8s, there is an error:
"error building image: error building stage: error removing sbin to make way for new symlink: remove /sbin/docker-init: device or resource busy".
However, if k8s node's docker version is 18.03, it works. So, i want to know how to fix it when i run kaniko in k8s which cluster node docker version is 19.03. Thanks very much!

[hanlaipeng]

docker (version 18.03) init is
/dev/init --/bin/bash
but docker (version 19.03) init is
/sbin/docker-init --/bin/bash

[wintersolutions]

is this a possible duplicate of #1001 ?

[tejal29]

@hanlaipeng Can you paste your container logs along with your dockerfile? Please run kaniko with -v=debug?

[bitsofinfo]

ugh, same issue w/ gcr.io/kaniko-project/executor:debug-v0.17.0, with latest i get #1104.

[bitsofinfo]

only reverting to 0.15.0 lets me actually get working images

[hanlaipeng]

oh, I read the code and find error in this:
// Check if something already exists at path (symlinks etc.)
// If so, delete it
if FilepathExists(path) {
if err := os.RemoveAll(path); err != nil {
return errors.Wrapf(err, "error removing %s to make way for new file.", path)
}
}
if /sbin already exists in docker, remove it , however the docker-init in /sbin

[tosheer]

+1. Facing same issue with Kaniko version : v0.17.1.

[0005] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/cpu false} {/sys/fs/cgroup/cpuacct false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/net_cls false} {/dev/mqueue false} {/dev/shm false} {/sbin/docker-init false} {/ssh-agent-DEFAULT false} {/ssh-agent-DEPLOY false} {/secrets false} {/radar-48903579.sh false} {/workspace false} {/tmp false} {/etc/resolv.conf false} {/root/.netrc false} {/pieci/bin false} {/root/.cache false} {/root/.npmrc false} {/dev/random false} {/root/.gradle false} {/run/rio false} {/etc/hosts false} {/root/.m2 false} {/etc/hostname false} {/root/.cache/rio false} {/root/.gradle/init.gradle false} {/root/.m2/settings.xml false} {/root/.gradle/gradle.properties false} {/root/.lein/profiles.clj false} {/root/.sbt/.credentials false} {/root/.ivy2/cache false} {/dev/null false} {/dev/random false} {/dev/full false} {/dev/tty false} {/dev/zero false} {/dev/urandom false} {/dev/console false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/latency_stats false} {/proc/timer_list false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
DEBU[0005] Not adding /dev because it is whitelisted    
DEBU[0005] Not adding /dev/null because it is whitelisted 
DEBU[0005] Not adding /dev/urandom because it is whitelisted 
DEBU[0005] Not adding /etc/hosts because it is whitelisted 
DEBU[0005] Not adding /etc/resolv.conf because it is whitelisted 
DEBU[0005] Not adding /proc because it is whitelisted   
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busy

[gino-m]

Same here using executor:latest (v0.22.0), currently blocking google/ground-platform#289

The first run builds the image successfully, but successive runs return:

error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.workspace: unlinkat //workspace: device or resource busy

Dockerfile, sans comments:

FROM node:latest

WORKDIR tmp
RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \
    && apt-get update \
    && apt -y install ./google-chrome-stable_current_amd64.deb \
    && rm google-chrome-stable_current_amd64.deb \
    && rm -rf /var/lib/apt/lists/*
WORKDIR ..

WORKDIR workspace

WORKDIR functions
COPY functions/package*.json ./
RUN npm install
WORKDIR ..

WORKDIR web-ng
COPY web-ng/package*.json ./
RUN npm install
WORKDIR ..

Tried reverting to v0.15.0 but no change. I see mention of clearing cached files, but unsure how to do that. Running Cloud Build from GitHub checks.

[gino-m]

@tejal29 Update: We were able to get this working for our project by writing deps to /usr/src instead of /workspace, and then moving them to /workspace at build time (see https://github.com/google/ground-platform/pull/289/files). I'm not sure if this is the correct approach, but wanted to share in case it's a viable workaround and in case it can help isolate the cause.

[davidmccormick]

Issue still present in gcr.io/kaniko-project/executor:v1.0.0, and will affect any from image where /sbin is a symlink elsewhere.

[michaelkebe]

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/37003

A new version of the gitlab-runner triggers this problem. Downgrading gitlab-runner resolves it.

This is maybe some problem in the environment, rather than kaniko?

[evsasha]

I also confirm that GitLab Runner 16.6.0 creates this issue. Downgrading to version 16.5.0 helped.

I use kaniko v1.14.0.

[stanhu]

As explained in https://gitlab.com/gitlab-org/gitlab-runner/-/issues/37003#note_1661177356, GitLab Runner v16.6.0 enables the Init flag, which causes Docker to mount /sbin/docker-init by default. https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4488, which will likely ship in GitLab Runner v16.6.1, will put this behind a feature flag.

However, I think the existence of /sbin/docker-init should be handled by Kaniko gracefully.

Enabling --verbosity=trace shows right before this:

TRAC[0000] Creating dir /run                            
TRAC[0000] Creating dir /run/lock                       
TRAC[0000] Symlink from usr/sbin to /sbin               
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busy

It seems this happens because a Docker image that has /usr/bin symlinked to /sbin causes trouble since /sbin can't be removed if /sbin/docker-init is in use.

[priyanka0409bansal]

I also confirm that GitLab Runner 16.6.0 creates this issue. Downgrading to version 16.5.0 helped.

I use kaniko v1.14.0.

Can Anyone please explain me how we can downgrade the gitlab version?

[stanhu]

With regard to #946 (comment), I was wondering why Kaniko tries to extract files into /. If we could avoid this, then we could avoid the need to touch /sbin, which is mounted by Docker. It turns out there are related issues:

• COPY and RUN commands have full access to the filesystem #2153
• Conflicting files while building images with Kaniko on Kubernetes #2165

An attempt in #2169 was made to improve this, but it was closed due to #2169 (comment).

@hown3d Could we reconsider that pull request in light of this issue?

[hown3d]

With regard to #946 (comment), I was wondering why Kaniko tries to extract files into /. If we could avoid this, then we could avoid the need to touch /sbin, which is mounted by Docker. It turns out there are related issues:

* [COPY and RUN commands have full access to the filesystem #2153](https://github.com/GoogleContainerTools/kaniko/issues/2153)

* [Conflicting files while building images with Kaniko on Kubernetes #2165](https://github.com/GoogleContainerTools/kaniko/issues/2165)

An attempt in #2169 was made to improve this, but it was closed due to #2169 (comment).

@hown3d Could we reconsider that pull request in light of this issue?

The PR is closed due to the way kaniko is designed which makes it hard to implement chroot isolation. I don't have time at the moment to implement such a big change, which should also be discussed beforehand somewhere.

In my time implementing the PR mentioned above, kaniko wasn't able to straight up run without seccomp and apparmor configuration of the container, which makes it not suitable for default usage.