feat: chroot isolation

[hown3d]

This PR implements isolation between build commands with chroot.
Should fix #2165 and #2153.

Description

Chroot isolation is implemented by creating a temporary directory inside the HOME directory which is later used as the base for chroot. Before chroot, the context directory, /proc and /dev are bind mounted into the new root directory.
Isolation is wrapped between each build command execution.

Isolation can be toggled with a flag --isolation which resolves by default to chroot. Providing anything else will disable isolation.

TODOs before removing Draft:

• Integration tests are passing
• New unit tests for chroot

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes unit tests
• Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

• The code flow looks good.
• Unit tests and or integration tests added.

Release Notes

User facing changes:

• kaniko adds a new flag --isolation to specify the isolation method. The default will be chroot. Removing isolation can be done by setting the flag to none

[imjasonh]

This looks promising! Thanks for working on this. 👍

[imjasonh]

We may want os.UserHomeDir() instead, which uses $HOME and other heuristics.

[imjasonh]

These {}s are unnecessary:

switch s.opts.Isolation {
case "chroot":
  newRoot, err := ...
  ...
case "none":
  ...
}

[hown3d]

During my development on this PR, I ran more and more into duplications with established isolation tools like runc and crun. Is it maybe worth considering investing into using those tools that are conform with the OCI spec standard? Reimplementing those isolation techniques is imo not easy for the long run.

I'll look into the possibilities with runc's libcontainer and how it plays with kaniko.

[imjasonh]

During my development on this PR, I ran more and more into duplications with established isolation tools like runc and crun. Is it maybe worth considering investing into using those tools that are conform with the OCI spec standard? Reimplementing those isolation techniques is imo not easy for the long run.

I'll look into the possibilities with runc's libcontainer and how it plays with kaniko.

First of all, thanks for digging into this!

I'm excited to learn what our options are if we want to retain the invariant that Kaniko doesn't require the kind of privilege that docker build, Buildah, etc., require, and expects to be run inside a container itself. This is an area I'm not very experienced with, so I'll mostly defer to your expertise and findings here.

[hown3d]

During my development on this PR, I ran more and more into duplications with established isolation tools like runc and crun. Is it maybe worth considering investing into using those tools that are conform with the OCI spec standard? Reimplementing those isolation techniques is imo not easy for the long run.
I'll look into the possibilities with runc's libcontainer and how it plays with kaniko.

First of all, thanks for digging into this!

I'm excited to learn what our options are if we want to retain the invariant that Kaniko doesn't require the kind of privilege that docker build, Buildah, etc., require, and expects to be run inside a container itself. This is an area I'm not very experienced with, so I'll mostly defer to your expertise and findings here.

I took a look around for other isolation tools (libcontainer, bubblewrap, crun) but all of those tools need privileged containers, because they create cgroups and those need writable /sys dirs, thus --privileged.

I'll try to implement a sort of style that buildah did with their chroot isolation.

Integration tests for isolation will be created aswell (breaking out of chroot, accessing "host" files).

[natalieparellano]

I apologize for my ignorance here, but would it be possible to explain at a high level how does this PR circumvent the limitations noted here: #107 (comment)? Would a platform running containers without --privileged need to pass --isolation none?

[hown3d]

Glad you asked, since this is a draft PR I'm not 100% settled on the final implementation, but the current idea is the following:

1. Unpack Rootfs of image into isolated directory under home directory of user
2. Create user & mount namespace to map current user to root (future rootless implementation?) and bind mount /proc /sys /dev /etc/resolv.conf and /etc/hosts (networking purposes) into the new root
3. pivot_root into newly created isolated directory where the rootfs lays and set the capabilities to the same ones that docker uses (because unshare creates a full set of capabilities).
4. Run original command

Currently, this implementation has to set the seccomp and apparmor profiles to "unconfined" because those profiles are very strict in docker (CRI-O, Podman etc. allow mount syscalls), but it's being discussed in the moby repo: moby/moby#42455

The final goal is to be able to run --isolation chroot without SYS_ADMIN privilege and with a slightly different seccomp/apparmor profile than default (when running with docker).

Also I think it's best to leave the default to --isolation none at first, since changing the default could break builds.

[gabyx]

Cool stuff: Some noobie question: Why is chroot isolation needed, what does it achieve? I know it solves a security issue...

[Lord-Y]

Hello guys, any news about this PR?

[gewei2000]

hi guys, is there any progress about this PR? @hown3d

[hown3d]

Hey everyone,
I haven't had the time working on that PR, but I'm also reconsidering if this is still a good idea to implement.
The thing that bothers me:

• Chroot itself is not enough of a isolation
  -> escaping from chroot is easily done when running as root, so we need to run kaniko as a non root user.
  When running as non-root we would need to implement running in user namespaces to achieve root-like behavior.
  This leads us into container in container issues, like having to run with a customized seccomp profile, because docker default seccomp profile blocks setns and unshare syscalls.

Kaniko is just not designed to provide more isolation inside the container environment.
If you want to build dockerfiles inside containers in a rootless way with more isolation inside the container, but more privileges on the host, consider using the following options:

• buildah rootless with chroot isolation (needs customized seccomp and selinux disabled).
  The containers team provides a container image with instructions on quay.io
• buildkit rootless in container
  Can be configured nearly in the same way as the buildah instructions aboth. Uses rootlesskit to enter new namespaces inside container

I'm hereby closing this PR, since I won't continue working on that feature.