Image which moves symlink aside and then adds file built with rootless container upsets k3s ctr images import

[adelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When an image is built which adds a symlink, moves it aside, and then adds a file instead, importing such image to k3s' containerd fails when it was built with rootless podman. The only difference that I could find compared to sudo podman and docker built images is ownership and permissions on the whiteout file.

Steps to reproduce the issue:

1. Have Dockerfile.busybox with

FROM busybox
RUN ln -s something /nothing
RUN mv /nothing /nothing-1
RUN echo test > /nothing

2. Built it with rootless podman build -t localhost/remove-symlink:podman-rootless -f Dockerfile.busybox .
3. Save the image with podman save localhost/remove-symlink:podman-rootless > remove-symlink-podman-rootless.tar
4. Try to load it to k3s' containerd: sudo k3s ctr images import remove-symlink-podman-rootless.tar

Describe the results you received:

time="2022-04-10T12:18:24Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:9e72f4e27c072a6bc1f1c77faf09e2117dfc2befde3f6a67ea7c9fc6643413dd: operation not permitted: unknown" key="extract-585114273-27IB sha256:41cd598a150f6411ee17fdf2b61edce6832112edb150be1499b2bec78f66d955"
ctr: failed to extract layer sha256:9e72f4e27c072a6bc1f1c77faf09e2117dfc2befde3f6a67ea7c9fc6643413dd: operation not permitted: unknown
unpacking localhost/remove-symlink:podman-rootless (sha256:9815822087fdd3be70734758e26c6b48df795f2060e39c4eccd00b7999565d8c)...

Describe the results you expected:

unpacking localhost/remove-symlink:podman-rootless (sha256:9815822087fdd3be70734758e26c6b48df795f2060e39c4eccd00b7999565d8c)...done

Additional information you deem important (e.g. issue happens only occasionally):

The manifest.json says (output of jq):

[
  {
    "Config": "6fc663a03b13e60415c34e7324a1b3bc0a2f90aedaee4c21a0b0ff481e5a8336.json",
    "RepoTags": [
      "localhost/remove-symlink:podman-rootless"
    ],
    "Layers": [
      "797ac4999b67d8c38a596919efa5b7b6a4a8fd5814cb8564efa482c5d8403e6d.tar",
      "75674f56cdd6476f7862501017a9e372ed64d83f2fd8921bbb18f3965dba0ff2.tar",
      "9e72f4e27c072a6bc1f1c77faf09e2117dfc2befde3f6a67ea7c9fc6643413dd.tar",
      "2003b99c9ada9eb6a2dd5eaeb6e5f942f58344855ba03c7fd02acf69b9f7ed16.tar"
    ]
  }
]

The third tar which moves the symlink aside is

-rw------- root/root         0 2022-04-10 12:18 .wh.nothing
lrwxrwxrwx root/root         0 2022-04-10 12:18 nothing-1 -> something

When I do the same steps with sudo podman or docker, that tar has the whiteout files with permissions completely cleared:

---------- 0/0               0 2022-04-10 12:18 .wh.nothing
lrwxrwxrwx root/root         0 2022-04-10 12:18 nothing-1 -> something

While I did not find any information about the expected permissions on those whiteout files, it seems rootless podman does something slightly different / incompatible. It should likely create the .wh.nothing file in the tarball with permissions 000. And/or use the numerical uid/gid.

The GitHub Actions reproducer is at https://github.com/adelton/freeipa-container/actions/runs/2143864363.

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 2
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: fv-az167-487
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 121
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.13.0-1021-azure
  linkmode: dynamic
  logDriver: journald
  memFree: 3708432384
  memTotal: 7283847168
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 4m 22.59s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/runner/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /tmp/podman-run-1001/containers
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Listing...
podman/now 100:3.4.2-1 amd64 [installed,local]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

I see this on GitHub Action's ubuntu-20.04.

[flouthoc]

Hi @adelton , Thanks for creating the issue, are you using fuse-overlayfs ? I am unable to reproduce the issue with docker load or podman load but as written in issue above i think its happening only with k3s.

I am not at all sure if build has anything to do with whiteout files directly but I'll tag @nalind if he can re-call anything from top level.

[flouthoc]

Maybe this issue can be moved away from podman -> buildah, this does not looks like a podman issue at first glance.

[adelton]

All I really know about that podman configuration is in that podman info output above. I would assume fuse-overlayfs is used ... but is there a way to make sure?

[mheon]

Looks like kernel overlay, not fuse-overlay, from the output of podman info - no mount program is mentioned

[nalind]

The difference looks like it's a native-diff/naive-diff difference. Whiteouts created by naive diff (which is what you're seeing when you're root) have 0 permissions, but whiteouts converted from native diff output (which is what you're seeing when you're not root) are getting their mode set to 0o600.
That's aside from whether or not containerd should care about the permissions on whiteout entries, which is probably better discussed elsewhere.