nerdctl doesn't work with kind

[tuananh]

I use nerdctl rootless setup, create an alias for docker to make kind happy but then it still complains that

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

Upon checking the src code, it seems that it relies on docker info command to detect mem,pid and cpushare supports. which nerdctl currently not emit

my nerdctl info --format '{{json .}}' output is like this, while docker output has more, notably MemoryLimit, CPUShares, etc...

{
  "ID": "86232191-2d46-475b-be0c-1472c5174763",
  "Driver": "overlayfs",
  "Plugins": {
    "Log": [
      "json-file"
    ],
    "Storage": [
      "native",
      "overlayfs"
    ]
  },
  "LoggingDriver": "json-file",
  "CgroupDriver": "systemd",
  "CgroupVersion": "2",
  "KernelVersion": "5.13.14-xanmod1",
  "OperatingSystem": "Pop!_OS 21.04",
  "OSType": "linux",
  "Architecture": "x86_64",
  "Name": "x300",
  "ServerVersion": "v1.5.5",
  "SecurityOptions": [
    "name=seccomp,profile=default",
    "name=cgroupns",
    "name=rootless"
  ]
}

I'm not sure if this alone would be enough to make kind works with nerdctl?

[cndoit18]

I have the same problem

[fahedouch]

some fields are not yet implemented on nerdctl side

[tuananh]

the problem with kind seems deeper than this

upon fixing the fields in nerdctl info command, i got the following error.

ERROR: failed to create cluster: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=test --format '{{.Names}}'" failed with error: exit status 1
Command Output: Incorrect Usage: flag provided but not defined: -filter

so looks like --filter is not implemented for nerdctl ps as well.

[afbjorklund]

There was a lot of pulling of hairs, to get kind and minikube to work with podman.

Expect similar docker bug-compatibility fixes, to get them to work with nerdctl too ?

[AkihiroSuda]

Upon checking the src code, it seems that it relies on docker info command to detect mem,pid and cpushare supports. which nerdctl currently not emit

Being implemented in #426

We still need --filter at least to make kind work, though

[alexellis]

Just landed here from your blog post @tuananh it looks like an interesting idea :)

[AkihiroSuda]

Cross-posting: kubernetes-sigs/kind#2317 (comment)

I'd be interested to see someone lay out what is missing or just different between docker and nerdctl + containerd and discussing how kind might approach this (e.g. CNI networking will probably be an interesting one), at which point we can look at writing or accepting PRs to implement this (see also: https://kind.sigs.k8s.io/docs/contributing/getting-started/).

Docker	nerdctl (as of v0.14.0)	Usage in kind
docker inspect --type=image	nerdctl image inspect	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/images.go#L56
Label "com.docker.network.bridge.enable_ip_masquerade"	(N/A)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L151
Label "com.docker.network.driver.mtu"	(N/A)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L154 https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L166
docker network create --ipv6	(N/A)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L157
docker network ls --filter=FILTER	(parse and filter the JSON output manually)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L240 https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L256
docker ps --filter=FILTER	(parse and filter the JSON output manually)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provider.go#L120
Label "desktop.docker.io/ports/%d/tcp", for docker run -p without explicit host port number (?)	(N/A; nerdctl run -p needs explicit host port number)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provider.go#L179
docker run --restart=on-failure:1	(N/A; workaround would be to use --restart=always)	https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provision.go#L166

[zvonkok]

/cc @zvonkok

[BenTheElder]

So the bridge network management instead of default network is mostly about gaining docker's embedded DNS functionality which simplifies some problems around node IP management and users controlling the DNS resolver via controlling the host resolver where dockerd runs.

We have this gap with podman as well though, for the moment.

The other aspect is trying to ensure dual stack is available for creating clusters.

For the ports we could just add a custom kind specific label with the port forwards and fetch that, except in the case of random port, that's a problem for us. We want to let the runtime pick a random free port and then discover it from the running container later in some cases.

The rest of the list seems manageable.

Instead of attempting to alias nerdctl docker we'd detect docker vs nerdctl and use nerdctl specific code in kind (some for podman).

That's already the case for node management and we have that behind an (complex, bad, internal) interface.

For image loading we can visit that in a follow up as there's an open discussion about designing a new command to handle portability better and image-archive will work for now with no nerdctl specific code.

[yankay]

Cross-posting: kubernetes-sigs/kind#2317 (comment)

I'd be interested to see someone lay out what is missing or just different between docker and nerdctl + containerd and discussing how kind might approach this (e.g. CNI networking will probably be an interesting one), at which point we can look at writing or accepting PRs to implement this (see also: https://kind.sigs.k8s.io/docs/contributing/getting-started/).

Docker nerdctl (as of v0.14.0) Usage in kind
docker inspect --type=image nerdctl image inspect https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/images.go#L56
Label "com.docker.network.bridge.enable_ip_masquerade" (N/A) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L151
Label "com.docker.network.driver.mtu" (N/A) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L154 https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L166
docker network create --ipv6 (N/A) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L157
docker network ls --filter=FILTER (parse and filter the JSON output manually) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L240 https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/network.go#L256
docker ps --filter=FILTER (parse and filter the JSON output manually) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provider.go#L120
Label "desktop.docker.io/ports/%d/tcp", for docker run -p without explicit host port number (?) (N/A; nerdctl run -p needs explicit host port number) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provider.go#L179
docker run --restart=on-failure:1 (N/A; workaround would be to use --restart=always) https://github.com/kubernetes-sigs/kind/blob/0abda0bca8c61a15ec6ec5cf6368d2bb440cd515/pkg/cluster/internal/providers/docker/provision.go#L166

Thanks @AkihiroSuda

The issues of has been almost fixed, and the cluster can be create successfully with a little code change.

• [WIP]Add ipv6 error message check to support the nerdctl kubernetes-sigs/kind#3396
• Fix nerdctl ps --format {{.Names}} behaviour #2594
• Support create network with com.docker.network.bridge.enable_ip_masquerade option #2592

[yankay]

close by kubernetes-sigs/kind#3429