Discussion: should we drop testing inside a container?

[apostasie]

Our testing rig is designed around a container, and the CI itself is currently testing nerdctl purely inside a docker container.

This may be a good idea for server-side components that are also deployed and used inside containers.
It also may seem like a good idea to more easily "reproduce" specific target environments.

However, nerdctl is not a server component used in controlled environments. It's a cli, most of the time ran directly on (a variety of) hosts that we do not control.

It is very obvious that nobody is running the test suite directly on the host.
Because it fails. Despite recent efforts to fix it, it is still not passing on the most obvious and simplest environment of all: the default lima install.

Furthermore, by limiting developers' testing to that specific, artificially controlled container environment, we get a false sense of success, and tend to think that things are working, while really they are not.

I am suggesting that we stop encouraging testing inside the docker container, and instead that we re-design our testing to just run natively, and that we encourage contributors and users alike to test in their specific environment.

I am also suggesting that we remove the extra level of indirection on the CI if feasible.

Thoughts?

[AkihiroSuda]

SGTM if the test env can be installed (and uninstalled) easily.

[apostasie]

SGTM if the test env can be installed (and uninstalled) easily.

Ok.
I'll see what I can do.

[fahedouch]

LGTM