podman run with --uidmap options randomly causes Host ID ${HOSTID} cannot be mapped to a container ID error

[nia1048596]

Description:

If we specify --uidmap options to create container, the storage driver will chown files in container image filesystem to the specified uid/gid range.

storage/drivers/chown_unix.go

Lines 15 to 31 in 3db6883

	func platformLChown(path string, info os.FileInfo, toHost, toContainer *idtools.IDMappings) error {
	st, ok := info.Sys().(*syscall.Stat_t)
	if !ok {
	return nil
	}
	// Map an on-disk UID/GID pair from host to container
	// using the first map, then back to the host using the
	// second map. Skip that first step if they're 0, to
	// compensate for cases where a parent layer should
	// have had a mapped value, but didn't.
	uid, gid := int(st.Uid), int(st.Gid)
	if toContainer != nil {
	pair := idtools.IDPair{
	UID: uid,
	GID: gid,
	}
	mappedUID, mappedGID, err := toContainer.ToContainer(pair)

There is a race condition here during chowning. The above function platformLChown gets called in pwalk.walk, which spawns many go routines to chown in parallel.

storage/drivers/chown.go

Lines 53 to 62 in 3db6883

	chown := func(path string, info os.FileInfo, _ error) error {
	if path == "." {
	return nil
	}
	return platformLChown(path, info, toHost, toContainer)
	}
	if err := pwalk.Walk(".", chown); err != nil {
	fmt.Fprintf(os.Stderr, "error during chown: %v", err)
	os.Exit(1)
	}

If there are hard linked files, they might be chowned many times. After the first chown, the second chown will fail to map its uid/gid, which causes the Host ID ${HOSTID} cannot be mapped to a container ID error.

Steps to reproduce:

We can use busybox as example image, because there are 400 hard links in /bin/*, all files linking to /bin/busybox.

1. remove busybox:latest image

   podman rmi busybox:latest
   

2. setup podman subuid/subgid range

    $ grep podman /etc/{subuid,subgid}
   /etc/subuid:podman:2100000:2000000
   /etc/subgid:podman:2100000:2000000
   

3. launch new busybox container with --uidmap=2200000

    $ sudo podman run --rm -it --uidmap=0:2200000:65536 --gidmap=0:2200000:65536 docker.io/library/busybox true
   Error: error creating container storage: error creating an ID-mapped copy of layer "d31505fd5050f6b96ca3268d1db58fc91ae561ddf14eaabc41d63ea2ef8c1c6d": exit status 1: error during chown: error mapping host ID pair idtools.IDPair{UID:2200000, GID:2200000} for "bin/adduser" to container: Host ID 2200000 cannot be mapped to a container ID
   

/bin/adduser's uid was formerly chowned by other /bin/* files to 2200000, but the expected uid is 2100000 which is the default subuid range for podman user in step 2.

[giuseppe]

opened a PR: #1144