/dev/console of docker implementation breaks systemd >= 231

[DirkTheDaring]

Hi guys,

I heard bad reports abouts that systemd and docker folks cannot get along.
Hope this is not the case. With fedora-25 (alpha, now approaching beta) systemd stopped working correctly within a docker container as process 1, if it configured to log to /dev/console. It worked until fedora 24 because systemd actually contained a bug, which made docker "just work." Now with the technically more correct implementation /dev/console stopped working with docker, which actually needs now work the docker guys.
Please have a look at

systemd/systemd#4262

It contains a technical sound description of Lennart, what needs to investigated/changed into docker to make it work again in the comments to the broken /dev/console

Without any change, systemd will stop working from this point which is, not a systemd, but a docker issue, from a technical point of view.

[LK4D4]

ping @mrunalp
Probably something for you. I'm not sure if it's possible to run systemd in a container with vanilla docker.
Let me know if you understand educated guess of Lennart there and thread overall.

[mitchcapper]

systemd/systemd#4262 (comment) is the primary relevant comment in the ticket.

[justincormack]

@LK4D4 yes, systemd does run in vanilla Docker if set up correctly.

[mrunalp]

Yeah, it should work with some tweaks like mounting --tmpfs at /run --tmpfs at /tmp, bind mounting /sys/fs/cgroup/systemd read write and setting signal to SIGRTMIN+3. I'll take a look at the systemd report to see what's happening with /dev/console.

[DirkTheDaring]

Yes systemddefinitly runs in fedora 22,23,24.

Although it is broken now for fedora 25, when you try to run it in
"serious" environment, which needs all output routed to /dev/console
(which in turn is collected by kubernetes and elk). See ticket for why.

Actually if you use some of Lennarts tricks he provides with machine
manager it works with docker with something like

dnf install --installroot=destdir  <yourpackages>

cd destdir
tar  -c .| docker import master.fedora-24-full

Here is a docker recipe, which I use, there are so many broken systemd
recipes for docker out there. Note e.g. the STOPSIGNAL statement, and
also the disabling of certain services which are not needed in a container.

---

FROM master.fedora-24-full
MAINTAINER [email protected]

# A systemd container will probably never run correctly under MacOS (sorry!)
# \* /sys/fs/cgroup is needed  until kernel work for cgroups is completed (started at 4.6 ongoing)
# \* tty is needed to output on for docker logs
# \* SYS_ADMIN capability is needed for systemd in order to be able to change user ids for services
#
# Example how to run the image from docker
# docker run -d --tty  --security-opt=seccomp:unconfined --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:ro --hostname systemd-fedora.prod.cloud --name systemd fedora.prod.cloud  images.fedora-24-full.systemd
#
# To follow logs
# docker logs -f systemd-fedora.prod.cloud
# Tell systemd service that it is running in a docker container
# https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
# Section "Environment Variables/1"

ENV container docker

# Turn off services which usually are only needed for
# handling hardware or desktop/virtual machine operation
# Some services like sshd may require turning on stuff
# but this can be seen in the logs if a requirement
# for a service fails
# Nothing else done by the standard images!

RUN for service in\
 console-getty.service\
 dbus.service\
 dbus.socket\
 dev-hugepages.mount\
 getty.target\
 sys-fs-fuse-connections.mount\
 systemd-logind.service\
 systemd-remount-fs.service\
 systemd-udevd\
 systemd-vconsole-setup.service\
 dnf-makecache.timer\
 ;do systemctl mask $service; done

# systemd complains in newer versions if this is not set
# probably not needed if journal persistence is not used
RUN chattr +C /var/log/journal

# Redirect the logging of journald purely to console
# which enables docker logs  to catch everything
# also any persistence of the log in the container
# is turned off
#
ADD install/journald.conf /etc/systemd/journald.conf

# Turn of [OK] messages and log coloring
ADD install/system.conf   /etc/systemd/system.conf

# Required for systemd: This should go away at some time
VOLUME [ "/sys/fs/cgroup" ]

# Terminate systemd with the proper signal (inits shutdown)
# "--stop-signal=$(kill -l RTMIN+3)
STOPSIGNAL 37

# We should return tu /sbin/init at some time
CMD    [ "/usr/sbin/init" ]

# NOTE: Run container with "--stop-signal=$(kill -l RTMIN+3)" option to

[keszybz]

@DirkTheDaring: for various services that you disable, some of them have been split out into systemd-udev.rpm in F25+, so you can just not install that rpm. You'll save some space and bother.

[crosbymichael]

From reading the stuff it does not seem to be a problem handling signals but that we close the ptymaster when the copy returns because systemd closes the stdin.

@DirkTheDaring do you have an easy way to help verify a fix if I create a patch?

[mrunalp]

@crosbymichael That's my read of the problem as well. One solution that I read about was detecting that the slave is being opened using inotify.

[crosbymichael]

@DirkTheDaring can you give us more information about the docker run command that you are running? Are you using the -i or -t flag?

[DirkTheDaring]

I will be happy about anything which makes my parch go away

[crosbymichael]

@DirkTheDaring can you share your docker run command line?

[DirkTheDaring]

for commandline see the dockerfile here. the master image was build with a
dnf installroot ....
and then uploaded to docker.
It basically what you install as fedorua-minimal.

FROM master.fedora-25-full

# * /sys/fs/cgroup is needed until kernel work for cgroups is completed (started at 4.6 ongoing)
# * tty is needed to create output for docker logs
# * SYS_ADMIN capability is needed for systemd in order to be able to change user ids for services
#
# Example how to run the image from docker
# docker run -d --tty  --security-opt=seccomp:unconfined --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:ro --hostname systemd-fedora.prod.cloud --name systemd-fedora.prod.cloud  images.fedora-25-full.systemd
#
# To follow logs
# docker logs -f systemd-fedora.prod.cloud

# Tell systemd service that it is running in a docker container
# https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
# Section "Environment Variables/1"
ENV container docker

# Turn off services which usually are only needed for
# handling hardware or desktop/virtual machine operation
# Some services like sshd may require turning on stuff
# but this can be seen in the logs if a requirement
# for a service fails
# Nothing else done by the standard images!

RUN for service in\
 console-getty.service\
 dbus.service\
 dbus.socket\
 dev-hugepages.mount\
 getty.target\
 sys-fs-fuse-connections.mount\
 systemd-logind.service\
 systemd-remount-fs.service\
 systemd-udevd\
 systemd-vconsole-setup.service\
 dnf-makecache.timer\
 ;do systemctl mask $service; done


# systemd complains in newer versions if this is not set
# probably not needed if journal persistence is not used
RUN chattr +C /var/log/journal

# Redirect the logging of journald purely to console
# which enables docker logs  to catch everything
# also any persistence of the log in the container
# is turned off
#
ADD install/journald.conf /etc/systemd/journald.conf
# Turn off [OK] messages and log coloring
ADD install/system.conf   /etc/systemd/system.conf

# Required for systemd: This should go away at some time
VOLUME [ "/sys/fs/cgroup" ]

# Terminate systemd with the proper signal (inits shutdown)
# "--stop-signal=$(kill -l RTMIN+3)
STOPSIGNAL 37

CMD  [ "/sbin/init" ]

[crosbymichael]

@DirkTheDaring is there anyway you can just push this image so that I can pull it so I have the same base image created from dnf?

[DirkTheDaring]

you can try this to create the image on fedora (24) with the script below (if you replace "dnf" with "yum" on centos it should also work). Then you just need to upload the created install into docker
with
OS_VERSION=25
cd /var/lib/build/fedora-$OS_VERSION/01
tar -c . |docker import - master.fedora-25-full

#!/bin/sh
function create_repo
{
  local REPO_SHORT_NAME=$1
  local REPO_NAME=$2
  local REPO_VAR=$3
cat<<EOF
[$REPO_SHORT_NAME]
name=$REPO_NAME
failovermethod=priority
metalink=https://mirrors.fedoraproject.org/metalink?repo=$REPO_VAR\$releasever&arch=\$basearch
enabled=1
metadata_expire=6h
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-\$releasever-\$basearch
skip_if_unavailable=False
EOF

}

OS_VERSION=25

mkdir -p /var/lib/build/fedora-$OS_VERSION/full/etc/yum.repos.d
mkdir -p /var/lib/build/fedora-$OS_VERSION/01

create_repo fedora          "Fedora \$releasever - \$basearch" "fedora-" >/var/lib/build/fedora-$OS_VERSION/full/etc/yum.repos.d/fedora.repo
create_repo updates         "Fedora \$releasever - \$basearch - Updates" "updates-released-f" >/var/lib/build/fedora-$OS_VERSION/full/etc/yum.repos.d/fedora-updates.repo
create_repo updates-testing "Fedora \$releasever - \$basearch - Test Updates" "updates-testing-f" >/var/lib/build/fedora-$OS_VERSION/full/etc/yum.repos.d/fedora-updates-testing.repo

cat<<EOF >/var/lib/build/fedora-$OS_VERSION/full/dnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=true
cachedir=/var/lib/build/fedora-$OS_VERSION/full/var/cache/yum/$basearch/$releasever
reposdir=/var/lib/build/fedora-$OS_VERSION/full/etc/yum.repos.d
pluginconfpath=/var/lib/build/fedora-$OS_VERSION/full/etc/dnf/plugins
EOF

dnf -y -c /var/lib/build/fedora-$OS_VERSION/full/dnf.conf --releasever=$OS_VERSION --nogpg --installroot=/var/lib/build/fedora-$OS_VERSION/full/01 '--disablerepo=*' --enablerepo=fedora --enablerepo=updates --enablerepo=updates-testing install bash rootfiles vim-minimal sssd-client e2fsprogs dnf dnf-yum fedora-release procps-ng

[crosbymichael]

@DirkTheDaring What is your exact docker version output. I cannot even get the Dockerfile to build so I'm guessing you have the RH version with patches that I don't have.

Also your docker info output would be helpful.

I am also missing your journald.conf and system.conf that you are adding in the dockerfile to successfully build a reproducible image for your bug.

[DirkTheDaring]

what kind of error do you get ? i am actually using the docker rpm from the docker site

docker --version

Docker version 1.12.1, build 23cf638

rpm -qf /usr/bin/docker

docker-engine-1.12.1-1.fc24.x86_64

cat /etc/fedora-release

Fedora release 24 (Twenty Four)

[crosbymichael]

@DirkTheDaring ok, whats the output of docker info? It could be how you have journald configured on your daemon. I don't have that logger enabled and was getting an error on build when it was looking for journald

[DirkTheDaring]

docker info

Containers: 51
Running: 40
Paused: 0
Stopped: 11
Images: 1500
Server Version: 1.12.1
Storage Driver: btrfs
Build Version: Btrfs v4.5.2
Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host overlay bridge null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.5-200.fc24.x86_64
Operating System: Fedora 24 (Cloud Edition)
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 31.16 GiB
Name: production.fritz.box
ID: 4JDQ:ZMCA:Y5BF:VL7D:BOJ5:PDJI:XYX4:YKJE:MIG7:2HK7:NL2S:LZ6N
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8

[crosbymichael]

@DirkTheDaring I have tried following your scripts but I have been unable to reproduce the issue. If you could build a small reproducible image that you can push and that I can download and replicate the problem you are having that would help very much. You were adding various files in the build that I didn't have to configure systemd and journald, I tried to replicate them but still having issues reproducing.

[DirkTheDaring]

I put it here
curl -O https://kaupon.de/images.fedora-25-full.systemd.tar.gz

md5sum:
0f5cc2dc7dc3e9bb21fb309c00c63948 images.fedora-25-full.systemd.tar.gz

Please remember that this needs the original docker-engine and not the patched fedora/redhat version (which breaks /dev/console in different way). This is what is use to install docker-engine in /etc/yum.repos.d/

cat /etc/yum.repos.d/dockerproject.repo
[dockerrepo]
name=Docker Repository
#baseurl=https://yum.dockerproject.org/repo/main/fedora/$releasever
baseurl=https://yum.dockerproject.org/repo/main/fedora/24
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

[DirkTheDaring]

i tested it with the new 1.12.3 - still the same problem

[DirkTheDaring]

Any news?

[rhatdan]

@mrunalp Have you been able to figure out anything?

[mrunalp]

/sys/fs/cgroup/systemd needs to be mounted rw.

…

On Nov 29, 2016, at 7:41 AM, Antonio Murdaca ***@***.***> wrote: @crosbymichael since you've been asking, the following simple docker run triggers this issue (by not showing any output): $ docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /run --tmpfs /tmp fedora:rawhide /usr/sbin/init Unable to find image 'fedora:rawhide' locally rawhide: Pulling from library/fedora 7b6448343582: Pull complete Digest: sha256:b8dd62c42b48b2ccecb3f6396784ff8c11eabc0fb9cab84070ace36fae1c8e30 Status: Downloaded newer image for fedora:rawhide Failed to determine whether /sys is a mount point: Operation not permitted Failed to determine whether /proc is a mount point: Operation not permitted Failed to determine whether /dev is a mount point: Operation not permitted Failed to determine whether /dev/shm is a mount point: Operation not permitted Failed to determine whether /run is a mount point: Operation not permitted Failed to determine whether /sys/fs/cgroup is a mount point: Operation not permitted Failed to determine whether /sys/fs/cgroup/systemd is a mount point: Operation not permitted [!!!!!!] Failed to mount API filesystems, freezing. Freezing execution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[runcom]

/sys/fs/cgroup/systemd needs to be mounted rw.

@mrunalp I can't get that running w/ -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw either

[mrunalp]

Yeah it won't fix this issue.

…

On Nov 29, 2016, at 8:07 AM, Antonio Murdaca ***@***.***> wrote: /sys/fs/cgroup/systemd needs to be mounted rw. @mrunalp I can't get that running w/ -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw either — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cpuguy83]

Closing since it looks like this was fixed in the console library being used (containerd/console#10).
Feel free (as always) to correct me here.

Thanks all!