Bazel HEAD build fails on Ubuntu 16.04.2 LTS VirtualBox VM

[jbduncan]

Description of the problem / feature request / question:

Compiling the latest version of Bazel from HEAD (commit 9797560 as of time of me writing this), with the command bazel build //src:bazel, fails with the following error.

jonathan@jonathan-VirtualBox:~/dev/bazel$ bazel build //src:bazel
............................
WARNING: /home/jonathan/dev/bazel/tools/build_defs/docker/docker.bzl:19:1: The docker_{build,bundle} rules bundled with Bazel are deprecated in favor of:
https://github.com/bazelbuild/rules_docker. Please change BUILD loads to reference: @io_bazel_rules_docker//docker:docker.bzl and add the following to your WORKSPACE:
git_repository(
    name = "io_bazel_rules_docker",
    remote = "https://github.com/bazelbuild/rules_docker.git",
    commit = "...",
)
load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories")
docker_repositories().
INFO: Found 1 target...
ERROR: /home/jonathan/dev/bazel/src/main/tools/BUILD:42:1: C++ compilation of rule '//src/main/tools:linux-sandbox' failed: Process exited with status 1 [sandboxed].
src/main/tools/linux-sandbox-pid1.cc:193: "mount(/tmp, /tmp, NULL, MS_BIND, NULL)": Invalid argument
Use --strategy=CppCompile=standalone to disable sandboxing for the failing actions.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 10.990s, Critical Path: 0.11s

Running bazel clean --expunge && bazel shutdown beforehand or restarting the virtual machine (see Environment Info) do not seem to resolve the problem.

FWIW, bash ./compile.sh compile /usr/bin/bazel works with no problems.

If possible, provide a minimal example to reproduce the problem:

Environment info

• Operating System: Ubuntu 16.04.2 LTS. Running as a guest in VirtualBox 5.1.20, itself running on Windows 7.

• Bazel version (output of bazel info release): release 0.5.0

Have you found anything relevant by searching the web?

I have tried searching the GitHub issues with the search terms sandbox and build, but AFAICT there's no pre-existing bug report that matches or seems similar enough to the problem that I'm experiencing.

Anything else, information or logs or outputs that would be helpful?

Running bazel build //src:bazel --verbose_failures generates the following output:

jonathan@jonathan-VirtualBox:~/dev/bazel$ bazel build //src:bazel --verbose_failures
WARNING: /home/jonathan/dev/bazel/tools/build_defs/docker/docker.bzl:19:1: The docker_{build,bundle} rules bundled with Bazel are deprecated in favor of:
https://github.com/bazelbuild/rules_docker. Please change BUILD loads to reference: @io_bazel_rules_docker//docker:docker.bzl and add the following to your WORKSPACE:
git_repository(
    name = "io_bazel_rules_docker",
    remote = "https://github.com/bazelbuild/rules_docker.git",
    commit = "...",
)
load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories")
docker_repositories().
INFO: Found 1 target...
ERROR: /home/jonathan/dev/bazel/src/main/tools/BUILD:3:1: C++ compilation of rule '//src/main/tools:logging' failed: gcc failed: error executing command 
  (exec env - \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/local-fastbuild/bin/src/main/tools/_objs/logging/src/main/tools/logging.pic.d '-frandom-seed=bazel-out/local-fastbuild/bin/src/main/tools/_objs/logging/src/main/tools/logging.pic.o' -fPIC -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/main/tools/logging.cc -o bazel-out/local-fastbuild/bin/src/main/tools/_objs/logging/src/main/tools/logging.pic.o)

Use --sandbox_debug to see verbose messages from the sandbox.
src/main/tools/linux-sandbox-pid1.cc:193: "mount(/tmp, /tmp, NULL, MS_BIND, NULL)": Invalid argument
Use --strategy=CppCompile=standalone to disable sandboxing for the failing actions.
Target //src:bazel failed to build
INFO: Elapsed time: 0.465s, Critical Path: 0.08s

[jbduncan]

Upgrading to VirtualBox 5.1.22 and updating the Guest Additions within my Ubuntu VM to 5.1.22 seem to have no effect.

Furthermore, manually deleting the contents of ~/.cache/bazel/ and running bazel clean --expunge && bazel shutdown && bazel build //src:bazel also has no effect.

[jbduncan]

Running bazel clean --expunge && bazel shutdown && bazel build //src:bazel --verbose_failures --sandbox_debug produces the following log.

jonathan@jonathan-VirtualBox:~/dev/bazel$ bazel clean --expunge && bazel shutdown && bazel build //src:bazel --verbose_failures --sandbox_debug
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
........................
.........................
WARNING: /home/jonathan/dev/bazel/tools/build_defs/docker/docker.bzl:19:1: The docker_{build,bundle} rules bundled with Bazel are deprecated in favor of:
https://github.com/bazelbuild/rules_docker. Please change BUILD loads to reference: @io_bazel_rules_docker//docker:docker.bzl and add the following to your WORKSPACE:
git_repository(
    name = "io_bazel_rules_docker",
    remote = "https://github.com/bazelbuild/rules_docker.git",
    commit = "...",
)
load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories")
docker_repositories().
INFO: Found 1 target...
ERROR: /home/jonathan/dev/bazel/third_party/protobuf/3.2.0/BUILD:120:1: C++ compilation of rule '//third_party/protobuf/3.2.0:protobuf_lite' failed: linux-sandbox failed: error executing command 
  (exec env - \
    PATH=/home/jonathan/bin:/home/jonathan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  /home/jonathan/.cache/bazel/_bazel_jonathan/481558f4c3d2e28e5d5b53ee3e13dfca/execroot/bazel/_bin/linux-sandbox -D -w /dev/shm -w /tmp -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++0x' -g0 -MD -MF bazel-out/host/bin/third_party/protobuf/3.2.0/_objs/protobuf_lite/third_party/protobuf/3.2.0/src/google/protobuf/stubs/status.d '-frandom-seed=bazel-out/host/bin/third_party/protobuf/3.2.0/_objs/protobuf_lite/third_party/protobuf/3.2.0/src/google/protobuf/stubs/status.o' -iquote . -iquote bazel-out/host/genfiles -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem third_party/protobuf/3.2.0/src -isystem bazel-out/host/genfiles/third_party/protobuf/3.2.0/src -isystem external/bazel_tools/tools/cpp/gcc3 -DHAVE_PTHREAD -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/protobuf/3.2.0/src/google/protobuf/stubs/status.cc -o bazel-out/host/bin/third_party/protobuf/3.2.0/_objs/protobuf_lite/third_party/protobuf/3.2.0/src/google/protobuf/stubs/status.o).
src/main/tools/linux-sandbox.cc:169: linux-sandbox-pid1 has PID 21781
src/main/tools/linux-sandbox-pid1.cc:171: working dir: /home/jonathan/.cache/bazel/_bazel_jonathan/481558f4c3d2e28e5d5b53ee3e13dfca/bazel-sandbox/8192938379222705133/execroot/bazel
src/main/tools/linux-sandbox-pid1.cc:189: writable: /dev/shm
src/main/tools/linux-sandbox-pid1.cc:189: writable: /tmp
src/main/tools/linux-sandbox-pid1.cc:193: "mount(/tmp, /tmp, NULL, MS_BIND, NULL)": Invalid argument
src/main/tools/linux-sandbox.cc:209: child exited normally with exitcode 1
Use --strategy=CppCompile=standalone to disable sandboxing for the failing actions.
Target //src:bazel failed to build
INFO: Elapsed time: 15.977s, Critical Path: 0.20s

[jbduncan]

Re-cloning Bazel from the GitHub repository and running bazel build //src:bazel on the fresh clone does not resolve things either AFAICT.

[laszlocsomor]

@philwo : rings any bells?

[philwo]

@jbduncan Something's up with your /tmp directory, causing the sandbox to fail to remount it writable (which it tries to do by default).

Would you mind sharing the output of "ls -ld /tmp" and "mount"? Maybe that gives a hint on why that operation fails.

[jbduncan]

@philwo I wouldn't mind at all. :)

Here's what those commands have to say about my VM.

jonathan@jonathan-VirtualBox:~$ ls -ld /tmp
drwxrwxrwt 11 root root 4096 May 29 17:36 /tmp

jonathan@jonathan-VirtualBox:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=2588884k,nr_inodes=647221,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=522164k,mode=755)
/dev/mapper/ubuntu--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10790)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda1 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=522164k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
jetbrains-toolbox on /tmp/.mount_Y1RN4q type fuse.jetbrains-toolbox (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Hope this helps.

[philwo]

jetbrains-toolbox on /tmp/.mount_Y1RN4q type fuse.jetbrains-toolbox (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)

That's it. The current workaround is to unmount that.

I'll prepare a real fix this week, too.

[jbduncan]

Thanks for the very quick response! I'll take a look into the concept of unmounting, as I'm unfamiliar with that aspect of Linux, but if you have any advice or thoughts whilst I investigate it, I would appreciate it. :)

I'll prepare a real fix this week, too.

Many thanks in advance. 👍

[philwo]

Cool! One workaround would be to uninstall the Jetbrains Toolbox in the VM and install the Jetbrains apps that you need (IntelliJ, Clion, ...) individually instead. I think it would solve the problem - the name "jetbrains-toolbox" hints that the Toolbox is responsible for this mount point and that it might go away after uninstalling that and rebooting the VM.

[jbduncan]

Thanks @philwo! It seems that simply closing JetBrains Toolbox causes the "mount" to unmount itself. After closing JetBrains Toolbox, bazel build //src:bazel appears to no longer encounter the error I initially experienced (I'll need to wait for the command to finish before I can fully confirm this). So I'll investigate uninstalling JetBrains Toolbox and reinstalling IntelliJ separately if needed.

[jbduncan]

Indeed, bazel build //src:bazel runs without any problems when JetBrains Toolbox is closed!

[Helcaraxan]

(Friendly ping) Any news on the real fix @philwo?

Unfortunately we use IntelliJ a lot on most of our dev machines and the work-around is not really usable in a practical manner for us.

This blocks us effectively from migrating from 0.4.5 to 0.5.0 and blocks further work on using Bazel as a dependency.

[philwo]

@Helcaraxan A fix for this has been submitted by a colleague and should land in the GitHub repo later today. :) I tagged it with this bug so it will automatically be pinged.

Would be appreciated if you could build Bazel from HEAD then and give it a try to make sure that it really solves the issue for you.

[Helcaraxan]

@philwo We just tested a fresh HEAD build and it looks like it has solved the issue.

Now hoping that this gets ASAP into 0.5.1 and released. 😄

[hlopko]

Sorry Duco, just pushed the button for 0.5.1 (which is already too late if you ask mac people :). 0.5.2 will be cut in 2-3 weeks, is that acceptable for you?