JVM11 crashes during bazel build (never crashed with JVM8)

[vam-google]

Description of the problem / feature request:

bazel build fails sporadically with JVM crash on Kokoro instance (Ubuntu 16.04) for googleapis repository CI builds (owned and maintained by Google). Most often it fails on java_proto_library target build.

The heavier the build the more likely it is to fail. Currently reproducible on the n1-standard-64 GCE instance (64 vCPUs, 240 GB). Typical build time on this machine is 10 minutes.

The machine has 240GB ram, the JVM failures are not always about memory (see below for the log snippet). I'm not sure if it has anything to do with memory issues/leaks.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

It fails sporadically, but the fails started only after switching to JDK11 (the one bazel downloads from remote source). When bazel was using JDK8 for its builds there wasn't a single failure like this.

What operating system are you running Bazel on?

Ubuntu 16.04, standard Kokoro instance.

What's the output of bazel info release?

release 3.0.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/googleapis/googleapis.git
1b8b77db10a0bd0ff47a4259ae9804b6237bc1c7

Have you found anything relevant by searching the web?

I could not find anything relevant.

Any other information, logs, or outputs that you want to share?

ERROR: /tmpfs/src/github/googleapis/google/cloud/bigquery/connection/v1beta1/BUILD.bazel:12:1: Couldn't build file google/cloud/bigquery/connection/v1beta1/libconnection_proto-speed-hjar.jar: Compiling Java headers google/cloud/bigquery/connection/v1beta1/libconnection_proto-speed-hjar.jar (1 source jar) failed (Aborted): java failed: error executing command external/remotejdk11_linux/bin/java -Xverify:none -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' ... (remaining 47 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox java failed: error executing command external/remotejdk11_linux/bin/java -Xverify:none -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' ... (remaining 47 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f752d9d13b9, pid=2, tid=3
#
# JRE version:  (11.0.6+10) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.6+10-LTS, mixed mode, tiered, compressed oops, parallel gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xc633b9]  PerfMemory::alloc(unsigned long)+0x59
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /tmpfs/tools/bazel_root/586020c997d340d7957ffe3ab7890b94/sandbox/linux-sandbox/8752/execroot/com_google_googleapis/core.2)
#
# An error report file with more information is saved as:
# /tmpfs/tools/bazel_root/586020c997d340d7957ffe3ab7890b94/sandbox/linux-sandbox/8752/execroot/com_google_googleapis/hs_err_pid2.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
#

[jin]

FYI @philwo, @fweikert

[philwo]

This is a duplicate of #3236.

Please use the flag --sandbox_tmpfs_path=/tmp to solve this. We hope to be able to remove the need for this flag in the future, but for now this flag is the solution.