bazel: 0.22.0 -> 0.24.0 #58147

groodt · 2019-03-22T20:50:25Z

Motivation for this change

Things done

marsam · 2019-03-22T23:27:41Z

there is already #58116

groodt · 2019-03-23T00:27:31Z

there is already #58116

Haha! I should have checked! I know @uri-canva ;)

groodt · 2019-03-25T06:58:11Z

Ok, I'm going to try again.

groodt · 2019-03-25T07:09:29Z

@marsam @Profpatsch
I'm looking at this line: https://github.com/NixOS/nixpkgs/pull/58147/files#diff-4746b81564a01e241a84d455575437e7R244

Would that usage of TMP cause sandbox builds to fail?

Locally, I get a different error to hydra. I get the following when running with sandbox = true:
mkdir: cannot create directory '/tmp/.bazel-501': Operation not permitted

When running with sandbox = false it appears to build on Darwin if I create a fake directory /blah with 777 chmod and change that line to use /blah

Hydra timing out after 600 seconds of silence, might be because the compilation takes longer than 10 minutes? I suspect that maybe hydra doesn't have the same TERM or something to see the fancy colored Bazel compilation progress. It takes at least 20-30 minutes to build.

groodt · 2019-03-25T11:38:40Z

I've hacked the derivation a bit and it's almost working now. Failing at the installCheckPhase.

In file included from examples/cpp/hello-world.cc:1:
./examples/cpp/hello-lib.h:4:10: fatal error: 'string' file not found
#include <string>
         ^~~~~~~~
1 error generated.
INFO: Elapsed time: 34.407s, Critical Path: 0.16s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
//examples/java-native/src/test/java/com/example/myproject:hello      NO STATUS

I'll continue looking at this tomorrow. In the meantime, if anybody can review my changes and suggest better ways to do it, I'm happy to modify things within reason.

groodt · 2019-03-25T11:48:25Z

Note, my error above is with sandbox = false (I suspect it's a CPP or C toolchain issue).

If I build with sandbox = true I get failed with exit code 134 so I suspect I'm doing something that is banned in sandbox mode.

uri-canva · 2019-03-25T13:22:03Z

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

groodt · 2019-03-25T21:17:03Z

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

Good point. I'll check this. I thought that I was copying the .bazelrc along for the ride and then cd ./bazel_src would make it available. I'll try to debug.

groodt · 2019-03-25T22:10:50Z

Ok, this is now building successfully on my Mac with sandbox = false.

With sandbox = true I still get failed with exit code 134

uri-canva · 2019-03-25T22:41:32Z

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

groodt · 2019-03-25T22:43:03Z

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

Yes, I suspect the same thing.

Building Bazel from scratch
builder for '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed

groodt · 2019-03-26T09:59:16Z

Patched my build to output tracing. I think this is the reason why I fail to build with sandbox = true.

...
++ test -z /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5
++ JAVAC=/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
++ [[ -x /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac ]]
+++ /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac -version
++ JAVAC_VERSION='dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()'
+ run_atexit_handlers 134
+ local exit_code=134
...

Profpatsch · 2019-03-26T10:46:01Z

@GrahamcOfBorg build bazel

Profpatsch · 2019-03-26T11:45:19Z

Darwin build is timing out.

uri-canva · 2019-03-26T12:57:28Z

There's headless versions of jdk11 and jre for Linux. If we could have headless versions of jdk8 for macOS, then the build wouldn't attempt to access Cocoa and it might work. Did something change recently with how sandboxing is enforced? I think system libraries on macOS were still being access in non-pure fashion until a couple of months ago.

groodt · 2019-03-26T19:09:42Z

@Profpatsch

Darwin build is timing out.

I see that. What would you suggest now? This is the same as others have seen trying to get a new version of Bazel over the line. How can we get more build output? On my own Mac, I see a lot more output.

Given the local sandbox failure loading Cocoa, I think it's possible that it never built successfully on Darwin with sandbox = true or the sandboxing mechanisms have changed recently.

I'll keep digging, but it's looking like this might have to be changed to a linux only pkg if sandboxed Darwin builds are a blocker.

groodt · 2019-03-26T21:22:55Z

Quick post to summarise the difference between failing sandbox builds on a Mac vs on Hydra.

Mac (Tracing disabled. The underlying failure is due to a javac call reaching outside the sandbox for Cocoa libraries):

fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
Building Bazel from scratch
builder for '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed

Hydra (https://logs.nix.ci/?key=nixos/nixpkgs.58147&attempt_id=2a30e2bb-eabb-41f2-8786-a53b3be2509a):

fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
building of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' timed out after 600 seconds of silence
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed

Note the missing "Building Bazel from scratch" output in Hydra. That line comes from ./bazel_src/compile.sh. Either we are losing logging somehow or buildPhase isn't even starting.

I'm going to try enable tracing and kick off a build, but I'm unsure if the bot will listen to me.

groodt · 2019-03-26T21:24:58Z

@GrahamcOfBorg build bazel

groodt · 2019-03-26T22:25:49Z

Just in case you're curious, even if I patch away the line in the build where it initially sets JAVAC_VERSION to hardcode "javac 1.8" it still fails with sandbox = true. This is failing with the core invocation of javac to compile the Bazel java classes.

/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac -classpath third_party/truth8/truth-java8-extension-0.42.jar:third_party/javax_annotations/javax.annotation-api-1.3.2.jar:third_party/javax_annotations/javax.annotation-api-1.3.2-sources.jar
<SNIP ... />
-sourcepath src/java_tools/singlejar/java/com/google/devtools/build/zip:src/main/java:src/tools/xcode-common/java/com/google/devtools/build/xcode/common:src/tools/xcode-common/java/com/google/devtools/build/xcode/util:tools/java/runfiles:third_party/java/dd_plist/java:/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/src -d /private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/classes -source 1.8 -target 1.8 -encoding UTF-8 @/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_nuJxfgmI/param
dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
++ exit 0
+ run_atexit_handlers 0
+ local exit_code=0
+ local failed=no
+ for handler in ${ATEXIT_HANDLERS}

Apologies for all the spamming.

The above indicates to me that it is not possible to compile Java applications on Darwin with sandbox = true unless there is a way to prevent javac from reaching out for Cocoa. I hope that I'm mistaken. Does anybody know of any other Java applications that are building on Darwin with sandboxing enabled?

If not, I think this leaves Bazel 0.23.2 on Darwin dead in the water as an official Nixpkg unless a headless jdk comes along for Darwin.

uri-canva · 2019-03-26T23:25:21Z

0.24.0 was just released. @Profpatsch any ideas how we can debug this issue? It looks like the issues we get when building locally on macOS are completely different from building in hydra on macOS to be honest, I'm not sure spending time fixing what we're encountering locally is even helpful.

groodt · 2019-03-26T23:57:13Z

I can close this one if people wish to try 0.24.0. I've just attempted it and it's failing to compile for me on OSX, so I don't think it's ready just yet.

I'm not sure spending time fixing what we're encountering locally is even helpful.
I think it would be great to see why Hydra isn't outputting any logs once it starts the build. My current branch has tracing enabled, so if we can get that to run on Hydra, we'll at least see where it stops.

I agree that the Hydra Darwin builds outputting no logs could be a different issue / PR though. That assumes it's a general issue with Hydra and Darwin and not something specific to building Bazel.

groodt · 2019-03-30T22:24:41Z

Status update.

Building successfully on Darwin with these settings in /etc/nix/nix.conf:

sandbox = true
extra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

Failing on linux. Investigating now, but it looks like a standard issue.

groodt · 2019-03-30T22:37:03Z

@groodt the linux build succeeds when the sandbox is turned off (it already is on Darwin). The reason is that in order to be sandbox friendly, you need to prefetch dependencies, since there is no network access inside the sandbox. We already need to do this, so you probably just need to add new tarballs to srcDeps (grep Bazel's default.nix).

@mboes This is true somewhat, but if you look at the game of whack-a-mole I've been playing in this thread, it's not so straightforward. Ofborg has been running with sandboxing enabled, so Darwin builds have been "timing out" (actually SIGABRT sandbox errors).

This resulted in us chasing our tails and making changes to the already working derivation of 0.22. It might be that some of these changes I've made to make Darwin sandbox happy, are causing it to fail on linux.

The early builds of this derivation were passing on linux and "timed out" on Darwin. Now it is working on Darwin and failing on Linux. 👍

Anyway, it's probably just a matter of downloading additional files or rolling back some of my changes that made Darwin sandbox happy. I'll take a look today.

uri-canva · 2019-03-31T03:04:38Z

One thing I thought of in the meantime: skipping 0.23.x could make it harder for users to upgrade, since Bazel keeps only one version of backwards compatibility. We should have two separate upgrade commits, even if they're one right after the other, so that people who pin nixpkgs can upgrade to 0.23.2 first and then to 0.24.0.

uri-canva · 2019-03-31T03:14:13Z

I reopened #58116 in case we want to go ahead and merge that first.

groodt · 2019-03-31T04:04:59Z

Having a Hydra build of bazel 0.23.2 for pinning sounds good to me.

I'm currently testing linux sandboxed builds of this PR, but I think it's also ready now.

groodt · 2019-03-31T04:44:36Z

This is now building and ready for review @Profpatsch @mboes Got there in the end!

Please see @uri-canva comment about thoughts of doing 0.22 -> 0.23.2 first. I don't mind if that is done first. I don't see too much benefit in sticking to an old Bazel for very long, so we should get this one merged shortly after if possible.

See below table for how I've tested this derivation on local Darwin and Linux. I'm confident that it will work on Hydra now. Linux should work on Ofborg and Hydra. Darwin might fail on Ofborg, but should succeed on Hydra given the testing that I've done.

Status	OS	sandbox	Notes
✅	Linux	false	single-user Nix 2.2.1 on privileged debian:jessie
✅	Linux	true	single-user Nix 2.2.1 on privileged debian:jessie
✅	Darwin	false	sandbox = false
✅	Darwin	true	sandbox = true extra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

infinisil · 2019-03-31T06:17:54Z

@GrahamcOfBorg build bazel

LnL7

Looks good from the darwin side. Thanks for fixing the sandbox issues and sorry for the confusion.

LnL7 · 2019-03-31T10:40:07Z

pkgs/development/tools/build-managers/bazel/default.nix

  };

+  # Necessary for the tests to pass on Darwin with sandbox enabled.
+  # Bazel starts a local server and needs to bind a local address.
+  __darwinAllowLocalNetworking = true;


I assume this is inherent with bazel, is this a random or fixed port? If it's static hydra and local builds will have problems if multiple bazel builds are running at the same time.

Yes, it is inherent to Bazel. It uses a daemon for faster builds. There used to be a --batch option, but it is being removed: https://docs.bazel.build/versions/master/command-line-reference.html#flag--batch

The code that I looked at seems to allow the kernel to choose a random port.

mboes · 2019-03-31T12:26:51Z

@Profpatsch As far as I'm concerned we can merge.

groodt · 2019-04-01T17:54:33Z

@Profpatsch Any feedback? There is typically a new Bazel release every 2-4 weeks, so maybe we should land this one after all. If somebody needs an older version, they can call a custom derivation. Anybody using Bazel has to stay up to date with new versions anyway.

…

On Sun, 31 Mar 2019 at 23:26, Mathieu Boespflug ***@***.***> wrote: @Profpatsch As far as I'm concerned we can merge. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#58147 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU9d0uoR4vSn6NiVSrYri-lz5cAcdcYks5vcKmPgaJpZM4cEZQ6> .

Profpatsch · 2019-04-02T09:53:53Z

Yep, we should merge even if we skip 0.23, the releases are too fast for the manpower we have here.

Thanks for the great change!

flokli · 2019-04-03T16:25:07Z

Should this be backported to 19.03 as well?

groodt · 2019-04-03T21:15:01Z

Should this be backported to 19.03 as well?

I'd like this too, but it's a bit late in the game for 19.03 I think. They've been working really hard to get it out the door. I don't know what the policy is for late inclusions. I guess with Nix it should be safe, but 🤷‍♂️

Profpatsch · 2019-04-04T09:45:58Z

Nah, it’s a major, incompatible update, so we shouldn’t backport it.

Per NixOS#58147 (comment) sandboxing is not fully functional on macOS. .github: specify where sandboxing can be run Co-Authored-By: Jon <[email protected]> .github: remove macOS note

groodt requested a review from Profpatsch as a code owner March 22, 2019 20:50

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Mar 22, 2019

groodt closed this Mar 23, 2019

groodt deleted the greg/bazel-0.23.2 branch March 23, 2019 00:27

groodt restored the greg/bazel-0.23.2 branch March 25, 2019 06:57

groodt reopened this Mar 25, 2019

groodt changed the title ~~bazel: 0.22.0 -> 0.23.2~~ [WIP] bazel: 0.22.0 -> 0.23.2 Mar 25, 2019

groodt changed the title ~~[WIP] bazel: 0.22.0 -> 0.23.2~~ bazel: 0.22.0 -> 0.23.2 Mar 26, 2019

groodt force-pushed the greg/bazel-0.23.2 branch from f1e5086 to 75d2384 Compare March 31, 2019 02:44

bazel: 0.22.0 -> 0.24.0

d019fd8

groodt force-pushed the greg/bazel-0.23.2 branch from f0f9222 to d019fd8 Compare March 31, 2019 02:47

uri-canva mentioned this pull request Mar 31, 2019

bazel: 0.22.0 -> 0.23.2 #58116

Closed

10 tasks

LnL7 approved these changes Mar 31, 2019

View reviewed changes

LnL7 merged commit 2e2ab46 into NixOS:master Apr 2, 2019

Profpatsch mentioned this pull request Apr 3, 2019

Make compatible/update to bazel 0.24 tweag/rules_haskell#808

Closed

flokli mentioned this pull request Apr 3, 2019

bazel: 0.22.0 -> 0.23.0 #56587

Closed

10 tasks

marsam mentioned this pull request Aug 8, 2019

Cannot build tendermint in darwin #66307

Closed

azuzunaga mentioned this pull request Oct 28, 2019

.github: clarify things done re: sandboxing #72201

Merged

10 tasks

hlolli mentioned this pull request Nov 6, 2019

libuv on darwin fails in sandbox #72938

Closed

marsam mentioned this pull request Oct 29, 2020

grafana-loki: 1.6.1 -> 2.0.0 #101986

Merged

10 tasks

bazel: 0.22.0 -> 0.24.0 #58147

bazel: 0.22.0 -> 0.24.0 #58147

Conversation

groodt commented Mar 22, 2019 • edited Loading

Motivation for this change

Things done

marsam commented Mar 22, 2019

groodt commented Mar 23, 2019

groodt commented Mar 25, 2019

groodt commented Mar 25, 2019

groodt commented Mar 25, 2019

groodt commented Mar 25, 2019

uri-canva commented Mar 25, 2019

groodt commented Mar 25, 2019

groodt commented Mar 25, 2019

uri-canva commented Mar 25, 2019

groodt commented Mar 25, 2019 • edited Loading

groodt commented Mar 26, 2019

Profpatsch commented Mar 26, 2019

Profpatsch commented Mar 26, 2019

uri-canva commented Mar 26, 2019

groodt commented Mar 26, 2019

groodt commented Mar 26, 2019

groodt commented Mar 26, 2019

groodt commented Mar 26, 2019

uri-canva commented Mar 26, 2019

groodt commented Mar 26, 2019

groodt commented Mar 30, 2019

groodt commented Mar 30, 2019

uri-canva commented Mar 31, 2019

uri-canva commented Mar 31, 2019

groodt commented Mar 31, 2019

groodt commented Mar 31, 2019

infinisil commented Mar 31, 2019

LnL7 left a comment

Choose a reason for hiding this comment

LnL7 Mar 31, 2019

Choose a reason for hiding this comment

groodt Mar 31, 2019

Choose a reason for hiding this comment

mboes commented Mar 31, 2019 via email

groodt commented Apr 1, 2019 via email

Profpatsch commented Apr 2, 2019 • edited Loading

flokli commented Apr 3, 2019

groodt commented Apr 3, 2019

Profpatsch commented Apr 4, 2019

groodt commented Mar 22, 2019 •

edited

Loading

groodt commented Mar 25, 2019 •

edited

Loading

Profpatsch commented Apr 2, 2019 •

edited

Loading